This is a thoughtful and wellarticulated question. I do think you should organize your data, and how depends a bit on your long term goal.

If you are not confident you can trace the problems you found to components of the system doing I would agree [1] is not the best option.

When it comes to [2]: I would first organize the data and look at the theory later to avoid the risk of fitting the data to the theory (confirmation bias).

For [3] this seems fine but some clustering and theme hunting would improve your results. Look at affinity diagramming for example (e.g. Beyer, Hugh, and Karen Holtzblatt. Contextual design: defining customer-centered systems. Elsevier, 1997.). Grounded theory would be an alternative approach but seems a bit heavy weight for this problem. 

Further: distrusting your intuition seems like a bad idea in general :) The most important things in handling qualitative data are iterating between levels of abstraction findings and data and keeping trace of the links between your findings and the data that fed it.   

This is a thoughtful and wellarticulated question. I do think you should organize your data, and how depends a bit on your long term goal.

If you are not confident you can trace the problems you found to components of the system doing I would agree [1] is not the best option.

When it comes to [2]: I would first organize the data and look at the theory later to avoid the risk of fitting the data to the theory (confirmation bias).

For [3] this seems fine but some clustering and theme hunting would improve your results. Look at affinity diagramming for example (e.g. Beyer, Hugh, and Karen Holtzblatt. Contextual design: defining customer-centered systems. Elsevier, 1997.). Grounded theory would be an alternative approach but seems a bit heavy weight for this problem. 

Further: distrusting your intuition seems like a bad idea in general :) The most important things in handling qualitative data are iterating between levels of abstraction findings and data and keeping trace of the links between your findings and the data that fed it.   

I think the best option with data of this nature is to organise it in terms of themes. So for example you can look at the data from a ‘Interpretive Phenomenological Analysis’ (IPA) perspective to elicit people’s experiences of a particular phenomenon. Then after you have a set of themes you look for relationships between themes to create a smaller supergroup of key themes that encapsulate the responses in relation to the questions you are trying to address.

Then for each supertheme you can bring example quotes from your interviews to support and highlight it's relevance and possible solutions.

See http://www.ipa.bbk.ac.uk/about-ipa or see this publication:

Watts, L & Reeves, A.J (2005) Feeling Personal: The Emotional SIDE of Media Selection Dilemmas. Workshop on 'Virtuality & Emotion', EGOS 2005, Berlin.

In my publications for an example of where IPA has been used.

Hope it helps!

Thank you to those who've given answers so far. I really appreciate your thoughtful guidance. I will definitely be exploring the suggestions you've offered.

To answer your question, Heikko, it will feed back into the system, yes. I did a heuristic (non-user) evaluation on the prototype, the user lab study was on iteration 2 of the system, and the in-the-wild user study will be on iteration 3.

I'm not 100% clear on the difference between themes and clusters right now; I'm just about to look it up and find out!

Caroline, there may be some value in comparing results of another objective reviewer of the transcripts. From that you can compare overlap and get an interrater reliability measure that may provide some assurance to your committee that there are others who at least somewhat agree with your list.  If there  is no agreement, you may want to consider what that other objective reviewer of the transcripts is saying. They should, in advance of reviewing, have some level of expertise, or you may consider training them (or having someone else train them) until they are competent to evaluate the transcripts. 

In terms of other researchers who have some related experience, I would encourage you to consider seeing research of Jesse Crosson (on ResearchGate; was at New Jersey School of Medicine and Dentistry, now at Princeton Health and has experience with Medical Informatics). 

You may also see the research articles from Mihaela Vorvoreanu (also on ResearchGate).  Both have specialized for a number of years in more qualitative methods related to human-computer interaction.

Their research (and the research of their students or co-authors) will be considered credible in view of your committee members.  If additional questions arise, you may contact me directly for any follow up.

I would not recommend working entirely 'bottom up' with material acquired through a lab study that's done as part of a design project. There is, after all, a reason why the system is built at all, and why it is constructed the way it is. Grounded theory and open phenomenological approaches are more suitable for studies of real-world phenomena, and with some luck you may even get to do some of this during your final 'in the wild' study.

I think you need to articulate better why you are doing this study.

Given that you are planning to do a field study next it might be that the study is part of your iterative design cycle. If that is true, your primary goal would be to find aspects of the system or interaction design that need to be improved before the next iteration. In this kind of 'in between' study you can be less careful about your method of analysis, to foreground the observations that motivate a change in design. Such observations must be tied to the system - they must combine what your users did with what they said. If you have video I would recommend doing an interaction analysis, but else you need to entangle this relationship from your notes. (In design you can, and should, use a mix of theories. You need not find a singular theory to match all of your salient observations and neither should you expect one to exist - check out e.g. Stolteman Forlizzi and Evenson http://dl.acm.org/citation.cfm?id=1240704)

This would make for a decent thesis chapter, but if you want to write an article on the study it gets a bit more tricky. I don't quite understand what research question you will be answering with this specific study and analysis. What was your reason to studying the system in the lab at this point of time? A wild guess is that you wanted to explore what is editing expertise, and then you could possibly analyse your data from the perspective of where the system designers' expectations did not quite match what experts in this domain actually know and do. Possibly.

Hi Caroline

I think your uncertainty owes to some confusion about the relationship between anticipated findings, transcriptions and inferences. I worked with Ahmad Reeves some years ago on a thoroughly experiential HCI research project, in which applied Interpretive Phenomenological Analysis. There is a lot of value in it but that value does depend on the focus of the investigation. HCI investigators bring a variety of concerns to their treatment of the things people say. All who have responded to you so far probably agree that there is value in listening to the things people say as they interact with a computer. I'd say 'we' all see this value in terms of evidence of the thinking your participants would ordinarily bring to bear when confronted with analogous interactive experiences. However, I shall return to this assumption at the end of my comment*.

The concerns of HCI investigators are not at all uniform and, as a result, condition the meaning and treatment of their data. 

Anticipated findings

You say you are interested in the design of a visual analytics system that visualises text cohesion, with the intention of helping editors to improve the coherence of their documents. That suggests you are mainly concerned about the quality of the work editors carry out, rather than experiential matters such as pleasure, frustration or flow. This is important for you because it delineates the type of information you wish to extract from the transcripts: presumably you intend to highlight the talk that included coherence-related terminology or concepts. I appreciate that you are writing informally here but when you say you 'had them play with the tool (using their own data as well as several other example sets of data)', I wonder how concerted their efforts may have been with respect to the way they might ordinarily invest their energy in editing work. It matters because you expect their talk to reflect their practices in a meaningful way. So you must be careful to explain how you set up a session with a domain expert with this in mind. 

Some investigators create theoretically motivated categories of event or action in advance of an analysis and then inspect a transcript to 'spot' when and where they are in evidence. They thus bring a focused, deductive attitude to their analysis. An independent analyst is then trained in the categories and required to go through a random sample of transcripts (or sections) to determine the integrity of the 'spotting' process (this is known as inter-rater reliability). This approach is only worthwhile if there is a strong pre-determined framework for analysis and risks ignoring events that are highly material to the success of the interaction design. 

Other investigators adopt an inductive attitude, attempting to discover patterns in their transcript data by working successively from low-level, research relevant observations up to aggregated categories of observations. These are then presented as findings. No inter-rater reliability applies because the reporting requires direct inclusion of the original data to demonstrate the meaning of the categories that have been generated. It is, however, normal for investigators to work together to reach agreement on the boundaries of such categories. This approach is common in Grounded Theory methods but is more general. I like the multi-phase strategy described by Braun and Clarke ('Using thematic analysis in psychology' Qualitative Research in Psychology 2006; 3: pp. 77􏰀-101).  Don't be put off by the 'in psychology' bit. The categories, as I put it, are usually presented as 'themes'. 

Transcriptions

I take it you are referring to a verbatim account of the things both you and your domain expert said during your lab sessions. However, it is probably important for you to include a mechanism for referring to aspects of the material they were working with. This is because your material is visual. So I suspect your data set are more than just the things people said. In any case, transcriptions should be all about things people did and said and, as far as possible, produced independently of your anticipated findings. 

Inferences

Inferences are, in effect, claims you wish to make on the basis of your data. For example, X is a problematic element of the visual analytics system, or Y facilitates insights that are rare in the experience of the editors you recruited. The progressive elaboration of categories (ultimately 'themes') represent the argument you make for the validity of your inferences. Alternatively, comparison of 'scores' for codes applied at different phases of the interaction. Inferences are not proofs. They are statements about matters that could be influential in the realm you are investigating, supported directly by the evidence you present. 

I think it is helpful to distinguish between anticipated findings and the inferences you finally make on the basis of your analysis. This is because one of the great benefits of qualitative research is the opportunity to look beyond the restrictions of pre-defined categories of interest. In other words, you might just discover something to do with the the business of achieving coherence in a document which was simply not on the agenda when you embarked on your study. At the same time, to be fair, a rigorous approach to an investigation must be adopted to guard against the risk we all run of attributing meaning to events which turn out to be chance happenings. Nobody wants qualitative research to be a Rorschach Inkblot: more about the investigator's prejudices than the data they encounter. That is why I value the six-phase progressive approach promoted by Braun and Clarke. It helps me to avoid getting ahead of myself. 

Hope that helps and good luck with your research - it sounds very interesting to me. 

Leon

*Methodological challenges with investigations of 'think aloud' data stem from the risk of self-censorship, additional cognitive load, distraction. You will need to discuss how these as factors may limit your ability to extrapolate your findings to every-day user experience. Similarly, the fact that you gathered your data in a lab will influence certain aspects of your participants' behaviour, such as being away from any familiar resources that they might ordinarily make use of in carrying out similar work. You are planning a study in-situ later is great because you'll be able to reflect on elements of user behaviour that both did and did not translate from the lab to the real world.  

Thanks so much for the detailed guidance, everyone. It's enormously helpful!