Please let me know if this reference/site is useful to you:

Proposed Indicators to Measure Adherence to and Effects of Rights

...

http://evidenceproject.popcouncil.org/wp-content/uploads/2015/11/

Proposed-Indicators-of-Human-Rights-and-Family-Planning.pdfThe Evidence Project uses implementation science—the strategic ... because we do not know which metrics accurately measure adherence to and effects of ... Family Planning 2020: Rights and Empowerment Principles for Family Planning.

Dennis

Dennis Mazur

Thanks for replying Dennis.

I looked at the pdf but I couldn't see how it related to my interest in methods of validating  the interview process. Maybe I'm missing something.

James

There are quite a few references in the following online article: 

Understanding Reliability and Validity inQualitative Research

Nahid Golafshani

The Qualitative Report

University of Toronto, [email protected]

http://nsuworks.nova.edu/cgi/viewcontent.cgi?article=1870&context=tqr

Also, you'll find articles in Qualitative ResearchJjournals. There is an online list at http://www.slu.edu/organizations/qrc/QRjournals.html

Thanks for the links Hendrika.

The Golafshani paper is interesting in a general sense but I can't see how it addresses the issue of a systematically testing an interview for "validity" or "reliability" or "trustworthiness". Triangulation is useful for comparing the analysis of data obtained by different methods but it does not examine how the data was collected in an interview.

Given there are some well-documented ways an interviewer can unintentionally 'lead the witness'  (e.g. introduction of content, leading question structure, presupposition, framing, priming, tacit metaphor, gesture), and that many interviewees will not be aware of the effects of these features of communication, are there any methodologies which enable us to determine whether these verbal and nonverbal features have compromised the data gathered in an interview (or at least cast doubt on its authenticity)?

Hello James

   It seems to me you are trying to bring a level of certainty more in line with quantitative research to the qualitative arena. I would answer with a "no" your question "...are there any methodologies...".  In qualitative interviewing, the point is to elicit the interviewee's understanding of the phenomenon of interest, not to follow the interviewer´s preconceived notions. Therefore, most of the hard work of constructing an adequate interview guide is up front, not after the interviews are completed. The researcher needs to be familiar with the setting/phenomenon to be studied and have some knowledge of the lanuage used by participants in order to help avoid some of the traps you noted above. It is sometimes a good idea  to "pretest" the interview with people who will not be part of the final sample, to make sure your questions are giving you responses that are useful. Experienced interviewers may prefer to begin an interview with a very open question, such as "Can you tell me about X?" and then proceed with further questions using the informant´s words to describe the object of interest. In short, I am not aware of any instrument that can test the "validity" of interview responses, and as the analytic strategy in qualitative research is generally interpretive, I don´t see the possibility if constructing one.

Good luck with your project.

You might scan through the 300+ citations to Rousten's (2010) article on this topic, but I certainly am not aware of anyone who has guidelines for assessing these distinctions.

I really appreciate your reply Andrea,  since it address exactly those points I am trying to investigate. So I will respond to each of your points.

• It seems to me you are trying to bring a level of certainty more in line with quantitative research to the qualitative arena. I would answer with a "no" your question "...are there any methodologies...".

I see a difference between ‘quantitative’ and ‘systematic’ methodologies, and that 'systematic qualitative' methods are possible. The advantages of systematic protocols is that, to some degree, the planned method can be compared to what actually happens. When this doesn’t happen (and I think it is quite common in interviews) it undermines the trustworthiness of the research.

• In qualitative interviewing, the point is to elicit the interviewee's understanding of the phenomenon of interest, not to follow the interviewer´s preconceived notions. Therefore, most of the hard work of constructing an adequate interview guide is up front, not after the interviews are completed.

I agree, and I am looking for methods are available to validate whether what takes place in the interviews is what was planned in the interview guide. For example, several papers say the interviewer “avoided leading questions”. But while the intent maybe there, how does the reader of the paper know whether that is a fair or misleading statement? (I have many examples of research interviewers asking unintended leading questions.)

• The researcher needs to be familiar with the setting/phenomenon to be studied and have some knowledge of the lanuage used by participants in order to help avoid some of the traps you noted above. It is sometimes a good idea to "pretest" the interview with people who will not be part of the final sample, to make sure your questions are giving you responses that are useful.

All good advice. I also recommend as part of the "pretest" phase the interviewer is interviewed (using the guidelines) so they know what it is like to answer the questions and to be aware of their own opinions/experiences.

• Experienced interviewers may prefer to begin an interview with a very open question, such as "Can you tell me about X?" and then proceed with further questions using the informant´s words to describe the object of interest.

Your example illustrates my concern beautifully. If we assume that an interview guide recommends, as you suggest, “very open questions” then your opening question does not conform to the guide since it is an example of a “closed” question (the interviewee can legitimately answer ‘yes’ or ‘no’). As a reader of the research I want to know how often this kind of variance from the interview guide happens in the interviews. If it happens repeatedly, it would raise doubts in my mind as to the reliability of any analysis or conclusion based on these interviews.

• In short, I am not aware of any instrument that can test the "validity" of interview responses, and as the analytic strategy in qualitative research is generally interpretive, I don´t see the possibility if constructing one.

I too cannot find any instrument (which is why I asked my question), however I do think it is possible to create one. In fact, I have attempted to create such an instrument, called a “cleanness rating”, which has been used in a number of studies (links listed).

Thanks once again for your thoughtful reply.

James

Article Eliciting Metaphor through Clean Language: An Innovation in ...

Article Using Clean Language to explore the subjectivity of coachees...

Chapter Clean Language Interviewing: Making qualitative research int...

Chapter Nehyba, J., & Svojanovský, P. (2017). Clean language as a da...

To be honest David, as a full-time practitioner who conducts research and provides support to academic researchers in my spare time, I was hoping to use the combined wisdom of this group to save me reading 300+ articles.

Your reply adds to the consensus of answers to my question. It seems that if there are any such methods there is little awareness them and, if they exist, they are rarely used.

The nearest I have found, is an instrument created back in the 1940’s to assess the degree of ‘non-directive’ interventions by Rogerian counsellors:

Porter, E.H. (1943) The Development and Evaluation of a Measure of Counseling Interview Procedures. Educational and Psychological Measurement 3(1): 105-125 https://doi.org/10.1177/001316444300300111

Please also see my reply to Andrea Schuman for links to papers referring to the instrument I have created.

Regards

James

There is a longstanding tradition in survey research of investigating what difference it makes to ask questions one way rather than another (for example, whether to use 4 points or 7 points in an agree-disagree response format, or whether to ask questions in a negatively or positively worded direction). 

But I'm not at all sure that such a research program would work with regard to qualitative interviewing. At a minimum, it would require research questions of the form: "Does interviewing approach A produce responses that have the following characteristics..." Next, it would require a way to analyze transcripts that would answer such questions.

Lets consider Andrea Schumn'a proposal:

"Experienced interviewers may prefer to begin an interview with a very open question, such as "Can you tell me about X?" and then proceed with further questions using the informant´s words to describe the object of interest."

I certainly agree that this is a widely recommended strategy and one that I attempt to follow in my own work, but how would we "operationalize" beginning with a very open question? And how would we determine what difference it whould make to use either a more open or less open question? Perhaps more to the point, what, specifically would we do if we did have the answers to such procedural questions?

If we had the large Ns available to survey researchers, and their commitment to statistical analysis, then we could conduct a series of interviews that were divided, half and half between our proposed questioning formats, followed by some kind of quantitative content analysis on the transcripts.

I personally know that this can be done on a small scale, because I have tried it (see the attached article, which is forthcoming), but this was just a pilot effort. Based on that experience, I think it would be very difficult to undertake such a research program with regard to qualitative interviewing.

Without such a research program, I'm not sure what difference it would make to learn whether researchers exhibit "cleanliness," other than to know if they live up to their own standards. Or to put in terms of a research question, how would know if a lack of cleanliness produced lower quality results?

Article A System for Coding the Interaction in Focus Groups and Dyad...

Thanks for taking the time to reply David. The questions you raise are exactly the ones I have been wrestling with for some years and I am honoured to discuss them with someone of your experience.

I appreciate you attaching a paper which involves research into the dynamics that take place inside focus groups and dyadic interviews and I refer to it later.

I have commented on the first part of Andrea Schuman’s example, and the second part is also of interest. While interviewers may aim to ask “further questions using the informant´s words”, is this what actually happens in the interview? Interviewers think this is what they are doing, detailed analysis of transcripts show how often they unwittingly substitute words, paraphrase, add in extra words and reframe the interviewee’s words. It is relatively easy to assess to what degree this happens and it’s one of the feature of the “cleanness rating” I mentioned.

Your question, “what, specifically would we do if we did have the answers to such procedural questions?” translates in my mind to the vital question, ‘So what?’

There is a large body of evidence which demonstrates that question construction; changing a single word; introduced metaphors and interviewer gesture can all materially affect responses (e.g. Thibodeau & Boroditsky, 2011; Gurney et al, 2013; Heritage, et. al. 2007; Loftus, 1975; Loftus & Palmer, 1974; Loftus & Zanni, 1975)

I work in fields where the authenticity of the data collected in interviews is of major importance. For example, my colleagues and I have worked with police interviewers to improve their questioning of vulnerable witnesses after prosecution cases collapsed when the judge rejected the videos of interviews as “leading the witness”. I have also trained Critical Incident and Root Cause Analysts who collect data on utility accidents (including in nuclear power stations). The aim of these experienced interviewers was to gather trustworthy accounts with minimal interviewer imposition, yet detailed analysis of typical interviews casts doubt on the authenticity of some of the data collected.

If the interviewer does not know that they unwittingly introduce content, assumption, leading question structure, presupposition, framing, priming, tacit metaphor and gesture, how can they do anything about it? The problem is that interviewers think they are avoiding these the “threats to validity and reliability” (Seidman, 2006).

And what are readers of the subsequent data analysis meant to conclude when many qualitative papers have little mention of what happens in the interview and virtually nothing about independent checks that interview guides were adhered to? As far as I can tell, the peer-review process pays great attention the validity of data analysis but more or less ignores the reliability of the interview process upon which the analysis is based.

What we can do is, first raise awareness of these phenomena; second, train interviewers how to minimise unintended imposition; and thirdly, hopefully raise the standards of qualitative research.

Your other questions also go to the heart of the matter. You wonder, is there anything to be learned “other than to know if they live up to their own standards.” Personally, I think this is an essential ethical issue: do researchers not only conduct research in the way they say they are going to, are they also seen to do so? Secondly, it is a methodological issue. In your paper you say, “The approach chosen was a relatively non-directive style with little active probing.” I appreciate you stating the interview guidelines, and as a reader I want to know: to what degree did the interviews adhere to this approach? How are you measuring the degree to which the interviews were “non-directive” and how “little active probing” took place? If the interviews varied substantially from the guidelines, the results would, in my mind, be less trustworthy.

You ask, “how would [we] know if a lack of cleanliness produced lower quality results?” and this is indeed a fundamental and as yet unanswered question.

There have been attempts to compare the number of “information units” collected using a Clean Interview and a traditional method. However, much more robust research is needed, and everything will depend on how “quality” is defined.

As an aside, I do not directly equate “cleanness” and “quality”. There are ways of measuring an interview’s quality which have nothing to do cleanness. For example, while it is generally accepted that asking multi-part or multiple questions is poor practice, it has no bearing on whether the questions are clean or not. In my mind the two concepts form a quadrant of four possibilities. A perfectly clean interview may collect little relevant data if the interviewer does not keep it on topic.

The primary value of an interview which has a high cleanness rating is the degree of trust that the data gathered is representative of the interviewee’s views, thoughts, feelings, mental model etc. Interviews with a low cleanness rating will involve “strongly leading” questions and statements. If the concepts, metaphors and presuppositions introduced by the interviewer are subsequently used by the interviewee, it may increase the chance that the collected data reflects the interviewer’s model-of-the-world and not the interviewee’s. I am interested in investigating whether Clean Interviewing can minimise these effects.

My approach has been to separate the question into two parts: How clean? and What quality? I have demonstrated how to measure “cleanness”, measuring “quality” is a greater challenge.

Lastly I am in complete agreement with the final statement in your paper, “our preferred approach is research-based—not just with regard to interaction in focus groups, but anything else that might affect the quality of the data that this method produces”. I am willing to collaborate with any research project that seeks to assess how the relative cleanness of an interview affect the quality of data collected.

Thanks once again for your stimulating reply.

James

I can certainly see the value of what you are advocating for working with the police and other aspects of the legal system, but I am not fully convinced that it matters as much with qualitative interviewing.

Still, one step at a time, and I fully agree with you that most publications on interviewing say far too little about how the data were collected, and this has been something that has bothered me for some time. In particular, we seldom even get a sense of what questions were asked, let alone how the interview was conducted. 

As a first step, I would advocate that journals have online appendices that include such crucial information as the content of the interview questions. This is quite common in medical journals, where much of the methods section is placed on line, so there certainly is precedent for it. But to date, we have let authors get by on minimal standards for publishing interview research.

As for my own project, in our coding, we we marked everything that the interviewer (in most cases me) said. In terms of face validity, that certainly showed a low degree of involvement in sense of making few comments and using few words in making those remarks, but we did not do any detailed coding on that aspect of the interviews.

In that paper, we were comparing focus groups and dyadic interviews, so it was important to keep the moderating style consistent across those formats. But as we note in our suggestions for future research, it would also be possible to study the differences in moderating style, in terms of how they affect the interaction in the group. So, that is another example of specifying how something about the data was influenced by something the way the interview was conducted.

Thank you David for continuing the discussion, it is very helpful to have your perspective on these matters.

It seems we agree that more transparency on what happens inside qualitative interviews would be beneficial. In that regard I appreciate you defining your “low degree of involvement” as “making few comments and using few words” – something that could quite easily be validated as long as “few” is quantified. And it is not just the number of words, the source of those words also matter – something assessed by a cleanness rating.

Also, as you say, comparing interviewer-moderator styles would be a step on from that. You may be aware of a simple yet effective approach adopted by Tomori & Beavin Bavelas (2007), Using Microanalysis of Communication to Compare Solution-Focused and Client-Centered Therapies.

There can be a large variation between the amount of clean and leading questions in interviews within the same research project. For example, Nehyba & Svojanovsky (2017) documented the variation between four interviewers on the same project as 99%, 92%, 64% and 34% of questions assessed as ‘clean’. When two thirds of questions in an interview are ‘leading’ (when that was not the intention), raises concerns in my mind about the authorship of the data collected.

Why is that?

I have given workshops at universities and conferences on three continents and it is clear to me that there is little awareness of a fundamental property of language: that in order to comprehend a statement or question the listener initially has to accept the given premises, even if they reject them afterwards. Premises are relayed not only in the content but also through many other aspects of language, most of which are processed automatically and out of awareness, e.g. question structure, presupposition, framing, priming, implicit metaphor, paralanguage and gesture.

If, as much research shows, the kind of questions asked can materially and systematically bias responses given about an observed event (e.g. in a police interview), is it more or less likely that these kinds of questions will influence responses when the topic is less tangible (e.g. experiences of people with early-stage dementia)?

Rejecting the premises of these linguistic features, especially when used repeatedly, requires an extra cognitive load which may be enough to tip the balance towards agreement. Added to this are well-known interviewee biases, such as the consistency effect, acquiescence bias and the friendliness effect which mean interviewees may (unconsciously) look for cues from the interviewer about how to answer. Given many qualitative interviews often involve topics that the interviewee may not have spent much time reflecting on, I suggest that the chances of the interviewer unwittingly providing these cues is likely to be high.

I am planning a comparative analysis between interviewers trained in Clean Interviewing and those using conventional techniques, however, getting hold of sufficient transcripts is proving a sticking point. In the meantime, these discussions are helping me get clear on the subtleties of research in this area.

My own sense is that the problem here is an empirical one: how do we specify what difference it would make to ask questions one way or another, and how do we demonstrate that such differences exist?

I took a look at the Tomari and Beavin Bavelas article you recommended, and what I saw was another specification of how to detect differences in interviewing style, but no analysis of what difference it made to do the interviews one way rather than another.

Similarly, your specifications about "clean questions" supply a research question of the form: "What difference does it make if questions are more or less cleanly asked?" In essence, this specifies an independent variable, but to me the real question is how to specify and capture the dependent variable -- i.e., what is it that systematically varies as a result of differences in asking questions?

I think it would be very hard to produce answers to the kinds of questions I am raising. That doesn't mean that we shouldn't try, but I personally do not have any real insights on how we should proceed.

David, this is a really valuable discussion for me. I appreciate you making clear the distinctions between detection and analysis of the difference between styles of interviewing. Also the distinction between specifying and measuring independent and dependent variables.

I agree it would be ideal to conduct a large-scale trial that was able to examine whether, and what kind of difference it makes to how qualitative interviews are conducted. My guess is that a number of independent small-scale studies might be as good as it gets. And if I had any good ideas about how to set up that research, believe me, I would share them.

So, until that question is answered, what can we do?

I think the evidence from a whole range of disciplines is clear that how we communicate makes a material difference to the responses we get. I have cited experiments that demonstrate, for example, that asking a question one way compared to another can systematically vary the result.

Furthermore, almost every qualitative researcher using interviews tacitly acknowledges this. Otherwise why would we seek a “low degree of involvement” by “making few comments and using few words” if we didn’t think that a high involvement would produce a different response? The same applies to Andrea Schuman’s suggestion to ask “a very open question” and to proceed with further questions “using the informant´s words”. Why bother, unless doing the opposite would influence the data in a unwanted direction? Just about every text book I have read on interviewing recommends not asking “leading questions”. Why, if we don’t think it make an unwanted difference? Because it would seem these questions potentially compromise the authenticity of the data collected, and therefore introduce doubt about its trustworthiness.

This suggests a general consensus that it does make a difference how interviews are conducted – and that some interview methods can bias the data. My proposition is that there are many more ways for interviewers to unwittingly influence the data than is commonly recognised. As you point out, how to specify and measure the dependent variable is the hard part. However, from a practical viewpoint, how can we help improve the quality of interviews while we are waiting for answers to this question?

Here are my suggestions:

1. Raise awareness of the multiple ways bias can potentially happen in interviews.

2. Use more sophisticated methods for minimising the potential “threats to validity and reliability” during an interview.

3. Acknowledge that research interviewing is a specialist skill that requires training, practice and quality feedback.

4. And, as we have discussed, demand more transparency about what actually happens during interviews and in particular, information about the degree of compliance with interview protocols.

I’ll be giving this topic more thought. If you have any more ideas please do let me know.