We face a similar challenge in the selection of qualitative methods in the survey research field for the testing and evaluation of survey questions:  The method is applied to evaluate our materials - but how do we know that the method itself is any good?  Assessing the quality of qualitative methods has received a lot of attention, and tends to divide into several key approaches:

1) Reliance on theory:  Is the method supported by theory, and is the theory itself supported?  For example, the use of 'think-aloud' techniques in psychology to study problem solving relies on Ericsson and Simon's (1980) theory concerning the veracity of verbal report processes.  

2) Empirical evaluation of the method:  Research to develop and empirically evaluate qualitative methods appear in journals such as Field Methods.  Although methods evaluation is not at all an exact science, if one can develop criteria for 'method reliability and validity' then these are amenable to testing.  For example, a valid and reliable qualitative method is one that will produce similar results when the same data are assessed over time, or by independent sets of researchers/analysts.

3) Extension from other fields:  Qualitative methods that are accepted as valid and useful in one field may be 'borrowed' in application to another one, with the expectation that if there is sufficient overlap iin key features, the method is likely effective.  For instance, the think-aloud processes applied to chess playing and anagram puzzle solution were later extended to the realm of respondent processing of survey questions.  Of course, this approach begs the question of whether the method was really useful in the original application, and whether it still applies in the new one.

4) Mixed-method approaches:  Sometimes we may be able to apply both qualitative and quantitative methods to the same problem.  In these cases, the quantitative approach may provide some quality metrics, such that the researcher can conclude that method to have utility (e.g., a psychometric analysis may reveal instances of Differential Item Functioning (DIF) across g(racial, gender) groups for a particular self-report item.  If the subsequent application of qualitative/cognitive interviewing reveals a coherent explanation for the quantitative findings, one can infer that the qualitative method (perhaps in conjunction with the quantitative) exhibits the property of method validity.

As a researcher, one should not give it up, by considering it '(almost) impossible task'. Quality of 'qualitative studies' should (methodologically) be measured. We have examples of comparing (and concluding as result of a qualitative measure) outcomes of parallel projects, on same issue.

As suggested by Gordon Willis mixed method approaches can be of use.

It depends on the meaning of quality. In Discourse Analysis quality may mean consensus formation (Habermas).

I think that my real problem is with the word 'measure'. I can see that 'quality criteria' can be reasonably applied, and there are plenty of sources for this. But how can you measure something that is fundamentally interpretive and subjective?

I use critical discourse analysis and it seems that the quality of thismodel is a relative as it depends on several factors. However, I am now a applying Fairclough's three dimension model of analysis and can personally say that it is more quality than, say, the ten question model. This is because the analysis usually starts with syntactic features of text and moves up to the context and the interpretation levels. Each level need to base on the level prior to it resulting it a decrease in the researchers' biased interpretation.

Since you mention psychology, I am guessing that what you mean by "discourse analysis" is closer to Potter's Discursive Psychology rather than critical discourse analysis. If so, you may find Potter's discussion of "validation" useful in the attached book chapter.

Potter is a Brit. Critical Theory is mostly German, reflecting an entirely different historical struggle, against power and domination. It is also more philosophically oriented , against domination and for Freedom.

I agree with you about the inherent tensions in using the word 'measure' in relation to quality of qualitative research, but also agree there is a need to establish robust and defensible interpretations and claims.

I've found the following one-pager very useful in providing a snapshot of how the various quality criteria compare across the quant and qual paradigms. I would add researcher reflexivity to the list of quality critera - this refers to the researcher considering and articulating their personal, theoretical, ethical, and epistemological views and position relative to the research, the participants, the context etc.

Frambach, J. M., van der Vleuten, C. P., & Durning, S. J. (2013). AM Last Page: Quality criteria in qualitative and quantitative research. Academic Medicine, 88(4), 552.

Hope this is of help

Koshila

I agree it can be problematic to use the word "measure". Most qualitative research is based on an interpretive epistemology, this is why we need different quality criteria to the ones used for positivist science.

I found Sandberg's contribution on how to enhance the quality of interpretive approaches very useful (he refers to Habermas also). He speaks about three strategies (“truth constellation of criteria”):

1) Communicative validity (creating a common repertoire to assure that researchers and participants can understand each others)

2) Pragmatic validity (checking the knowledge produced by practitioners, engaging with practitioners, discussing with them our understanding of their actions)

3) Transgressive validity (challenging the “taken for granted”, look for more explanations and contradictions)

I also find the word "measure" problematic, both as a noun and a verb. In particular, I think the strengths of qualitative research are not in measuring things or producing measures. For those purposes, I would advocate quantitative methods and their vocabulary of reliability and validity.

For qualitative research, I would emphasize the goals of producing meaning and interpretation, which are not measurable. Of course, we do need to defend the quality of the procedures that we use to produce meaning and interpretation, which leads to the ultimate criterion that Lincoln & Guba propose: the "credibility" of our results (Naturalistic Inquiry, 1985).

Another reference is Kathryn Roulston (2010) Considering Quality in qualitative interviewing. Qualitative Research 10 (2): 199. She looks at a range of approaches including discourse analysis.

I agree with  Dr Willis.  I think he makes some very important points about how you might assess validity. 

You could also look at Messick's four-fold facets of validity in Messick (1995) Validity of Psychological Assessment: Validation of Inferences from Persons’ Responses and Performances as Scientific Inquiry into Score Meaning, American Psychologist, September, Vol.50, No. 9, pp. 741-749.  Much of the article is not relevant to your concerns but the four-old facets are worth thinking about.