Dear Yimin

1. When designing your QCA research, the best (if imperfect) way of trying to address the problem is to base your selection of conditions as much as possible on existing theoretical and substantive knowledge, so as to ensure you have included potentially relevant conditions. You can also combine several conditions into higher-order constructs, in order to keep the number of conditions at a manageable level (once you have conceptualized these conditions, this can be implemented with the compute() function in the QCA package based on calibrated component sets).

In case-oriented variants of QCA, a good way of identifying problems with omitted variable is via in-depth study of cases. On the one hand, cases that are not covered by the solution can tell you about alternative pathways to the outcome. On the other hand, in-depth study of typical cases and deviant cases for consistency can tell you about whether the conditions you included in the truth table actually provide a good explanation, or whether there is something you left out. I find the various works by Rohlfing and Schneider on set-theoretic multi-method research very useful in this regard. It can all be implemented via the SetMethods package by Nena Oana and Schneider.

Such insights can then feed back into a redefinition of the relevant explanatory framework in order to be able to explain more cases--that is part of QCA as an iterative approach.

Data mining might enable you to maximize coverage and consistency, it might not guarantee that the identified models adequately explain the cases and reflect relevant concepts and theories--these are eventually interpretations to be made using criteria external to the QCA algorithm and parameters of fit, such as theory, case knowledge, and additional methods of analysis. But these are more general issues in empirical social research, not really peculiar to QCA. A middle way between in-depth study of cases and simple data-mining can be to try out several equally plausible and meaningful options (for example, in terms of conditions considered, or in terms of options for conceptualizing, measuring, and/or calibrating the sets) considering the state of theoretical and substantive knowledge, and then compare the resulting models regarding their performance.

2. The coverage measure can be interpreted as % cases covered by the solution for crisp sets. It cannot be interpreted as such for fuzzy sets, so the analogy has clear limits there. But of course, it is about how much of the variation in the outcome we could describe.

Low coverage means that there is a large share of your empirics that is described or explained by other conditions than the one you have considered. So, it might be a good motivator to go back to point 1 and try and think of additional or different conditions to consider. There may be situations when you are not interested in covering as many cases as possible, but some particular cases of theoretical relevance to you--in which case coverage might be less of an issue. Still, as a researcher I would want to know why I could not cover the other cases, so my analysis would not stop there. There is also an approach to QCA that requires solution coverage of at least 0.75 (see the work by Michael Baumgartner), in order for a solution to be causally interpretable. But most applied QCA currently does not feature concrete thresholds for coverage sufficiency. As a general principle, I would say that the lower the solution coverage, the more our solution is mainly to be interpreted as a descriptive statement of the complex configurations that characterize the cases covered; and the less it can be interpreted more generally as the conditions that typically imply the outcome under question. We've demonstrably missed out on a lot of those.

I hope this helps, all best,

Eva

Dear Yimin,

These are two very good questions, particularly the first one.

QUESTION 1. In regression analysis, adding control variables is usually thought to reduce omitted variable bias, but this is not generally true. The amount of bias induced is a function of the pairwise correlations between your included variables and the excluded variables as well as their individual magnitude. Adding even relevant controls my thus make the problem worse (a good reference for this is Wooldridge 2002, pp.87-92 [or any later editions]).

In QCA, low consistency and coverage scores usually indicate a serious omitted-variables problem. However, the same logic as in regression analysis also applies: it may well be that you have low coverage, but your results may not be biased at all if the omitted relevant variables are not located on the same causal path to the outcome as the relevant variables that you have included (see more on this in Baumgartner and Thiem, 2017, p.8, and Thiem 2015, p.728).

My strategy to minimize this problem is the following one (I have already successfully implemented this in, for example, Haesebrouck and Thiem 2018, and I also employ it currently in other papers): using my QCApro package for R, I have a function which evaluates all possible factor combinations of your input variables and ranks the solutions according to their consistency / coverage / model complexity, finding the smallest set(s) of variables contained in all your input variables that achieve(s) your desired cut-offs (e.g., consistency higher 0.9, coverage higher 0.7, maximally 4 models per solution). Thus, I would advise you to implement a similar "data-mining" strategy (to me at least, data-mining is something potentially very useful as long as you know exactly what it does and how it works).

QUESTION 2. Your reviewers seem not to have understood the problem of omitted variables very well, neither in regression analysis nor in QCA. Again, you can have extremely low raw solution coverage (say 0.2), but still be perfectly correct in all your inferences. An example for this you find also in Baumgartner and Thiem (2017, pp.16-18, tests IIa and IIb). Our model has "only" 0.545 coverage, but all its inferences are correct because the omitted variable does not share a path with any included factor (we could have driven coverage down much further without changing this conclusion).

Of course, the non-existence of shared causal paths is nothing you can test or see. You simply have to introduce it as an assumption of your analysis, but as always, there's no free lunch in empirical research.

Best wishes,

Alrik

References

• Baumgartner, Michael, and Alrik Thiem. 2017. "Often Trusted but Never (Properly) Tested: Evaluating Qualitative Comparative Analysis." Sociological Methods & Research. Advance online publication. DOI: 10.1177/0049124117701487.
• Haesebrouck, Tim, and Alrik Thiem. 2018. "Burden Sharing in CSDP Military Operations." Defence and Peace Economics 29 (7):748-65.
• Thiem, Alrik. 2015. "Using Qualitative Comparative Analysis for Identifying Causal Chains in Configurational Data: A Methodological Commentary on Baumgartner and Epple (2014)." Sociological Methods & Research 44 (4):723-36.
• Wooldridge, Jeffrey M. 2002. Introductory Econometrics: A Modern Approach. 2nd ed. Mason: South-Western Cengage Learning.

Dear Prof. Thiem,

Thank you for your immediate reply every time.

I am trying to understand your explanation.

Yimin

Dear Yimin,

For example, if you have ten possible input variables, you may want to start with analyzing all combinations of five variables because one or several combinations may already exceed your cut-offs, and there's thus no need to include further variables.

E.g.: Your potential input variables are A, B, C, D, E, F, G, H, I, J; but the combination A, C, E, F and J already gives a solution with perfect consistency and coverage, and two models with several essential prime implicants. The same figures are generated with B, D, G, H and I, but this combination produces vastly more models with no essential prime implicants. So you may want to pick the first combination for your final analysis. If you automate this process, the risk for generating hardly interpretable results will be minimized and you will be able to pick the best-fitting combination of input variables.

Best wishes,

Alrik

Dear Yimin,

Regarding your first question about omitted variables, I found a much better explanation in the paper written by Radaelli and Wagemann (2018):

Article What did I leave out? Omitted variables in regression and qu...

They compare regression and QCA and present a range of diagnostics (and solutions) that I found very intuitive and easy to understand.

Regarding your second question, the more a solution covers the better but a large coverage is not necessarily the best thing. Schneider & Wagemann (2012) introduced the concept of "Relevance" (of necessity) and contributed an additional parameter of fit called RoN. This is by default being presented in the main package QCA from R, and also produced by the package SetMethods that is directly developed by Carsten Schneider.

You can also find an extensive explanation of the analysis of necessity and about relevance (and about how to perform those with R) in my recent book Dusa (2019), which has an online version as well (send a private message to get access).

I hope it helps,

Adrian

References:

• Dusa (2019) QCA with R. A Comprehensive Resource. Springer International Publishing.
• Radaelli and Wagemann (2018) "What did I leave out? Omitted variables in regression and qualitative comparative analysis", European Political Science.
• Schneider & Wagemann (2012) Set-Theoretic Methods for the Social Sciences: A Guide to Qualitative Comparative Analysis

Dear Yimin

1. When designing your QCA research, the best (if imperfect) way of trying to address the problem is to base your selection of conditions as much as possible on existing theoretical and substantive knowledge, so as to ensure you have included potentially relevant conditions. You can also combine several conditions into higher-order constructs, in order to keep the number of conditions at a manageable level (once you have conceptualized these conditions, this can be implemented with the compute() function in the QCA package based on calibrated component sets).

In case-oriented variants of QCA, a good way of identifying problems with omitted variable is via in-depth study of cases. On the one hand, cases that are not covered by the solution can tell you about alternative pathways to the outcome. On the other hand, in-depth study of typical cases and deviant cases for consistency can tell you about whether the conditions you included in the truth table actually provide a good explanation, or whether there is something you left out. I find the various works by Rohlfing and Schneider on set-theoretic multi-method research very useful in this regard. It can all be implemented via the SetMethods package by Nena Oana and Schneider.

Such insights can then feed back into a redefinition of the relevant explanatory framework in order to be able to explain more cases--that is part of QCA as an iterative approach.

Data mining might enable you to maximize coverage and consistency, it might not guarantee that the identified models adequately explain the cases and reflect relevant concepts and theories--these are eventually interpretations to be made using criteria external to the QCA algorithm and parameters of fit, such as theory, case knowledge, and additional methods of analysis. But these are more general issues in empirical social research, not really peculiar to QCA. A middle way between in-depth study of cases and simple data-mining can be to try out several equally plausible and meaningful options (for example, in terms of conditions considered, or in terms of options for conceptualizing, measuring, and/or calibrating the sets) considering the state of theoretical and substantive knowledge, and then compare the resulting models regarding their performance.

2. The coverage measure can be interpreted as % cases covered by the solution for crisp sets. It cannot be interpreted as such for fuzzy sets, so the analogy has clear limits there. But of course, it is about how much of the variation in the outcome we could describe.

Low coverage means that there is a large share of your empirics that is described or explained by other conditions than the one you have considered. So, it might be a good motivator to go back to point 1 and try and think of additional or different conditions to consider. There may be situations when you are not interested in covering as many cases as possible, but some particular cases of theoretical relevance to you--in which case coverage might be less of an issue. Still, as a researcher I would want to know why I could not cover the other cases, so my analysis would not stop there. There is also an approach to QCA that requires solution coverage of at least 0.75 (see the work by Michael Baumgartner), in order for a solution to be causally interpretable. But most applied QCA currently does not feature concrete thresholds for coverage sufficiency. As a general principle, I would say that the lower the solution coverage, the more our solution is mainly to be interpreted as a descriptive statement of the complex configurations that characterize the cases covered; and the less it can be interpreted more generally as the conditions that typically imply the outcome under question. We've demonstrably missed out on a lot of those.

I hope this helps, all best,

Eva

Four brief post-comments to Adrian's and Eva's replies:

• I don't find Radaelli and Wagemann (2018) very helpful for understanding the real mechanics of omitted variable bias. If you want to understand a statistical concept, rather than to merely think you understood it, you need to read a proper statistical/econometrical text (Wooldridge is quite reader-friendly and clear).
• Eva says that you should return to theory and your cases if you suspect problems with your research design, but that advice is hardly new, and hardly informative. It's usually the case that researchers have many more theories available than what can be meaningfully included in an analysis, provided they have done a thorough literature review and / or are year-long experts in their fields (I assume you are a year-long expert in your field, Yimin). A true lack of theory is something that occurs only in uncharted research fields, which rarely exist in the social sciences, or in research that has not sufficiently explored existing literature prior to running the analysis (often, pertinent theories cross-cut disciplines, such as in economics and psychology, or public administration and organization science).
• Eva then says that "Data mining might enable you to maximize coverage and consistency, it might not guarantee that the identified models adequately explain the cases and reflect relevant concepts and theories..." Of course, there's no guarantee anywhere in empirical research, but what do all good theories have in common? They fit the data very well, and data fit in QCA is measured via consistency and coverage. If a researcher presents a QCA solution whose data fit is not as good as another solution's, s/he should argue why the better-fitting solution is not the better explanation of the cases under study. Regularly, that proves not to be an easy task.
• Eva concludes with the following: "As a general principle, I would say that the lower the solution coverage, the more our solution is mainly to be interpreted as a descriptive statement of the complex configurations that characterize the cases covered; and the less it can be interpreted more generally as the conditions that typically imply the outcome under question." That statement raises more questions than it answers. If solution coverage is low, it simply means that only a small share of the instances of the outcome in the data can be explained by the prime implicants contained in the solution. For those instances, however, that is the explanation. It may be wrong because of omitted variable bias, but it may also be entirely correct under certain circumstances (see references given in my first reply above).

Best wishes,

Alrik

I have to disagree with Alrik Thiem and to endorse Eva Thomann's answer.

First and foremost, Yimin, I believe you should make your own opinion about Radaelli and Wagemann (2018). Diversity of opinions are welcome.

Second, QCA is anything but a statistical or econometric analysis. It is, as it can be extracted from Eva's answer, a case based method and returning to cases is the best advice one can get in QCA.

And especially since QCA is a case based method, which necessarily implies an usually small to moderately large sample, it prevents statistical analysis. That is known theory from the very beginning of QCA and I am highly surprised that Alrik Thiem suggests econometric analysis on small size data. What he describes is similar to data mining, which requires "big data" (millions of cases, which is definitely not the situation in QCA).

I have to also object with the definition of "data fit" in terms of consistency and coverage. Those are "parameters of fit" but they entail very little in terms of theory (or about how well the data fits theory). Ragin's (1987) original book specifically mentions the fit in terms of the intersection between a QCA solution and a theoretical expectation. You can find a function in the main package QCA called "modelFit" that does this kind of intersection.

I would be very cautious about data driven (data mining like) conclusions, especially with small sized data. Theory should be guiding your research, not the parameters of fit of a QCA solution obtained from a (sometimes very particular) set of cases.

Best,

Adrian

Dear Alrik, Adrian and Eva, thank you all for your discussion of my questions.

I have read your opinions and those related reading materials, including the article from Radaelli and Wagemann (2018).

For the omitted variables problem, I think Radaelli’s paper is more care about research design than technical fix. They provide some approaches to reduce this problem, but, as the number of conditions (often 4-6) is limited in QCA, this method will always been questioned as the omitted variables problem seems to be so obvious. Alrik’s strategy is data-based and attempts to find a technical scheme to reduce this problem, while Adrian and Eva give solutions that are more traditional but better understandable.

I think QCA is in its nature a qualitative analysis approach, with some quantitative techniques being used to make the causal relationships much clear.