Thank you Pieter for the hint and Levente for your elaborate contribution.

Levente, you may be right in stating that meta-analysis is simply a tool. But I would prefer a more complex tool as metaphor, probably a mowing machine (sorry for that joke).

I agree with a lot aspects you mentioned. But in my opinion there is a stronger relation between a method and its object (for example fever and fever-thermometer). And the objects of meta-analyses are primary studies, so their qualities (not only good or bad, rather meant as characteristics) are of relevance in meta-analysis, as you know. Therefor the characteristics of primary studies influence meta-analytical procedures and decisions and they are part of procedures, possible results and interpretations (In psychological tests we also assume that objectivity, reliability and validity are attributes of the test itself.).

I also agree with you that the use of this corpus of methods (meta-analyses) by different scientific communities is one of the central problems, especially in health care systems.

Reflecting the named problems, I think

1. that garbage-in-garbage-out-bias belongs to meta-analysis, because all applied meta-analyses make fuss about inclusion (This is an example for the association of method and object. Meta-analysis is more than pure statistics it at least must be seen also as decision making).

2. that apples-and-oranges-problem indeed implicates methodological heterogeneity. And with it some questions come up. In which fields, with which data-bases and with which measurements meta-analysis leads to reliable results.

Increasing abstraction and reductionism (see attached slides) in my opinion play a major role. Meta-analysis is always a comprehending tool and therefor it is inherent to ignore some information. It is hard to balance what is relevant and what is not. Therefor this has to be done in a very sophisticated way.

I absolutely agree with you that there are more or less informative meta-analyses and that the method itself should be used, performed and read carefully. Probably more elucidation about this method and training for readers is necessary.

With your comments the question can probably be meliorated:

• How reliable are meta-analyses in specific fields? Is there a maximum of possible reliability? (Reliability)

• What is the (general and specific) question meta-analysis gives an answer to (in general and in specific areas of research). (Validity)

• What conclusions can be drawn? (Validity)

• What are indication criteria for meta-analysis?

What do you think?

Regards Thomas

Here is my attachment.

Http://www.ncbi.nlm.nih.gov/pubmed/22262733

http://www.ncbi.nlm.nih.gov/pubmed/22241776

Dear Levente,

Thank you for interesting thoughts and answers.

• Reliability of meta-analysis

I agree with your statement that truth is relevant in discussing meta-analysis. Science in any case is built on truth as a fundamental concept (e.g. regulative idea). Therefor applying an epistemological perspective, and in consequence a methodological perspective on meta-analysis includes a concept of truth. Otherwise meaningful propositions are impossible. If we assume that in the case of meta-analysis results are context-specific every single meta-analysis would need a specific explanation or specific estimation to draw conclusions.

• Validity of meta-analysis

I do not totally agree with your statement. Drawing conclusions imply that there is something of interest that in the end can always be presented as hypothesis or as question. I agree with your statement ‘it is we who answer questions’. But methods are tools in doing so.

Validity is the crucial point to draw conclusions from or with support of meta-analyses.

• Indication of meta-analysis

Talking about indication implies questions. It seems to me that all of your ideas in this topic can be interpreted as questions we try to answer using meta-analysis.

The question if meta-analysis is rather a statistical method (your statement) or a statistic based research method (my opinion) is interesting but I am not sure about the consequence of both approaches.

My opinion is that it is more than a pure statistical procedure because it is full of decisions. But for both approaches the question is relevant ‘how to transfer empirical relations (second order in meta-analysis; findings) to numerical relations’.

It seems to me that one of your central assumptions is that this ‘complex procedure should be evaluated only in relations to the specific circumstances and aims that we are intended to follow’. I tend to call it the specificity-approach to meta-analysis. But we have to respect that specificity is rather relative. Does is mean specific to a field, to a concrete data set, to a question of interest? And respecting specific circumstances does not in the same time mean that there are no generalizable regularities in meta-analyses that can or should be discussed.

To draw useful (pragmatism) or meaningful conclusion (realism) by the use of meta-analysis means to clarify the context, the specific field of application, the data set, the possible biases and errors, reliability, validity as well as the logical structure of drawing conclusions … .

If these aspects are internal problems of meta-analysis or external depend on how we interpret meta-analysis only as statistical procedure or as research method. But in both cases the questions ‘What can be concluded?’, ‘What do the results mean?’ are of central interest. Not doing so opens the door to misinterpretation and arbitrariness.

Regards Thomas

Levente,

thank you for nice discussion and the great ideas in your answer. I like some of them, especially the ideas of locality and constructivism in the context of meta-analysis. Interpreting meta-analysis with this theoretical background would lead to a changed question: ‘How to interpret meta-analysis and the results of meta-analysis?’

I do not fully agree with your statement that misinterpretation is based on a position of being right. Being right or especially to know that something is right would imply a naïve realistic point of view. As we know from fallibilism and critical rationalism, it is easy to know that something is wrong but perhaps impossible to know that empirical proposals are right. And we have to take into account that most psychologists and readers of meta-analysis are rather realists. Therefor, if meta-analysis is constructivistic and its results are rather local, readers should know about this, and more instruction manuals would be necessary. This would correspond with your idea, that meta-analysis as pure statistical method is not the problem, but use, application and interpretation are.

Misinterpretation can also mean over-interpretation as for example the mereological fallacy teaches us. This fallacy happens when conclusions are drawn from (research on) parts and generalized to the whole or to attribute properties of parts to the whole and vice versa (e.g. from brain function to what persons do, from literature analysis to the whole field).

This fallacy probably happens, if results from meta-analyses are used to evaluate intervention strategies as it is often done in the field of evidence-based-medicine.

I like your idea of reframing misinterpretation as subjectivity, but to stress this concept too much would lead to an idiosyncratic or solipsistic trap because science should be able to communicate procedures and results.

I agree with your statement: ‘They both indicate diversity, uncontrollable variability, which is in my opinion an essential characteristic of a scientific enterprise that aims at improving people's lives.’ But knowing that and behave different, for example using meta-analysis to make normative decisions could also lead to severe problems.

Regards

Thomas

El metaanálisis tiene que ver con la cognición