Thanks Shafig. I meant to ask not so much the included studies and their sampling strategies, but rather the very protocol underpinning the systematic review itself. Is the protocol driven by multistage sampling?

Thanks for the link Shafig, regarding from there vis-a-vis sampling/ selection & aggregation protocol underpinning systematic review:

"how and where to look – search strategy; time span; what literature databases; what languages; what journals to hand search; searching of reference lists in review articles, etc.

• inclusion and exclusion criteria – what criteria will a paper have to meet for it to be included,

and vice versa excluded?"

Would you call all these steps an example of multistage sampling?

I've seen multistage sampling being referred as sampling strategy of this systematic review, http://www.wmich.edu/evalphd/wp-content/uploads/2010/05/A-Systematic-Review-of-Theory-Driven-Evaluation-Practice-from-1990-to-2009.pdf

So I am going to assumed that is the case, unless otherwise contradicted...

Harris Cooper (2010) describes a 7-step model for conducting systematic reviews in his book Research Synthesis and Meta-Analysis.

Yes Gary, I have looked at Cochrane, Campbell, and Prisma - which I take each to be Gold Standards in their field(s). As I understand SRs, there may not be some exact science in its methodology, but I thought with their emphasis on a priori protocol development, it is their whole point to make literature review a "science" (hence quantitative aggregation) rather than an "art" (like narrative literature reviews based on quantitative interpretations).

Of course, there are some who try to bring the two together* but the crust (as I understand it) is that sampling of literature is meant to be quantifiable. And there's where I am curious, even though not much has been highlighted, but whether there would be some explicit references to types of sampling strategies being deployed in the SRs' identification and selection processes.

* http://www.tandfonline.com/doi/abs/10.1080/13645570500155078#.U2CJ-PmSxx0

Toula, thanks for your suggestion, which led me to Cooper (2009)*, he talks about research collection procedures as: "Variation in searched sources and extraction procedures might lead to systematic differences in the retrieved research and what is known about each study" (p.9).

In Cooper (2010) previewed from Amazon, especially his third part, he goes to summarising the Cochrane and Campbell protocols, he covers some ground on key words used (natural-language terms versus controlled vocab codes such as classification codes or subject headings).

But despite his seemingly explicit position of SRs being a scientific process (Cooper, 2009), I must say I am abit surprised to see so far, the sampling strategies (underpinning the SRs' literature identification, selection, inclusion and exclusion processes) have not been given explicit analysis. For example, if SRs are scientific, are we talking about random based, non-purposive sampling strategies? Hence that's why protocols need to be a priori to the literature search?

* http://www.russellsage.org/sites/all/files/Cooper_Hedges_2d_Chap1_0.pdf

Thanks Gary, I guess this is the paradox regarding SRs' external validity which confuses me because of the seemingly tenuous relationship between representability/ generalisability and discretionary decisions about inclusion/ exclusion. Perhaps this places SRs closer to the qualitative spectrum than what most hardcore SRs exponents would disclose (from a sampling perspective).

If so, then there would not be a big chasm between SRs and narrative literature reviews. In turn, it may just substantiates one of my prof's position that SRs (and Meta-Ethnography) are really just literature reviews...

Excellent idea about posting the methodology on RG. Good to know that this is a REAL issue out there... thanks Gary.

I think when you are conducting an SR, SR approaches may be different between scientific disciplines, too. The pieces provided by Gary are usually what epidemiologists follow and the pieces offered by Tieu-Tieu are more to the social science. Pay attention to the subtleties between approaches. Devil's in the detail.

Thank you Gary Bourque.

You may be interested in this paper on conducting systematic reviews published by a colleague and myself in Higher Education Research and Development. It can also be accessed online, together with supporting videos etc., from Griffith University, Research Gate and Academia.edu:

http://www.tandfonline.com/doi/abs/10.1080/07294360.2013.841651

Sincerely - Jason

Thanks Jason, I've just glanced through your paper [so please excuse any shallow interpretation henceforth]. It is a great paper regarding its relevance to above discussions.

I agree with you about the rationale of SRs for rookies like me (p. 4 - 5), particularly being in the Education domain, it is very difficult to get access for onsite research [at least in Oz, so it seems unlike in Asia].

Here are some points I am curious about/ need further clarification:

1. Do you regard SRs as both literature review [text{s} in context] and research method/ methodology [tool/ instrument]?

Minimising bias seems to the key differentiator between narrative and systematic reviews: "An important limitation that applies to all literature review methods is how to deal with potential biases in searching for relevant literature. The use of systematic methods can minimise such biases, but will not completely eliminate them" (p. 13).

As I understand it, the whole point of SRs is to aggregate "unbiased" evidence for a host of different applications, policies, etc. So in that sense, SRs are a research method/ methodology [depending if you are Crotty, Huberman, Creswell, etc.]. In short, it may be regarded as an objective tool for data collection and analysis, hence all that fuss with protocols.

Then on the other hand, if SRs are to function as literature reviews, then their function would be to situate the research's contextual elements. Like in narrative literature review, which is explicitly there to contextualise one's contribution within [one's retelling of] the broader research narrative.

So does this brings us back to the duality that's been raging about research methodology as quantitative, objective, aggregative, a priori tools versus qualitative, subjective, interpretative, and reiterative discourses?

And if one were to have a foot on both rocky boats [to use a Chinese metaphor for "double timers/ infidels"], would this somehow mean the reformulation of SRs explicitly as mixed methods [which is what Harden & Thomas (2007)* have started to do, though I am not fully convince that you could do both simultaneously AND separately].

2. On the process component regarding your excellent paper, which I like enormously for its clarity with the steps (p. 6), it sort of reminds me abit more like theoretical sampling [though the reiterations among theoretical revisions, data collection and analysis are not so intertwined as qualitative studies]:

"In step 6, particularly with structuring of database: "Before proceeding further, it is worth estimating the number of relevant original research papers. A quantitative review may not be required if there are very few papers (15 or less) or may become unwieldy if there are too many (∼>300). In either case, it might be necessary to expand or narrow the topic under review or to undertake a different type of review (narrative if too few papers and meta-analysis if sufficient datasets are found). ...The student should then develop the structure for their own personal database on the topic (Step 6). ...Structuring this personal database involves selecting and defining categories and subcategories of data to be populated with information about each paper found through the electronic searches. This step may also necessitate revising the topic and questions to be addressed, so appropriate categories and subcategories are used."

Here I suppose sample size concerns the issue of generalisability, and thus reliability and validity [which of course is all linked back to bias]. So it seems to me that you cast a net as wide as possible [to keep up with my boating metaphor], you hope you could catch as many fish that there are to represent your designated population. Depending where you fish, the characteristics of your net, boat/ ship, and crew, you come up with 15 or 300 more, some of which may be relevant to you, some not... [and so we are now down to sampling frame], you decide that you keep ones longer than 30cm... [sometimes you might have a ruler which some might call SRs or meta-analysis], otherwise, you're down to your rule of thumb/ foot [for the nonMetric Imperials].

With this rule of thumb/ foot, it may not be fast like some volume-based filters [NVivo, Anova, Manova, SPSS, Excel, etc.], but you might pick up some thing about the fish which the filters just "blindly" sort... but what you see depends on your resources [time, research grant size = number of assistants] and most importantly, your "seeing" power, your expert capacity to recognise salient points... that's when I would say you BECOME the tool, or the tool BECOMES you [via Heidegger or Zhuang Zi]**.

In other words, it like your reference to the evaluation prowess of experts (p. 13): "Consequently, determining the criteria used to evaluate/weight studies [i.e the experts' "seeing" ability often unavailable to rookies] often requires detailed and diverse expertise; reviews using these types of criteria are typically undertaken by teams of experts collaborating over long periods (Petticrew & Roberts, 2006). Also, despite the prevalence of such weighting methods [i.e. rule of thumb/ foot] in some disciplines, there is still considerable divergence in how they are used, including which criteria [i.e. 30 cms or longer Sharks - going on in Western Australia] are used and how they are applied (Petticrew & Roberts, 2006). Finally, generally accepted weighting criteria are not available for many disciplines and are often inappropriate to use when reviewing trans-disciplinary research. Therefore, we feel that where the focus of the review is to map the breadth of the literature [i.e. SRs as street maps rather an topography, but ultimately as text(s) in contexts/ discourse?] rather than focus on evaluating its depth [i.e. SRs as meta-evaluations, experiments, or quasi-experiments?], the systematic quantitative review has logistical and methodological advantages, particularly for PhD students [i.e. are they expert enough to "see"?], over approaches that require studies be weighted."

This of course brings us back to validity, reliability, generalisability, objectivity, and subjectivity (p.g 13): "There are mathematical modelling techniques [i.e. the volume-based filters from above] that can be used to try to estimate and deal with the ‘positive’ research bias, although these modelling techniques can only be applied to certain types of methods and disciplines (Petticrew & Roberts, 2006). Finally, journals [i.e. where one fish] specifically for ‘negative’ results have been established to help to deal with this issue in some disciplines (Petticrew & Roberts, 2006). Despite this, for many literature review topics, this bias [i.e. the oceans are wide and deep, one can only capture and see so much and so far...] remains a major challenge with few easy solutions. Therefore, as with all types of literature review, students should acknowledge these limitations in their findings."

3. So when we are talking about limitations here, is it because we are hitting the wall [the boats are moving in opposing directions] when it comes to assumptions of SRs as objective instruments or ones subjectively embedded in the ultimate research tool of them all - our own idiosyncratic but more seeing/ sensing brain/ mind [subjected to each individual's strengths and weaknesses]?

And so sampling studies for SRs strike me as Zhuang Zi's butterfly - which might just pretty much sum up our existential dilemmas as researchers... [sorry, hadn't expected to come down this path...].

* http://www.tandfonline.com/doi/abs/10.1080/07294360.2013.841651#.U2Lx7vmSxx0

** http://en.wikipedia.org/wiki/Zhuangzi

More fish [Unsensibly not from Douglas Adams, http://en.wikipedia.org/wiki/So_Long,_and_Thanks_for_All_the_Fish ],

but from Zhuangzi (Wikipedia):

"Zhuangzi and Huizi were strolling along the dam of the Hao Waterfall when

Zhuangzi said, "See how the minnows come out and dart around where they please! That's what fish really enjoy!"

Huizi said, "You're not a fish — how do you know what fish enjoy?"

Zhuangzi said, "You're not me, so how do you know I don't know what fish enjoy?"

Huizi said, "I'm not you, so I certainly don't know what you know. On the other hand, you're certainly not a fish — so that still proves you don't know what fish enjoy!"

Zhuangzi said, "Let's go back to your original question, please. You asked me how I know what fish enjoy — so you already knew I knew it when you asked the question. I know it by standing here beside the Hao."

(17, tr. Watson 1968:188-9, romanization changed to pinyin)

The traditional interpretation of this "Daoist staple", writes Chad Hansen (2003:145), is a "humorous miscommunication between a mystic and a logician". The encounter also outlines part of the Daoist practice of observing and learning from the natural world." (Wikipedia)

*****************************************************************************************

4. Are limitations euphemistically called miscommunications/ inability to share even the same boat?

Again about Zhuang Zi from above Wiki: "To use the limited to pursue the unlimited, he said, was foolish. Our language and cognition in general presuppose a dao to which each of us is committed by our separate past—our paths. Consequently, we should be aware that our most carefully considered conclusions might seem misguided had we experienced a different past. Zhuangzi argues that in addition to experience, our natural dispositions are combined with acquired ones—including dispositions to use names of things, to approve/disapprove based on those names and to act in accordance to the embodied standards. Thinking about and choosing our next step down our dao or path is conditioned by this unique set of natural acquisitions."

Thanks for reading this far!

All due respect Jason and colleagues. :-)

Maybe I should add one more note that the difference between Zhuang Zi and us now is the availability of "meta-brain"/ computing power.

But how objective/ not bias such a meta-brain/ singularity/ your regular PC are big philosophical/ ontological/ epistemological questions/ limitations; - ones from which, I suspect, not even research minnows like me could really escape...

Phew. How many coffees have you had today? Your mind is racing!

Some great ideas in here. Unfortunately I am very limited for time, but would love to be able to discuss these ideas further.

We have had a go at operationalising our method. You can find two reviews that we have published using this technique here:

http://www.sciencedirect.com/science/article/pii/S1618866712000829

on urban trees and

http://www.sciencedirect.com/science/article/pii/S1618866712000830

community gardens.

Colleagues have also published reviews using this technique for sharks and birds:

http://www.sciencedirect.com/science/article/pii/S0301479711001411

Here is a link to some video resources:

http://www.griffith.edu.au/environment-planning-architecture/griffith-school-environment/research/systematic-quantitative-literature-review

As you note - there is going to be some ongoing debate about whether systematic reviews are a method or as you suggest 'texts in context'. For us, we are using the technique as a method. For the meta-analysis type reviews - they are probably a form of analysis. Some may also regard a systemic review as texts in context - I can't argue with that.

For us the most important reason to use the technique we have devised is to be able to rapidly assess the literature to identify strengths, limitations and gaps. A big limitation which a student of mine is currently working through is that of selecting only English-language publications. He is doing a systematic review in German, Japanese and English (he is trilingual) and he has found that we have to be aware of the limitations imposed by language. Such reviews would be much stronger if they included Chinese publications, Spanish publications and other languages too.

I agree about the dilemmas of working out when to stop casting the net. It is a tough one. For me it is a question of saturation - are you returning the same papers in different search engines? And a law of diminishing returns - if you expended 30% more time how many more papers would you find? At some point you will need to stop and get on with analysis and writing, so it is important to have a clear rationale about where to draw the line.

I am intrigued about your idea for 'volume based filters'.

Finally - there are some excellent annotated bibliography resources out there - such as Oxford Bibliographies Online, which are also systematic - and also narrative. See:

http://www.oxfordbibliographies.com/obo/page/geography

Best wishes - Jason

No coffee, I am afraid - only worst, a can of Diet coke! Still Jason, thanks for your clarifications above. I am very glad to see that:

1. You're based in Oz; and

2. You're an urban geographer.

Which might mean regarding [1] I can knock on your door sometime (though I suspect like all academics here, you are time-pressed). And [2], I suspect there is lots to explore with SRs as "maps" as you put it.

Still it was very useful to know that you regard SRs as method rather than "text(s) in context". Obviously there's alot at stake in terms of generalisability depending on which boat(s) you have your foot/ feet in.

Dear Tieu-Tieu, feel free to follow up with me - I am just up the road really (only 1,000 km away)! Catherine Pickering g runs regular courses on our technique - so feel free to follow up with her too: [email protected]

She is an ecologist but has learned that social scientists, though odd, have lots to offer.

Cheers - Jason

Thank you very much Jason. Will definitely look into all that.

Excuse me! Social scientists are odd! The training I have received from eminent medical scientists and social scientists definitely put SR/s into method rather than contextual text. I conducted a qualitative review and used the same method and rigour as a meta-analysis. The analysis revealed constructs that had not been identified in SRs of quantitative studies and provided context for the constructs, which is lacking in meta-analyses. Therefore, both types of SRs can be complementary as argued by Thomas and Harden. I look forward to catching up with all the information from Jason. All research, even randomised controlled trials, is biased, so it is a matter of recognising the fact in SRs, however, it is worrying that organisations that promote and engage in SRs are demanding higher and more detailed coverage of this, rather than focussing on the results gained from synthesising many studies!

Sorry Stephanie - no offence intended. I am a social scientist myself. It was just a bit of tongue in cheek humour. Regards - Jason

OK, taken it as such!

To Jason's defence, I suspect not only is he a social scientist [qualitative - quantitative oxymoron?], but that he's been in Oz for too long [says she, the touch-in-cheek Aussie ;-) ].

But on a serious front, here's a distillation [perhaps oversimplified] on how data is to be regarded in SRs and Meta-Ethnography:

1. Data extraction [often viewed as method] = relatively neutral "primary" data [quantitative, aggregative]

2. Theoretical sampling [often viewed as texts in contexts; discourses] = subjective "secondary data" [qualitative, interpretative, if it can be called data at all.]

But my suspicion is still with data extraction - sort of as if one is just disposing the "shell" [subjective textual "interpretations"] from the "nut" [pure data] and thereby rendering the data "pure" [relatively unbiased, reliable and valid "evidence"].

This may also explain why it is difficult to locate SRs which declare explicitly how they go about "extracting" random/ probability based samples [could you call extraction parametric sampling?].

I must say I am not fully convinced about the "purity" of the extracted data because there are post-priori reiterations/ changes after the protocol development. Would that not invariably introduce some interpretative elements?

Similarly when one is using studies drawn from multiple languages. The interpretation and translation work - is fluency [determined by whom] enough to secure reliability and validity? Irrespective how well-versed one is, is that not STILL interpretation and translation? What then gets lost [not just Bill Murray, though Scarlet is okay ;-) ]?

Or the "expert panels" in SRs whose judgements are considered benchmarks for reliability and validity. So expertise, once regular interpretations in qualitative kind, gets suddenly elevated into "neutral" / non-biased positions because one "knows" or "sees" better than academic minnows [say like me :-) ].

Moreover, it is difficult to locate literature which comes outright and admit that SRs still rely on secondary analysis [however quantifiable they go about it, everybody knows, numbers can lie and also do other dirty things texts do...], thus goes back to "the denuded nut analogy" as "pure data"/ evidence-based.

Obviously Stephanie, as you have made explicit ("All research, even randomised controlled trials, is biased, ...the results gained from synthesising many studies!), the mediated, constructed, intertextual ["text(s) in context(s)] and discursive elements of SRs still prevail... and thus long live the paradigm wars!

Or go the way with Thomas and Harden, and declare a truce via Mixed Methods.

Though on that front, I am not fully convinced by their proposed simultaneous AND separate deployment of quantitative and qualitative research methods. [But that's for another caffeine fueled session... which this one is not... ].

Hi Tieu Tieu, in my limited experience SRs do not rely on theoretical sampling. By construction an inclusion/exclusion criteria to locate studies to answer a specific question requires all relevant studies to be included in the review. This method can be pragmatic to include a broad range of studies or very tight, thus limiting studies that specifically answer the question. There is so much published research that it is difficult for any researcher to cover many subjects fully, therefore SRs are powerful, even with the inherent biases. There is no need for "paradigm wars", all researchers should consider how best to answer their research question and use the appropriate ethical method for that purpose rather than trying to fit a square peg in a round hole! By so doing the findings will be more comprehensive to allow researchers and therapists to find and use better interventions to solve the many human crises that exist.

Yes Stephanie - I was using theoretical sampling [in Meta-Ethnography] as contrast to data extraction [in SRs]. Ultimately, I suppose the pragmatics ["what ever goes"] would subsume objectivist and constructivist ontologies/ epistemologies in SRs [as you have highlighted in "pragmatic to include a broad range of studies or very tight"]. Pardon me if I have then taken some liberty to assert that this puts SRs in mixed methods "territories".

I did start with the question at the beginning, hoping for some objectivist stance on sampling in SRs. I was hoping for some scientific credibility to justify SRs as method rather than literature review [which has been the stance of my quant-trained prof].

But instead the closer I look, the more I find the prevailing SRs [as method] seemingly on tenuous grounds [for reasons mentioned above in my previous responses].

Ultimately, as I understand it, one would need to revisit the quant-qual [paradigms if you will] debates should one wishes to generalise whether the "sampled" studies are "evidence" or discourses [hence the paradigm wars if you will].

If that means SRs need to be problematised for their prevailing objectivist stance, I suppose you could charge me of fitting "a square peg in a round hole". I would sooner prove my prof wrong. But maybe he has a point afterall...

As the Methodology Topic seems to be out of order, I post the following comments here asking for hospitality.

The quantitative versus qualitative debate has taken place within our RG research team and significant steps towards a reconciliation of initial differences is taking place. As a means of introduction of the following considerations, I recall to have noticed quite a synthetic but effective message on our task of research designers:

As a starting observation I’d like to point out that the point at which the data analysis begins and ends depends on the type of data collected, which in turn depends on the sample size, which in turn depends on the research design, which in turn depends on the purpose.

Nevertheless, far more insidious discrimination remains. Systematic review methodology exhibits all the characteristics of "institutionalised quantitativism" in that criteria for a "good" review are almost entirely determined by the quantitative methods promoted. Nobody who understands qualitative research would insist that its primary studies demonstrate alien concepts such as "sample size" or "statistical power". Yet comparably fundamental incongruities persist with regard to qualitative syntheses. Why should systematic reviewers of qualitative research pursue a "gold standard" comprehensive literature search when concepts such as "data saturation" have an established pedigree? Why shouldn't they apply systematic, explicit and reproducible principles of thematic or concept analysis to create syntheses that advance our understanding of qualitative issues and highlight research gaps?

Due significance should be given to the methodological options at our disposal reckoned to be more appropriate in the different circumstances dependent on the nature of the study. Accordingly, we should make choices of methods that are both philosophically defensible and, at the same time, practicable and responsive. For instance, taking the case of researchers becoming aware that the purposes of their study often involve both quantitative and qualitative aspects, it follows that it will be more appropriate to explore the opportunity of developing mixed method research designs that may be better associated with their investigation purposes.

Then, following the ideal thread of earlier observations, I choose to concentrate on the ‘mixed methods’ methodological approach employed as a research configuration in social sciences at large especially in cases of integration or connections of quantitative and qualitative data.

On the first approach I observe that an increasing development of mixed methods research is commonly accepted as a distinctive feature of contemporary research designs to profit from the inclusion of both quantitative and qualitative sources of information, mostly when generalization of results and feedback evaluations are the purposes pursued. In fact, as it has been remarked, the insertion of qualitative data can help researchers to enlighten relationships emerging from quantitative data. Similarly, the inclusion of quantitative data can help in compensating for the fact that qualitative data normally cannot be generalized.

According to the prevailing literature on mixed research design, combining quantitative and qualitative analyses has been advocated when the process presents evident complementary strengths as in what Denzin (1978) dubbed triangulation of different data source, i.e. the process of testing the consistency of findings obtained through different instruments for the study of the same occurrence. Specifically, the combination of the two approaches seems useful when:”

1) results from qualitative interviews can help to identify unobserved heterogeneity in quantitative data as well previously unknown explaining variables and misspecified models;

2) results from the qualitative part of mixed methods design can help to understand previously incomprehensible statistical findings;

3) qualitative research can help to discover quality problems of quantitative measurement instruments;

4) and quantitative research can be used to examine the scope of results from a qualitative study and support the transfer of such findings to other domains” (Kelle,2005).

For its characteristics, mixed method research has been labeled: the third major research paradigm and gained the legitimacy of being a stand-alone research design. However, besides its strengths, controversial issues still hinder its potential and some of the details remain to be worked out by research methodologists (e.g., problems of paradigm mixing, how to qualitatively analyze quantitative data, how to interpret conflicting results). Then, a word of caution seems appropriate to be addressed to researchers (and better to research teams) who plan to design inclusive, complementary methods that are capable to embrace diverse perspectives, data and values within and across studies.

References:

Denzin, N. K., “Triangulation”. In N. K. Denzin (ed.), The research act: An introduction to sociological methods, 1978, McGraw-Hill, New York

U. Kelle, Mixed Methods as a Means to Overcome Methodological Limitations of Qualitative and Quantitative Research, Workshop on mixed-methods held on October 26-27th 2005 at the University of Manchester.

Thank Giannrocco [what a wonderful name - Barouque John in translation? :-) ]

There's alot there, certainly if not, a lot caffeine consumed? :-)

You have certainly given a big chunkful, of which I must first go away and regulate my own cognitive load explosion/ implosion before resurfacing... meanwhile, thanks for joining the conversation.

A comprehensive argument Giannrocco, however, how many interviews or studies does it take to gain data saturation? How can you be sure no other findings will emerge with further interviews? It makes the same assumptions as quantitative research and adds bias by selecting certain studies!

If it is the case that no further findings will emerge after 12 or 15 interviews, surely it can be argued that the findings are transferable or generalizable?

By including all studies that are available to answer a specific question, this problem does not arise. Having a comprehensive search and approach to study selection provide a complete view of the literature. As Hannes and Lockwood state " SRs are not just about gaining new insights from the pooling and interpretation of two or more studies; they should also be part of the process which informs primary research. A more comprehensive approach to searching and study selection furnishes the review with a strong platform from which to make comment about a given research field, its methodological strengths and weaknesses and where gaps in the literature exist". ( Synthesizing Qualitative Research, 2012, p. 167).

My own experience has shown that a comprehensive qualitative SR can reveal previously unconsidered factors, similarly, with Thomas and Harden.

With some deadlines hanging over me, this will be brief...

I see that:

1. SR is now being used interchangeably with qualitative or quantitative methods [for example, from Stephanie: "comprehensive qualitative SR"].

2. Real world experiences indicate that there have been epistemological and ontological crises among research teams trained in qualitative/ quantitative methods [thank you Gianricco].

3. Reconciliations between these two camps invariably involve mixed methods [again Gianricco].

To be sure, SRs [quantitative-driven] and their qualitative cousins [Meta-Synthesis, Meta-Ethnography and Qualitative Synthesis] - so it seems to me, are increasingly occupying the SAME spectrum rather than existing as binaries [thus my own radical departure from Thomas and Arden, interestingly, Petticrew has also collaborated in some of their joint research].

At the end of the day, they both provide an overview in ways that many other research methodologies cannot/ do not.

If the researchers had gone down the SR path, this overview would be articulated in more manifest ratios and percentages. For example:

http://www.sciencedirect.com/science/article/pii/S1618866712000829

In contrast, if the researchers had gone down the qualitative path, the overview entails making salient latent themes. For example:

http://www.sciencedirect.com/science/article/pii/S0360131511002533

These two papers are both excellent examples of revealing "previously unconsidered factors" [thank you Stephanie]. But beyond that, I must admit, I would hard-pressed to find other more substantial differences.

And this brings back to me all the above issues raised about SRs. For example, questions such as whether these are evidence or discourse? Or perhaps increasingly, maybe as a reconciliatory strategy, you could have discursive evidence...

When I ask a "pure" natural scientist what he would consider the differences between evidence and "dialogue", without hesitation, he said: "Predictability and repeatability".

No doubt, in SRs, if done well and systematic, I suppose you could repeat many things. But there are others aspects that make them discursive [texts in contexts, etc].

Namely if you were to change the context [e.g., different individual experts, or interpreters and translators], the texts/ sampled studies/ dialogues would also change.

Hence increasingly to me, such potential ambiguities in SRs [from population, sampling frame, sampling size, representativeness, generalisations, nature of data, biases, etc], the corpus of literature which best engages with such issues is found in the mixed methods debates [Gianricco's thread being a great example].

Sorry, I did say brief, didn't I? :-)