The moment you call your review a "systematic review"- then you have to adhere to the systematic review standards for gathering, analyzing and reporting evidence; which include two or more people doing the study selection and data extraction to ensure an objective and transparent approach for research synthesis and reduce bias. I think the "systematic" nature of a systematic review is actually in the process- and this is what sets it apart from other literature reviews. If your review is just a literature review or a scoping review and you have identified it as such, then you do not need multiple coders or data extractors.

This is very good question. I think it should not be any problem if you have few or single author(s) in meta-analysis. I have recently written 2 meta-analysis articles (one published and one in review) where I personally did data visualization, harmonization and integration. However, for the first article which is now published, I agreed to provide co-authorship to people who gave me access to raw data. That was the part of an agreement to reduce cost. However, in the second meta-analysis there were only 3 authors (including me), and I think that would not be a problem in review. I know at-least one very good meta-analysis in forestry where the article was authored by one researcher. You can use that as an argument if reviewer ask you this question. Therefore, in my personal opinion, number of authors should not be an issue to publish a good meta-analysis.

http://www.sciencedirect.com/science/article/pii/S0378112712004264

http://www.sciencedirect.com/science/article/pii/S0378112707007360

Superb answer, thank you very much for your experience and for sharing those resources.

I don't agree with the previous answer, because there is experimental evidence that shows that the use of two or more independent screeners/coders results in substantial reductions in errors in systematic reviews.

See: Buscemi, N., Hartling, L., Vandermeer, B., Tjosvold, L., & Klassen, T. P. (2005). Single data extraction generated more errors than double data extraction in systematic reviews. Journal of Clinical Epidemiology, 59, 697–703.

For this reason, the Cochrane and Campbell Collaborations require multi-authored reviews.

I do agree with Ms. Littell! Even I did not know that this is such a big issue in clinical review. Thanks for sharing these information. I spent lot of time in research synthesis in Ecology and Forestry but never came to know on this particular topic. May be my previous answer fits more in these fields. But again, I want to stress that it depends on fund, time and kind of data to be used in meta-analysis. Clinical meta-analysis under umbrella of the Cochrane Collaboration may use n number of coders based on the funds and needs, and these will definitely reduce the error. But, comprehensive research and database network such as Cochrane Collaborations is not presently existed in many other fields of research or currently in a preliminary phase. I think a researcher needs to decide about trade-offs (multiple coders vs. single coders) prior to do the meta-analysis. It also depends on the sensitivity of research question and amount of acceptable error. In my opinion, if a researcher can digitally able to extract data (by using data extraction software) then accuracy can be improved. But there should not be any prior rule that makes multiple human coding as "compulsory" and "standard" method for a successful research synthesis across all fields of research (e.g. sociology, ecology, education or psychology where research funding often not match with clinical meta-analysis). Rather, it should depend on research questions and dimension of research. If a young graduate student want to do a meta-analysis in his/her introduction section of thesis, the research should not be rejected from publication simply because he/she could not able to employ multiple coders. Nevertheless, a researcher (e.g. lonely coder) can perform sophisticated resampling tests ( e.g. Bayesian confidence interval, bias corrected confidence interval etc.) at the last step of meta-analysis to increase robustness.

Well, when empirical evidence suggests that multiple coders result in reductions in error (and no empirical evidence suggests that single coders are as accurate as multiple coders), I think it is fair to say that "evidence-based" practice means that one should always use multiple coders unless it can be shown that this is unnecessary in a particular review. To show that multiple coding are unnecessary, reviewers can test initial agreement (with Cohen's kappa) and use single coding only when double coding is shown (empirically) to be unnecessary. This has nothing to do with the substantive area (field of inquiry), nature of the search process, or amount of resources available; it's about the reliability and validity of the review. If resources are scarce, reviewers can often find "free" (no cost) labor in exchange for co-authorship. I have two doctoral students who are presently doing unfunded Cochrane reviews; both have acquired free coding in exchange for co-authorship. The burden of proof is on reviewers to show that they have made every attempt to minimize bias and error at each step in the review process. The burden of proof is on reviewers to show that the use of a single coder does not diminish accuracy. The example of direct (digital) extraction of data is a good one, but that is rare.

Good topic!

Well indeed this was the kind of debate I was hoping we could have here. The problem I have with the issue is that I absolutely know that multiple coders is better. Not just in terms of improved reliability in the statistical sense but also in terms of trustworthiness. And I can see why this is considered absolutely essential in medical research - systematic reviews there save lives, and bad ones may cost them. However, the practicality of simply getting another author to do it for free is also variable depending on the size of the review - pretty easy if you're doing a meta-analysis of a specific intervention - a bit harder if you're asking for a few months full time work in a scoping review, for example.

This reminds me of another debate I often have with medics in systematic review. The purists say only RCT's should ever be combined whereas I'm more of a view that the idea is the synthesis the best available evidence - if no RCT's exist or can exist, does this mean that field of research is worthless? Its the same problem I have here. I know ideally I'd have everything double or triple extracted, but given that is not feasible is the work of one reviewer better than not reviewing at all? My current thinking (and why I'm embarking on the project) is that the work will be sub optimally reliable but by following the procedures of systematic review rather than traditional narrative review at the very least the biases and potential for errors will be made explicit for others to critique or build upon as they see fit. I just thought I'd put that line of thinking out to the community so they can tear it to shreds if required before I progressed too far!

The moment you call your review a "systematic review"- then you have to adhere to the systematic review standards for gathering, analyzing and reporting evidence; which include two or more people doing the study selection and data extraction to ensure an objective and transparent approach for research synthesis and reduce bias. I think the "systematic" nature of a systematic review is actually in the process- and this is what sets it apart from other literature reviews. If your review is just a literature review or a scoping review and you have identified it as such, then you do not need multiple coders or data extractors.

I think that is a fair response in my case. I think the more general question may remain though. Could a researcher with limited resources engage in all procedures typical of a systematic review or meta-analysis apart from double extraction/coding and still claim to have conducted a systematic review? My feeling is that they have - just a flawed one. But not necessarily a critically flawed one. In the ref cited above for example while there was a significant difference in reliability of coding there was no significant difference in the effect size estimates produced. As I say, I think the costs to benefits ratio of double coding may not necessarily always be so clear cut outside of medicine. It might be interesting to approach the problem from an economic perspective actually.

The problem is in calling a review "a systematic review" when you have not followed the systematic review methods. It is like any other research methodology you can think of. In qualitative research for example, you can only say you used grouded theory if you actually used grounded theory. In quantitative research you can only say you conducted an RCT if indeed your study was an RCT. If you can't do a systematic review because of financial constrains then do the next best thing that is within your budget and call it exactly what it is. The problem is calling it a systematic review when it is not (you did not follow the methods or processes of doing a systematic review).

Well that I can agree with but I'm not so convinced double coding is as canon as you say - Little mention in PRISMA ("Regardless, we advise authors to tell readers who extracted what data, whether any extractions were completed in duplicate, and, if so, whether duplicate abstraction was done independently and how disagreements were resolved"), CRD only recommends it to avoid missing about 9% of studies as I recall. Moreover I think even in these documents there are debates to be had. PRISMA states "at least one" database should be searched for example. I think I'd have a hard time calling something a systematic review if they searched only one database. I dont think the specific procedures for what is and is not a systematic review are quite as cast in stone as anyone would like. I've always considered double extraction "gold standard" not necessarily a key part of the process. More than two coders is better than two coders after all. We all have to draw the line on feasibility somewhere, I'm just proposing that in areas where cash flows less freely double extraction/screening may be one of the more reasonable areas to permit compromise. I also think that if a study is done with a reproducible search, clear inclusion and exclusion criteria, and a summary of the results of all studies with risk of bias are presented and open to scrutiny then it probably deserves the title "systematic review" or at least some recognition that it is more than a traditional literature review.

Still, in principle I'm with you - certainly I think some effort should be made to double extract and where this is impossible the increased risk of bias acknowledged. A meta-analysis which examined whether or not single extracted studies exaggerated effect sizes or not would be wonderful. If anyone knows of an existing one do pass it on to me, and if it doesn't exist someone could do the world a favour by doing that study I think!

I agree with Noreen. The procedures for systematic reviews were created to minimize error and bias. We have 400 years of writing on confirmation bias, all pointing toward the human tendency to adopt information that confirms our prior opinions and ignore evidence to the contrary. Scientific methods (and that included systematic review methods) are designed to minimize this and other sources of bias and error. A single authored review is not good practice and is largely unnecessary (it is easier to get free help than you may think). As to the canon...

Double extraction is *required* in the US Institute of Medicine's standards for systematic reviews (www.iom.edu). Cochrane's minimal standards state that double coding is *mandatory* for eligibility decisions and extracting outcome data, and *highly desirable* for all other study characteristics.

Chandler, J., Churchill, R., Higgins, J., Lasserson, T., & Tovey, D. (2012). Methodological standards for the conduct of new Cochrane Intervention Reviews (Version 2.2). Retrieved from http://www.editorial-unit.cochrane.org/mecir

I have worked with CRD on three systematic reviews and they are actually very strict on following systematic review principles. In their guidance for conducting systematic reviews (in health care) the CRD actually specifies that the accepted minimum is one researcher extracting the data and a second independently checking the data extraction forms for accuracy (although if you read the Cochrane handbook I think it might say this is not enough). Of course the PRISMA would say what you quote above because it is not about standards of CONDUCTING a systematic review but about standards of REPORTING systematic reviews (Preferred Reporting Items for Systematic Reviews and Meta-Analyses)- telling the reader exactly what you did, that is the message.

I am not sure what the compromises you propose will serve really. There are other review methods that one can use which are not systematic review methods, so if one cannot do a systematic review then just use the other review method that you can. I have been in similar situations where a systematic review was not possible and I just did a literature review. Like I said earlier, it is the same as other methodologies- if you can't do an RCT because of financial, logistic or even operational realities, then do a 'quasi-experimental' study using another method which is feasible under your circumstances. But you can not then say I did an RCT when you didn't.

Try the following Refs (just copied and pasted from the Cochrane Handbook- I hope they will be useful):

Buscemi N, Hartling L, Vandermeer B, Tjosvold L, Klassen TP. Single data extraction generated more errors than double data extraction in systematic reviews. Journal of Clinical Epidemiology 2006; 59: 697-703.

Gøtzsche PC, Hróbjartsson A, Maric K, Tendal B. Data extraction errors in meta-analyses that use standardized mean differences. JAMA 2007; 298: 430-437.

Jones AP, Remmington T, Williamson PR, Ashby D, Smyth RL. High prevalence but low impact of data extraction and reporting errors were found in Cochrane systematic reviews. Journal of Clinical Epidemiology 2005; 58: 741-742.

Your responses passed me by, sorry to ignore you! I don't disagree with either of you I must say. I just wonder what we should call a review that has performed all the steps of a systematic review bar double extraction. It strikes me that this would still be more valuable than a traditional literature review in terms of transparency and I'm not sure if we should discourage it (Again the two ref's now cited dealing with the issue have found a greater number of errors but not to the extent effect size estimates are meaningfully different). Even if we do decide we won't call it a systematic review in order to keep that methodology pure - the approach is after all only recently something which has become more concrete, at least in medicine. Still, I think we've probably taken the debate as far as it'll go and I'm pretty sure the consensus lies with your position more than mine! And I absolutely agree that in most systematic reviews the number of papers is small enough that obtaining free help is not difficult.

I agree that systematic reviews should go by set rules discussed above which are put in place to reduce bias.

What do you think about "Systematic reviews authored by a single author, who also happens to a key-opinion leader and has conflicts of interest.

Here is an example,

http://www.ncbi.nlm.nih.gov/pubmed/21623685

Would a second author negate your concerns about the primaries conflict of interest?

Thanks Farhad, to be honest my question was more theoretical. The review I was planning on doing was really a scoping review and so many of the restrictions are somewhat relaxed. But I was curious about where people stood on this issue. I can certainly appreciate all the arguments against single coding. However I'm not yet convinced that a single coded review is utterly worthless and I feel that realistically, outside of medicine the money and skills base for multiple coders just might not be there (yet) - so why forgo the opportunity to have a more robust review of the evidence just because you can't have an optimally robust review of the evidence?

Hi Steven, If I understand your question you are going to do a systematic review in the social sciences. I think Farhad's advice is correct. If you can't have two coders, most unfunded projects couldn't afford this anyway and it might compromise a piece of work submitted for assessment which needs to be authored by one person. In any research, however, the next best thing, as Farhad suggests is to validate the protocol that guides the research. The Scoping Review takes the evidence requirement down one level, I agree but that depends on the rigour you want to achieve. I know where you are coming from so to speak, as to try and understand the methodologies involved (moving from Media to Medicine) I attempted a Scoping Review, with no intention I should say of taking it to a full review as an exercise in understanding process. There is a blog about it here [ http://scopingreview.blogspot.co.uk/ ] with some references you should read. The final out come, a hybrid, scoping bibliography has actually come in handy for colleagues. So fortune in the end did favour the brave. [ http://www.networks.nhs.uk/nhs-networks/nwas-library-and-information-service/documents/GuidetofrequentusersofEMSAS3.pdf ]. BW Matt

Steven, to answer your question on "whether having a second author will it negate the influences of conflict of interest"

Before I answer your question, I would like to give a brief background on publishing in medical fraternity. Ghost writing is prevalent in medical journals. It usually refers to writers sponsored by a drug or medical device company, who make a major but uncredited research- or writing contributions. These ghost writers are experts in the respective field but working in knowledge process outsourcing companies, who are the big pharma vendors to do these type of work.

Key opinion leaders who work with pharma companies hire ghost writers to write. This is becoming more menacing. A new cross-sectional survey found that more than 20 percent of the articles published in six leading medical journals during 2008 were likely written by honorary- and/or ghost writers.

And coming to the systematic review, as you know a well conducted systematic review is the best available evidence and would help influence policies. And if you come across a "Systematic review" written by a key opinion leader (with conflicts of interest), there is a very high probability that the paper would be biased towards his funders. Moreover, if the pharma company has roped in ghost writers to support the KOL in his writing, then the findings should be taken with a pinch of salt.

Hence, having a second author without any conflicts of interest may bring some credibility.

More articles on ghost writing below

http://www.ncbi.nlm.nih.gov/pubmed/23585648

http://www.plosmedicine.org/article/info%253Adoi%252F10.1371%252Fjournal.pmed.1001163

http://www.nytimes.com/2009/08/05/health/research/05ghost.html?pagewanted=all&_r=0

http://www.ncbi.nlm.nih.gov/pubmed/17259673

http://www.sju.edu/int/academics/centers/arrupe/pdf/Kim_Ethics%20Paper%20by%20Joseph%20Kim.pdf