As long as you trust (or, preferably, can verify) that the data were appropriately collected, analysed and reported, I do believe that the "grey" sources are worth using. Considering that you also have a good (even higher, 65%) proportion of your dataset from peer-reviewed sources, you can compare the consistency of their results. Unfortunately, there are lots of good-quality data that remain unknown to the scientific community. Many authors fail in publishing their results in peer-reviewed journals because of an inadequate level of training in scientific writing and/or poor English skills. I must also be critic here and state that datasets from peer-reviewed journals are far from being bias-free or bias-proof.

I agree with Júlio, that data from 'grey' literature is worth trusting to some extent especially so because they have been peer-reviewed.

I will add that we live in an information age where lots of good quality data are deposited and available in different repositories e.g. Dryad, GenBank, TRY, tolweb etc. One simply needs to mine and obtain such data. Just as it is good to populate the databases with additional data as they become available, it is equally good and acceptable to use data that has been collected by researchers in the past.

In a different light, to those who English is not their native language, reading lots of articles in English journals makes one familiar with scientific writing skills. Some of the best scientists I know are non-native English speakers yet write good quality articles for English journals.

In my opinion the grey literature is an unvaluable source of data in conservation and wildlife ecology studies. Grossly speaking most baseline biological information on populations and species came in grey form until a few decades ago. Also there is a lot of biological information that goes unknown to the scientific world because it is written in the form of unpublished reports as happens with reports produced by government agencies (though the web is making some of that work visible). My experience is that most reports -a typical grey literature product- are written by trained biologists and are quite trustable, specially as a source for raw data. Moreover, in many instances the grey literature presents data with a detail that is unthinkable for scientific journals (an arena where competition for space is ferocious). I would like to think that using grey data for scientific literature is a way to turn grey into... (which color is the scientific literature?).

Grey literature is not equal to bad literature. Actually is published information, in the sense that was made available to the public. The difference is that the distribution channels are often limited and accesibility generally is difficult. The other difference that you can have is the peer-reviewing process widely used in journals. However, some types of grey literature, such as degree theses, are reviewed by academic commitees in order to be approved to grant the degree (and sometimes this revisions can be harder than those of the journals' reviewers). So, my advice is that you can use those sources, being as critic as you are with the journal papers (you can't take the information of any source acritically just because is in a high impact factor journal and not in a technical report).

Grey literature, like published literature, can be good or not so good. Like others have said, the most important criteria ought to be whether the methods are clearly stated, so that one can properly evaluate the validity of the data presented. These methodological details are important both for judging the work, but also (especially for diet studies) for determining whether a study (grey or not) is comparable to others within your meta-analysis. Good luck!

Most research does not even make it to publication - so the grey is often an invaluable insight into work. Quality can vary hugely, however. You need to read the full text of it to understand the context, especially the sampling protocols. The other important issue concerns the ethics of authorship. Grey literature is often produced as a quick way to report project results to funders or other users of the results, and reflects preliminary results or preliminary analyses that authors intend to go ahead and publish later. Therefore with recent literature it is normal to ask the authors for permission. Indeed, many grey reports contain text saying that results should not be cited without permission, let alone mined for data! However, with older stuff this is unlikely to be a problem. Hard to know where the cut-off is.

Thanks all. This question arose because we have just been rejected by a reputable peer-reviewed journal on the basis that much of our data was obtain from the grey literature. That data was obtained using scat analysis (a commonly used technique in carnivore dietary studies) and the grey publications were either theses or by reputable scientists and/or were published by organisations headed by reputable scientists. My Indian colleagues and I are planning on lodging a complaint, but I thought it would be good to see what the ResearchGate community thought. Thanks again.

In my experience when journals make such stupid comments there is little chance in changing their minds - best to find another journal. What can help with such issues is if the grey literature is accessible online - or is at least in some report series or archive such that it can be tracked down. It is worse with things like "unpublished report" ..

However, if you do push, one of the key arguments is that failure to include the grey literature can result in (1) at best a reduced sample size, and (2) at worse a real bias against studies that either don't find sexy results or from regions without the resources to publish.

John - It's in PLoS - if you can argue that it's robust science, you'll get it in. (Still, I have suggested other journals, but the team are keen on PLoS).

So, argue for sample size (without the grey literature you won't have enough to say anything), argue for bias (central Asia is not known for its very strong English language publication, and the fact that most snow leopard work is motivated by conservation rather than science should also help), and underline the extent to which this literature is accessible. I would also suggest reading carefully through the methods sections to show that you have evaluated both the way in which scats were collected and how they were analysed. With that it should go through.

I have found that the best way to answer complaints about sources or quality of data is by providing transparency through excruciating detail. I've done this by making tables that highlight the sources and summarize the results of each source, but I've seen some papers that include their database of results as a supplement.

You could also (if you haven't already) make the issue of source a central point of your paper: does including the grey literature change estimation of diet, make it broader/more narrow, have conservation implications?

Matt I think most of data in unpublished sources used are not at all faulty. They are data collected by the same people who may be could not publish for some reason or the other. Take for example Jackson's thesis on snow leopard!! now there is data collected by researchers who have worked in the area for years and using standard techniques like you said. Sample size consideration is a big factor, but given the data deficiency with respect to the species, I think undervaluing the wealth of information here is not good for science or conservation.

I agree entirely Salvador/Emmi. Thanks.

I think the issue about grey literature is relevant to conservation science and merits further analysis. While I was swimming (to keep fit) one idea came to me: what about an analysis of grey literature use in peer reviewed journals? The idea would be to write a comment/short note for submission to (yes) a conservation journal. Well, just an idea. On the other hand, I may recommend the impressive essay by Boakes et al. 2010 (Plos Biology http://www.plosbiology.org/article/info:doi/10.1371/journal.pbio.1000385) on the biases associated to the sources of occurrence data: just have a look at figure S1and see what they found when confronting grey versus peer-reviewed literature. Their findings suggest that grey literature can be far more important than peer-reviewed (listed) literature for biodiversity assessments!

I think a lot of the so called grey literature has very high quality data. For instance there are many monitoring and survey data which are not really relevant and novel enough to make it to a peer-reviewed journal, but these data were collected in a precise way and can easily be used for long-term trend analysis. This is just an example. Another example would be the use of scattered data on sightings of rare or inconspicuous species, which often only appear on the grey literature, but when group in a single analysis can provide very valuable information on the patterns of spatial-temporal occurrence of such species.

All in all, I would say the grey literature is not just robust, it is an invaluable source of data for many conservation and ecologically-focused studies!

If unpublished monitoring data and databases are among "grey literature" - just go ahead and try to explain the robustness of the source. I fully agree, that in non-western world biggest half of dagta is not - and will never be - published.

This is actually evolving into an interesting discussion. On a recent book chapter I was writing I tried to find data on large carnivores like wolves killing dogs (which is one of the most important dimensions of conflict concerning wolves in Europe and North America). There was almost nothing published (just two papers from Finland), except wildlife management databases are full of records - however these are "not published". It was even worse with pumas, there is no peer-reviewed account of them killing dogs, but even a quick scan of online newspapers and wildlife management agency reports shows it is widespread. This is quite scary, because the bias concerns not just the quantification of an issue - but a total absence of even a qualitative reference to the existence of a behaviour. If we base our understanding of the world on the peer reviewed literature only it will be a pretty poor view ....

Surely all data is valuable data. I agree with Emmi to make the source of the data one of the focus points of the study. It could serve to raise the very question we're discussing here in a wider community. The significance of its inclusion to your results would make for an interesting analysis.

Sorry to jump in with an ignorant question here, but I was wondering where is the best place to look for grey literature? I know it won't all be conveniently found in one large helpful database, but are there certain organisations that have databases that share these kinds of data?

I'm looking for vervet monkey (Chlorocebus genus) locality and population data in sub-Saharan Africa as part of my PhD research. and I'm wondering where best to look for some extra data. It's not the easiest of jobs searching. Would anybody know if organisations like the Wildlife Conservation Society, East African Wildlife Society, or forestry groups and things like that would be the people to contact?

Apologies for the hijack but I thought this would be a good place to ask such a question.

Google gives you an amazing access to very many things - but you shoudl also trawl through the reference lists of articles for citations. Otherwise it depends on context. For species of management interest there are many report series and databases kept by national wildlife management authorities. Most national statutory bodies will also have report series. Beyond this it is the universities and conservation groups who have things - which may or may not be online. Basically you trawl through every possible source online and email everybody. For location data it is a good start to check out the various IUCN pages (e.g. the red list pages) and global databases like GBIF.

List of NGO may be helpful - not many of them keep things online, but you may contact them via e-mail. My practice show, that in USA, if you not only ask, but also explain why you ask for data, many sources you cannot find through google became open. Unfortunately, personal contact not always will work.

Aoife, Linas and John are correct. We got most of our snow leopard grey literature from The Snow Leopard Trust's unpublished reports. They're all done by well known experts on snow leopards and the trust's scientists are even better known. Theses, Google and the reference lists of all our sourced literature were invaluable.

Now on a separate note, would it be appropriate to refer the journal editor to this discussion in our rebuttal?

Yes, as the Editor-in-chief, I would accept such explanation - and arguments, plus good description of the sources used in the Letter to editor, or similar explanation.

I think Linas got the most important point (when it comes to publishing a study that makes use of "grey data"): make sure you detail the source, comment on the reliability (if possible), discuss using such data in detail, and explain why you had to...

That's great. Thanks John, Linas and Matt.

And I would also say yes to referring to this discussion in any rebuttal. It would be wasteful not to use grey reports so a strong justification for their inclusion would be a good way to go.

In fisheries ecology we use any source of data that can be utilised and justfied. Data are so rare and valuable. If you run a population dynmics model based on official data sources only, you may well end up with inaccurate results. For decades fisheries ecologist supplement official figures with their best guesses

Matt, can I ask you: will you use those data to perform a meta-analysis or it is just for a review? Either way, the decision is up to you, sometimes the texts on the grey literature are a good indicators of the techniques used and you can chose arbitrarily what set of data you´ll use. This is a topic widely discussed by people on meta-analysis and as far as I can tell you, the general consensus is to use whatever you want as far as you can justify why

Cheers

Facundo - we are using these data for a meta-analysis on prey preferences.

So called grey literature sources contain a treasure trove of data too precious to be ignored. In the stream macroinvertebrate field government reports are frequently cited. The experts who generate these reports are well respected and their mission differs from a typical academic's and does not necessarily include peer-reviewed publication; in fact I know of cases where it is actively discouraged. They do far more field work, scope work, and data crunching than anyone who also teaches. They are in it for the long haul and typically have better long-term biomonitoring datasets than an academic can muster. If I have a questionable organism to identify, my first port of call will be a state official (and prolific contributor to the grey literature). I have the height of respect for my colleagues in state and federal organizations and would not hesitate to use their data; they share it willingly and when I asked about collaboration on a paper, they told me to take the data and publish with a citation to the source. By all means share my comment with your editor.

Good luck

Declan