Either give what the package gives and its name or get a better package. Eg See

https://www.google.com/search?rlz=1C1CHBF_enUS917US917&ei=-Pt4X73-BMu4tAb44rf4Bg&q=Eric+Lander++R+for+everyone&oq=Eric+Lander++R+for+everyone&gs_lcp=CgZwc3ktYWIQAzIFCCEQqwIyBQghEKsCOgQIABBHOgIIADoOCAAQ6gIQtAIQmgEQ5QI6BQgAEJECOggIABCxAxCDAToFCAAQsQM6CAguELEDEIMBOgsILhCxAxDHARCjAjoECAAQQzoICC4QxwEQrwE6BAguEEM6BwguELEDEEM6BQguELEDOggILhCxAxCTAjoCCC46BwguELEDEAo6BQguEJMCOgcILhAKEJMCOgQIABAKOgQILhAKOgYIABAWEB46BQghEKABOggIIRAWEB0QHjoHCCEQChCgAVDD-xRY2qcaYIPJGmgCcAJ4A4ABuwGIAZInkgEFNDYuMTCYAQCgAQGqAQdnd3Mtd2l6sAEGyAEIwAEB&sclient=psy-ab&ved=0ahUKEwi92viVvZnsAhVLHM0KHXjxDW8Q4dUDCA0&uact=5

Wow that's some link, Best D. Booth

If the authors give the test statistic and df, then just calculate it, so if the paper says t(55) = 1.77, p = ns, you know (with any stats package) that the two-tailed p is .08. If they do not provide any information the data are censored. So if you can't find out from the authors, censored methods are necessary. But I assume if some one doesn't give the test statistic, the paper's result may not be that trustworthy.

Hello Dylan,

So you're saying you've found one or more relevant studies wherein the authors failed to give either: (a) an effect size; (b) sufficient summary statistics to compute an effect size; (c) the statistical test result (e.g., F-value & df); and no p-value in the case of "non-sigificant" results?

If (a), (b), or (c) are given, you can compute an effect size.

If none of these are given, you're really at a loss, and, as I see it, you can:

1. Try contacting the author/s for the missing information;

2. Do as you have done for the "p < .05" cases, and pick an arbitrary value, and insert that into your estimation (say. p = .50). Make note of this, and run any summaries with and without the unknown cases; or

3. Omit the studies from your analysis, and indicate that you are likely biasing the estimated ES upwards as a result.

Good luck with your work.

David Morse, Daniel Wright, and others- correct me if I'm wrong- but in my opinion the value of a meta-analysis, if any, comes from a thorough appraisal and re-analysis of the raw data from multiple studies. What you are describing sounds like a simple review paper. Ideally, you should be able to compute your own p-values based on the data. If you do not have access to the data, nor any additional and critical study background details, then the paper in question is of limited worth in a meta-analysis. Perhaps my criteria are overly strict, but then what is even the point?

Below is random example of a Cochrane review:

Article Appetite stimulants for people with cystic fibrosis

You can see that the authors try their best to acquire meaningful primary data, and report when it was not available.

Dylan, what kind of meta-analysis did you have in mind, though? Perhaps you can link to a paper in your field that adopts the format you are planning. This would provide context that would make it easier to answer your question.

Thanks for the suggestions everyone - I think I have a good course for proceeding.

Hi Miky Timothy , in a meta-analysis you should evaluate the quality of the actual data, but usually the analyses are done with summary statistics. That said, you can construct a multilevel model with a mixed of raw data and summary statistics. I believe that is in Goldstein's book (sadly in my office ... one I wish I brought home last March). But, I don't see that often. If the paper is in the last 20 years or so (since journals have said to report enough information like test statistics and effect sizes) and they won't give you more information, it might not be trustworthy (see https://www.degruyter.com/view/journals/edu/2/1/article-p1.xml).

One thing, Dylan Johnson , I assume you want the effect sizes not the p-values? What are you trying to estimate.

Hello all,

Miky Timothy 's comment needs a response. First, let's be clear: It's simply not going to be the case that individual studies furnish the raw data (the lone exception I can envision would be simple examples such as 2 x 2 contingency tables); print journals won't give up the space, and (unfortunately), many authors are reluctant to "give up" the actual data points, lest someone else show them up by posting a re-analysis. I hasten to say that APA guidelines suggest authors should be willing to open their data to others for five years after publication, but many of my students have encountered instances where authors: (a) would not reply to requests for or about data; or (b) would claim that data were no longer available.

Meta-analytic studies make use of effect size estimates from the studies that they include. ES values are ideally given in the original study write-ups (and are strongly encouraged by multiple professional organizations and a number of scholarly journals). However, the practice is not universal, as Dylan Johnson 's original query illustrates. That's a challenge for folks who want to complete meta-analytic studies.

You can, though, construct a great number of ES values from either: (a) summary statistics; or (b) the test statistic and N or df, without having the original data points. I think Dylan was trying to be able to include as many data points (ES values) as possible from the studies that he was able to find.

Best wishes to all.

To add 2 points to David Morse 's thoughtful response.

1. Hopefully things are changing. See https://www.cos.io/. And a lot journals encourage authors to submit their data. Here is my comment for the main journal that I am associated with (https://www.degruyter.com/view/journals/edu/2/1/article-p1.xml).

2. I recently wrote to an author to get their data. They spent awhile to see if their IRB allowed them to give me the data (I wanted to see how the data worked with something other than OLS ANOVA). IRBs often don't have this problem, but MoUs do, so it is important for authors to allow the data in some form to be released. For some data storing the raw data is trickier for, so I have to have a special lock on my office and a computer not on the internet and a safe and ... .

Thanks Daniel Wright and David Morse for your insightful responses to my comment.

Daniel Wright, regarding your excellent linked commentary, I think the reluctance in data sharing is similar to researchers' hesitancy in accepting open peer review. Both are plainly necessary, in my opinion, for viable, transparent, and reproducible scientific communication, and yet there is a cultural problem - One I think we can all relate to, i.e., would you really want people scrutinizing and searching for faults in hard-won and personally vetted work? It's a little intimidating and not an expectation encountered in training. There is no incentive for data sharing, and indeed it is de-incentivized, since providing open data/peer review offers no advantage while leaving the door open to criticism from others. So I think it simply must be compulsory, which is beginning to happen.

Daniel Wright and David Morse , your explanations on typical expectations in meta-analysis are informative. I am not aware of meta-analyses (in a form at all resembles that of Cochrane reviews, etc) being done in my field (comparative neuroanatomy and experimental neuroethology), though I may be wrong. Since the data generated in studies in my field are either in spatial units (i.e., microns), derived proportions from these units, or discrete counts of objects or behaviours, a raw-data synthesis would be more straightforward and manageable. Such a synthesis would be difficult also (with questions of whether to deal with the relative scale of an organism, for example), but would still be more manageable and interpretable than integrating a variety of what I imagine are ultimately qualitative categorical metrics from observer-or-self reporting. This is just the way things are, since not everything can be reduced to a caliper reading.

I guess the concept I had in my mind in the comment to Dylan Johnson , in terms of raw data, was not that such data, were it available, should necessarily then be combined into a kind of mega-omnibus. Rather, I meant that the point, error, and fit estimates/comparisons are difficult to interpret meaningfully without access and evaluation of the raw data. Every log transform, non-parametric "substitution," adjustment and factoring for "confounding" interactions, etc., is another gear in the black box of information loss that generate summary and results statistic tables.

Another issue is that meta-analyses are not a substitute for replication, and it seems a lot of the difficulty of meta-analysis is that no two studies are sufficiently similar for an uncomplicated analysis. The replication elephant is always lurking in plain sight!

Finally, and to make a very long post even longer :-) , below are some extracts from the excellent Meta-analysis chapter (by Sander Greenland and Keith O’Rourke) from the text "Modern Epidemiology":

• "Ideally, one would obtain the raw data from every study that recorded the exposure and outcome of interest, transform these data sets into a common format, merge the datasets together, and analyze the merged data set in a pooled analysis. These tasks would require some guidelines. For example, one would have to include a variable that identified the study from which each subject came; this variable would allow one to analyze effect variation across studies. One might also have to devote much labor to producing a common format for data from different studies, especially given the very different and often extensive database problems across studies..."
• "Unfortunately, the preceding ideal is often unattainable in epidemiologic settings. Original study data may have been discarded, or investigators may refuse to supply their data. Even if they do supply what they believe was the finalized study data, later editing may have made it impossible to replicate closely their earlier published findings and establish any confidence in what they have supplied...."
• "Most meta-analyses will require from each study both a point estimate of effect and an estimate of its standard error. The actual study data would arguably be preferable, if correctly coded and documented adequately. Therefore, in the interest of facilitating reviews, authors, referees, and editors should consider making the study data available with the least restrictions on availability as is ethically possible (some funding agencies may even require funding recipients to make data available). At the very least, published abstracts and reports should provide both the point estimate for an association and its standard error or confidence limits (from which an accurate standard-error estimate can easily be recovered). A point estimate with a P-value may not provide an accurate standard error, and so should not be considered sufficient for reporting purposes."