I basically agree with all previous answers - most important issues have already been addressed, ranging from the question if a quantitative approach is the appropriate one to very specific questions of the quantitative design, theoretcial framework, data collection, data quality, analysis and reporting. Thus, I only have to add that - regarding the evaluation of available data and results - I do pay not only attention to the sample size and related issues, but also the sampling strategy - which are crucial for performing statistical analyses.

To properly evaluate data quality as well as statistical analyses applied, raw data should be available. If this is not the case, respective information must be provided, important decisions in all steps of the research process must be well documented and argued.

First of all the approach of data analysis must be mentioned in the statistical analysis heading under materials and methodology. We become confident when researcher has mentioned about his statistical analysis part in minute details, like why the continuous values have been presented in mean, SD or median, IQR; why they conducted parametric / non-parametric tests with rationale; why specific parametric / non-parametric test; performing multivariate analysis - what is the dependent variable and the independent variables etc. When all these things are mentioned it becomes more credible to the readers.

Secondly, from the result and table section one can guess / link the rationality stated by the authors in the methodology (statistical analysis) section like someone has stated that data were distributed normally with absence of outlier, and from the mean & SD value someone can judge / predict that the data were either not distributed normally or outlier was present.

I would add to what Indranil said that the methodology and results should be as transparent as possible so that the findings are reproducible.

As is well known, simply finding a p-value of

Interesting question here.

From the outset, and as Aaron Levenstein once said; "Statistics are like bikinis. What they reveal is suggestive, but what they conceal is vital.” This line of thought is invariably aided by the prevalent publication bias that is now considered norm in the scientific community.

The almighty 'P value

Statistical Models and Credibility

Please let me know if these references/sites are helpful to you:

1. Statistical Models and Credibility Web · 

https://www.casact.org/pubs/forum/98wforum/98wf061.pdf Statistical Models and Credibility Leigh J. Halliwell. ... More recent statistical theory. Bay&an analysis for example, also addresses the use of[PDF]

2. Credibility Theory - Actuary.com

Web · 

www.actuary.com/seac/handouts/200906_03c_Credibility_Theory.pdf

Statistical Credibility Theory Donald F. Behan ... to consider in determining the degree of statistical analysis that should be used to project future results.

Dennis

Dennis Mazur

This is a great question. It looks simple and yet complex. One important aspect of critical appraisal is the thorough scrutiny of the statistical/data analysis approach used to answer a research question. First, the statistical method used must be a correct one to address the outcome of interest and answer the research question. In addition, all levels of analyses (univariate, bivariate and multivariate) must be well-described and followed.

Second, the analysis must be performed on an adequately powered sample size. Other than paying attention to assumptions underlying the choice/use of a particular statistical method and its properties, the exclusion of collinear variables, inclusion of statistically and clinically significant variables in multivariate analysis are critical. The reporting and interpretation of findings is crucial as well. At times, you may need to verify reported results by performing a simple analysis with the software on your PC.

In general, ensure a sound and correct statistical approach was used, and the results are free of chance and bias.

From my limited legal perspective I wrote some stuff on this in an article on regulatroy validity, available here: Working Paper Regulatory Validity

There are so many things to say.

1. Data quality and validity. The best approach to analyse the data is neglectable if the data are not good. In case of primary data you do have some control about it, in terms of secondary data you may face at least the following phenomenons/problems (a) up-to-dateness, (b) validity and reliablity (method of collection, do they really fit to your ressearch question, compareabilty), (c) are the data influenced by some kind of (political) interests (could also be seen as part of (b)). Is it possible that the sample used for data collection is biased (willingly or unknowingly).

2. Is the analytical method the correct one and do the statistical frame condisions apply? E.g. can ordinal scales be interpreted as interval-scales ans therefore methods used which need at least interval scales (e.g. regressions). Are e.g. linear relationships assumed (for a regression) and the relationship is not linear. Is the distribution a "normal distribution". Are the data respresentative for the total population (of whatever) and can the results extrapolated to the total population or just for the sample used etc. etc.

3. Even though there might be a statistically significant relationship, does common-sense also support an interpretation. See example on pages 19 and 20 in attached file. In case there is a significant relationship, may that be a spurious relationship (because there are other variables behind the ones measured)??

These things come to my mind spontaneously. Maybe I add some more later