While my descriptive statistics shows a degree of progress in students' performance after receiving academic writing instructions, the inferential statistics rejects the evidenced progress. Overall, 35 students took part in this study, but only 20 students completed the textual borrowing exercises. However, the small samples effected on the distribution of the variables, so the Wilcoxon Signed Rank Test was the only solution. Do I have to accept that the statistical contradictions have to do with the small samples or any other factors might have influenced the study? Any recommendation for overcoming the research difficulties and improving its power will be appreciated.
I doubt that your inferential statistics are inaccurate. If your descriptive statistics show only a "degree of progress," then it is highly likely that you small sample size is not enough to make this statistically significant. In general, it is very difficult to find significant differences in samples with N = 20, due to the "lack of power."
what are the confidence intervals for your mean etc.
they may very wide given the very small sample size, making it difficult to conclude that the intervention had any progress/effect.
how do you know that the inferential statistics are incorrect?
Many thanks to the answers. The results would not vary with a lower confidence interval, though we defined that to be 95 percent.
In my point of view, if the P-Values would confirm the mean differences, we could claim that the inferential statistics was accurate enough to support the descriptive statistics. As you know, whether or not, researchers expect that bout could strengthen each other, and if impossible with the small samples, they have to look for a solution. What can be the vital solution? Do we have to report the contradictory results or any other statistical methods are capable of reducing the lack of power and predicting a logical future for the small projects?
There is no contradiction here, just it seems that statistical tests do not have enough power. You can provide relevant descriptive statistics for us to comment on.
p values might be deceiving and sole reliance on them is not recommended. Instead of p values you can evaluate effect sizes and confidence intervals.
There are numerous threads I recommend on RG about this including the following
https://www.researchgate.net/post/What_else_should_be_incorporated_while_publishing_significant_p-Values2
Thanks for the helpful recommendations. For the descriptive statistics, I have calculated the mean percentages and standard deviations.
In addition to the effect size and confidence interval, I might be able to report the positive and negative ranks resulted from the Wilcoxon signed-rank test. The ranks more confidently support the descriptive statistics than the p-value does. Frankly, before watching the video link, I had no idea what the ranks would do.
What about the Wilcoxon ranks? Are they powerful enough to be included in this study?
If the p value is non-significant, that is telling you that no matter how large the difference appears to be, it is not large enough be to statistically significant. This is exactly what it means to say that your test has "low power" -- that your test is not strong enough to detect an effect of this size.
If you had a larger sample, a test result of the same strength might be significant, but with such a small sample there is too much variance around your effect size to say that it is larger than zero.
Then either increasing the samples or running another test might result in significant values. Due to the dependency of data, I have not been able to perform a Chi-square test. McNemar's test was also a p*p table in which any participant could only take one position of "yes or no." None of the preconditions met, therefore, the Wilcoxon Signed Rank was the only test for my purposes. Furthermore, three more statistics counselors confirmed that the Wilcoxon was the only chance. Is there any other non-parametric test for assessing the effects of time-series treatments?
You can raplicate the experiement yourself again using the same sample size, or using your negative results you can reuse the previous data and add more sample and amplify the effect. Would be good also if you publish your negavite results togther with data in the end, where you have gone wrong, so that other researchers would benefit from that.
I think it's inappropriate to apply statistical methods to such a small sample. I would try to assess results in some other way that makes more sense.
Good suggestion Dr. White. The current data could also be analyzed qualitatively with a content analysis approach. However, maybe wrong, but I think there is always a logical solution for any researchable and sophisticated problem in the universe, even if the problematic issue is a small sample or none significant value!
That you for responding, Moohebat: I tend to agree with your statement; and you're more of an expert on this than I. Brian
The answer to your question posed here may also be informed by the purpose of your research.
What was your original question? Have the results helped you to form (a) new question(s)? Have the initial findings made you look at the data differently?
It could be that your early findings will lead you in a different direction, and, eventually, you may want to include how it evolved in the story you tell.
I quite agree with Clare V. Besll. Your results should lead you to discover new areas of your study that hitherto were hidden. Your literature review should have been copious enough to help answer your research questions. It is important that your literature review should have looked at the other side of your study so that when your results are out, you will have literature to back bit up eitherbway
I believe that the sample size was small to the test used, I suggest another statitical test, data normally disributed better when sample is larger.
I totally agree with David that the reason could be the small sample size
This collision between a statistical significance test and qualitative data interpretation is a very common 'conflict'. Gerd Gigerenzer has labeled this situation as 'mindless statistics', a method used to test only a null hypothesis. The solution in such a situation as you describe is to develop other (new) subject theories from your qualitative data and then you can again make new statistical tests now with a larger sample.
Moohebat, while it is necessary to consider the size of your population, i think it is important for you to also consider other inherent variables such as factors which may have affected students' concentration and attention to instructions, the learning atmosphere/environment at a given period of the instruction, previous learning/educational background of the students involved, and also, probably if there is a little change in the style of the instruction. I believe this might also help cos I had a similar experience too way back in my teaching career.
The sample size was very small and i think qualitative data analysis procedures were more appropriate for this study.
In respect of the analysis section in your report: argue why you have carried out the Wilcoxon Signed Rank Test - correctly so, and I beg to differ with some of the colleagues, the sample size is absolutely fine - and report that the null hypothesis could not be rejected due to p > 0.5. The fact that you could not reject the null hypothesis, does not lower the quality of your work.
However, the Null Hypothesis Test (NHST) idea has some fatal flaws. One is, and this has been pointed out by several already, a small sample size will rarely give you a statistically significant result, and a very large sample size will almost always give you a statistically significant result. Geoff Cumming is one of the authors who has pointed this out very vocally, and has offered alternatives.
In your case, you may wish to report the effect size, either in original units, or in a standardized measure like Cohen's d. I attach an article by Cumming which you might find useful. If you wish send me a mail and I have a look at your data.
Good luck with your research
The emphasis on effect size would be more appropriate if the original results were statistically significant. In that case, we might want to dismiss a significant result that had very little effect (which is most likely to occur when large samples are tested with with what would amount to generous p values, such as .05).
But that combination is intended to exercise care against against a Type-one error (i.e., claiming an effect is present when it is not). That is not necessary, in this case, however because Type-one errors are precisely what significance tests are designed to prevent.
The issue here is Type-two error, where a "real" effect might be wrongly rejected. If this result was a test of a crucial research question in circumstances where it would be difficult to produce higher quality data, then a reviewer might be willing accept a value of p
Why do you have to use stats on 35 people? Isn't 20/35 enough to explain the matter at hand? Plus you did explain that your sample of 35 was small, and besides sometimes representation of data using stats can be misleading.... tell me what is it that you want to explain more or emphasize using stats besides 20/35? then ask yourself so what? Is your study qualitative or quantitative? And if its qualitative, how do you explain what people are saying or doing insignificant? Again I guess you know your study better...
It is unclear from your question whether you are trying to improve writing skill or to improve a disciplinary skill by using writing as a way to improve understanding or proficiency.
What if the skill that you are trying to measure develops too slowly to register a significant difference in the short span of time in research study? This kind of problem is common when attempting to measure gains in critical thinking or intellectual development. The skill is quite meaningful and measurable, but measurable gains become significant only over periods of several years and not over the time of a single course or module. There should be some information in the literature about how long it takes to develop the kind of skill that you are trying to develop. Writing skill develops slowly. Whatever it is, consult that literature and look for longitudinal studies.
How are you scoring the writing products? Is this done by a rubric? Can you do random number simulations for 20 participants on each of the rubric's dimensions and compare total randomness with your results? You might be able to use random number modeling of your instrument to determine how large a sample size is needed to generate reproducible random results. For your study, I do suspect that it is unlikely that 20 participants is sufficient. You cannot arbitrarily throw out a number and then say that is sufficient. What is sufficient depends on what you are trying to measure. Your problem seems less about statistical number-crunching to prove significance and more about establishing an understanding of the numeracy of what you are attempting to measure.
I appreciate your active discussion. Although based on Cochran’s formula the ideal sample size was 108, due to the reformation of limited population, the final sample size declined to 34 subjects. However, this is in a vacuum whether working on the 34 samples will not result in such statistical matters. To tell you the truth, before commencing the study, I was aware that involving the ideal samples might be impossible; but, two series problems made me not choose mixed methodologies: Neither was I a proficient qualitative researcher, nor had my supervisor already experienced qualitative studies.
I have been trying to improve the writing skills of M.A. students. This project especially focuses on M.A. students’ strategies for source text borrowing. We have also designed a qualitative scale for scoring the writing products. Hence, the outcome of the 120 writing tasks can be analyzed both qualitatively and quantitatively.
The first step is a review of the basic data and the main equations that have been relied upon and then determine the expected results within the right to work and re-analysis of the conditions.
sample based research will give you only estimates not exact results; some times simple % will tell us good results. Anthropological approach will also guide you
I think that you should pay attention to the results, remember that in a researching process it´s is possible find differences between the hyphotesis and the reality.
Yes, the small sample size strikes me as very problematic. Writing assessment research is fraught with methodological difficulties, and historically a majority of the research in this area fails to meet acceptable standards. Numerous faulty studies have used small sample sizes to make unsupportable generalizations.
Although you provided limited information about your study, something that gave me pause is the phrase "writing task exercises." If this means what I suspect it means, you have a problem with face validity. In addition, if you did not train the staff who performed the qualitative analysis, and if you did not socialize them to a rubric, you have a problem with reliability. These factors alone would lead to contradictory assessment results.
You have mentioned a very important hint that it is a research project. You have to give a research result one way or the other. Surely the correct solution also should be one or the other only.
A single sample will surely give you only one solution one way only. If you take a second sample it may end up 50% contradictory result or 100 % compatable results. Now when you have three samples, it gives you results 100% to 66.7% (i.e. 2 samples with True and one result with false result or all the three true result or false results). You have now a choice to select the samples and report which one suits you and gives you 100% one way (uncontradictory) result.
Therefore in any sample study, selecting or not selecting a sample is in your hand and you have a reason to reject because it was not only in agreement with your need (required result) but also because it deviates from majority and any of the measurements time, method, precision may be wrong for the neglected samples. This point also could be staed in your research if you are bold enough to counter in a presentation. All the best.
Ramakrishnan V.
This is a good article (link added) to pass along when you are presenting scientific findings that contradict deeply-held beliefs. It shows that when it comes to research on health, more studies aren’t necessarily better – it’s having the right kinds of studies.
http://evidencebasedliving.human.cornell.edu/2010/06/04/when-studies-collide-heres-how-to-understand-contradictory-research-findings/
You May need to establish whether your extreme observations are valid observations. Because if they are statistical extremes not belonging to your sample. You may exclude them by trimming the sample
I assume many researchers share this question. That is why there are so many interesting replies.
We should not neglect that it is not only the outcomes that matter but the process. As long as you stress the methodology you follow, the conclusions and the limitations, you are clear. Of course, it depends on what look for each time.
Personally, I find this contradiction interesting and food for further research.
What do you mean by contradictory results? Do you mean a) that the null hypothesis could not be rejected? or b) that the results were significant in the opposite direction that your predicted?
In the case of a) you need to recognize that you lost much of your planned statistical power because you got fewer data points and you backed down to a "weaker" test. Both makes it harder to find differences even if they are there. Most likely you did not have power to detect a relevant difference. What you can use the collected data for is that you now have a better grasp of the expected effect size and how much data is needed.
In the case of b) you got a strong indication that something else is at play. You should try to find out why the results went in the opposite direct, and you may learn something important or even discover something new. You might for example have many different levels (say courses) and each level might have progress and still the conflated levels could show a deterioration. I saw one interesting example of this yesterday, and I will try to find it.
Other things to try: if you have before and after data for the same individuals you may try to calculate paired differences. After all you should use a paired test, but sometimes pairing individuals and courses and calculate an explicit "progress" score could be very useful.
A possibly relevant blog: https://paulvanderlaken.com/2017/09/27/simpsons-paradox-two-hr-examples-with-r-code/
If you haven't already moved on to something else, given the fact that 4 years have since passed since you posted this question, I am curious if you were satisfied with the advice you received here? If not, perhaps an extensive literature review will begin to support your alternative findings. It may well be that your small-scale findings will inspire a deeper-look?
It takes courage to pave new pathways to knowledge. Good luck Moohebat Shoyukhi!
- Badges
- Science topic
- Similar topics
- Vectorization