Hi everyone,
my question is the following. I have two samples with n=4. They come from experimental measurements, and the number of measurement is low due to high costs and time-consuming procedures to get one of them.
According to Shapiro-Wilk normality test, the assumption of normal populations is satisfied, and a two independent samples t-test was carried out, showing that the difference between the two means is statistically different. I evaluated the confidence intervals too for a significance level of 0.05.
Is there any beneficial from performing a bootstrap on those data samples? I mean, I performed bootstrap with 1000 samples and the CI show a slight reduction, this gives more reliability to the analysis or not? Is it a necessary step for such a small sample size?
A few issues (e.g., accepting the null hypo for normality based on a small sample, what kind of bootstrap, that if you are running multiple procedures for the same hypothesis in order to find a low p this is bad for a few reasons), but I think the first need for commentators is for you to tell us about your power analysis or whatever method that you used to help guide your sample size and design. The answer might be that the sample size is too small for many purposes for doing this type of analyses, but this depends on the design. With small samples with small population effects you need to be careful for Type S and Type M errors (sign and magnitude, see http://www.stat.columbia.edu/~gelman/research/published/retropower20.pdf )
Dear Daniel Wright
Thank you for your answer. I will try to reply to your points.
The data set came from 2 different experiments, I received them "as they are". The objective of the study was to develop a FEM model able to replicate those results.
In particular, we were interested in the difference between the two groups.
I performed simple bootstraping with resampling.
Concerning the power analysis, I did it with the R code in the paper you provided me and it gives 28.8% power, type S error 0.13% and exaggeration 1.85.
Dear Luigi Candido,
In general, statistics has many limitations and one is the sample requirement. Bootstrap is a technique to resolve the problem of small sample applications in statistics. As you indicate that you workout for two samples each from two experiments, it would be wise to perform the Bootstrap to overcome the issue of small samples in the analysis. If you do the Bootstrap, you can statistically argue, justify and validate your results very strongly. As you are aware, the small sample size required is 20 to use student-t distribution in statistics. Try at least to meet the statistical sample size requirement. Good luck in your studies.
In fact, there has been a long-standing debate about the validity of t-tests and the associated p values. In recent years, some authors suggested to retire p values. An international journal, Basic and Applied Social Psychology (BASP), has officially banned the null hypothesis significance testing procedure (NHSTP) (p-values, t-values, and F-values) since 2015 (Trafimow and Marks 2015). People are looking for alternatives to t-tests and p values. One of potential alternatives is to conduct a SCI analysis. SCI stands for signal content index. It is a new descriptive statistic discussed in "Signal content index (SCI): a measure of the effectiveness of measurements and an alternative to p-value for comparing two means" https://iopscience.iop.org/article/10.1088/1361-6501/ab46fd/pdf
- Badges
- Science method