I have two independent random samples . The distribution of data in each of the two samples may not be normal (mesokurtic) . Moreover, the kurtosis may not be same in the two data sets of the two corresponding samples. Thus there is necessity of testing of the significance of the difference between the kurtosis of the two samples. However, it is beyond my knowledge on how to test the significance of this difference.
The question thus here is
" Does there exist test statistic or testing method for testing of the significance of difference between the kurtosis of the two samples ? "
Dhritikesh Chakrabarty,
I think it's reasonable to test if the difference in kurtosis between two samples is extreme compared to the distribution of differences in kurtosis from many permutations of the samples. It's basically a permutation t-test. I provided some R code below.
a
The test statistic for testing of the significance of difference between the kurtosis of the two samples has not been found specifically and clearly in the answer given by Caleb Ashton Aldridge.
Moreover, the method of permutation t-test is required to be explained clearly. Steps in the method are to be mentioned in order.
Dhritikesh Chakrabarty I hope the below is more helpful in supplementing my original answer.
Typically kurtosis is found using the formula in the attached image.
The basic steps involved with a permutation t-test are:
@ Caleb Ashton Aldridge ,
Steps in the testing method are clear now.
Lot of thanks.
I agree with Caleb's answer, but I would make two changes. I would suggest in that answer is that it is not a permutation test. Caleb has described a randomization test. A permutation test looks at all possible arrangements of the data. A randomization test randomly chooses a subset of all possible arrangements of the data.
The other change is that 500 randomization are probably not enough if you have sufficient data to make this worth doing. Estimating the kurtosis of the population based on a sample size of five is not a good idea. Please try plotting the histogram a few times with 50 randomizations. Notice how the histogram changes at each run. Now try 500, then 5000. My guess is that you want something between 20,000 and 50,000. It should take only a couple of minutes to run and the accuracy of the test is much improved. Think of it this way. There is a 95% confidence interval about your estimated p-value. The more replicates that you do the tighter this confidence interval becomes.
Timothy A Ebert is correct about the terminology. I use the terms interchangeably—incorrectly and in bad habit! I'm going to strive to be more aware of this. Stats give people (myself included) enough trouble and I don't want to perpetuate it with incorrect terminology.
Dhritikesh Chakrabarty, like Timothy said increasing the number of randomizations will be a little more computationally demanding but strengthens the confidence in the result.
And Timothy, am I correct in saying that in order to calculate CIs for the p-value one would have to repeat the randomization test multiple times (again a large number), calculating the CIs for the p-value from the distribution of p-values.
Building off the earlier example,
a
I am not skilled enough in R to work through the program. I get about 50%. Maybe some packages are needed in addition to base R? I tried running the program and get an error "There is no package e1071."
As a flow chart there are at least two options with different outcomes but both give a 95% CI about the p-value.
1) Randomization error (Caleb's answer)
a) generate the data: 1 sample from each of two populations
b) set up a loop-1
c) set up a loop-2
d) calculate the difference in kurtosis between groups.
e) Save result
f) end loop-2
e) find the p-value
f) save result
g) end loop-1
h) find mean, standard deviation for result in step f
2) sampling error (almost exactly the same but the order is different)
a) set up loop-1
b) generate the data: 1 sample from each of two populations.
c) set up loop-2
d) calculate the difference in kurtosis between groups.
e) Save result
f) end loop-2
e) find the p-value
f) save result
g) end loop-1
h) find mean, standard deviation for result in step f
There are three things that you can change in these programs.
1) Change the sample size
2) Change the number of randomizations (Loop-2)
3) Change the number of repetitions (Loop-1)
In general, it is best to set loop-1 to a large value. I often use 100,000 or more. I then leave this value alone. When first writing the program I will use a value of 10 to debug the program.
With simulated data (or with real data) program 1 answers the question about how the number of randomizations influences the estimated p-value. You get a 95% confidence interval and can then decide if that is sufficient. Program 2 is only for simulated data and answers the question about how likely it is to find a significant difference in kurtosis given a sample size of n.
The run time seems reasonable. Sometimes these programs can take hours to run. Start with a small number of randomizations and increase the value. Running overnight or over the weekend is quite reasonable if the computer is otherwise unused - just make sure the program works first.
If you start running longer programs, I would include something like print(j) in the program before the last }. This will print out the iteration that the program is processing and let you know that everything is still working. It also helps you figure out how much longer you will have to wait for the results.
The answers given by Caleb A. Aldridge and Timothy A Ebert have made me to understand some more concepts. Thanks to them for their answers with detailed explanations.
- Badges
- Science topic