I have performed a hypothesis test (Kruskal-Wallis) and estimated the p-value using Monte Carlo. My sample size was 500.

Now, I am trying to keep simulations fixed and only changing sample size to show a convergence of the test statistic and the p-values.

From what I understand, these are the steps:

1) Run the simulation with a sample size of say, 5, picked from the initial sample of 500.

2) Repeat #1 ten thousand times.

3) Repeat #1, but increase the sample size to 10,15,20,30,50,100 etc. then fill in or extend to larger sample sizes as needed.

My question is, for each simulation do I randomly pick a new sample of n=5 or do I do permutations on the same sample?

For example, if my data for the first n=5 is 3,5,10,20 and 30, do I do different permutations of this sample for each simulation? Or do I need to pick a fresh sample of 5 and then do a new simulation?