If I have understood correctly, you want to compare two empirical distributions. The distance between two distributions can be measured with Kullback-Leibler divergence. To test the null hypothesis that two empirical distributions come from the same theoretical distribution you can use Kolmogorov-Smirnov test, Cramer-von Mises test or even chi-squared test (the last can be used to compare histograms). If you want to obtain confidence intervals around natural distribution with respect to observed distribution, you can perform jacknifing (sampling natural distribution without replacement) with sizes of drawn random samples equal to the size of observed sample (n=30). This approach is valid though I am not certain about its high precision from statistical point of view. Theoretically, if all possible samples are checked, then confidence intervals can be estimated without additional assumptions (contrary to statistical tests)

To provide some context: I am studying the preference of archaeological sites for certain geomorphological/topographic variables. I wish to determine if the preference of sites to be located in regions of, say, low slope (% observed sites located in low slope area) is actually statistically significant vs the natural distribution of slope values throughout the study area.

If I have understood correctly, you want to compare two empirical distributions. The distance between two distributions can be measured with Kullback-Leibler divergence. To test the null hypothesis that two empirical distributions come from the same theoretical distribution you can use Kolmogorov-Smirnov test, Cramer-von Mises test or even chi-squared test (the last can be used to compare histograms). If you want to obtain confidence intervals around natural distribution with respect to observed distribution, you can perform jacknifing (sampling natural distribution without replacement) with sizes of drawn random samples equal to the size of observed sample (n=30). This approach is valid though I am not certain about its high precision from statistical point of view. Theoretically, if all possible samples are checked, then confidence intervals can be estimated without additional assumptions (contrary to statistical tests)

Dear Robbi, for me it also looks like Bootstrap is the way to go - especially if you have a big data set for the 'natural distribution'. So you have a low risk that this distribution is not really representative.

If I understand your question right, the only thing you need to botain a confidence intervals for your natural distribution, right? And I assume that you aggregate your individual measurements to discrete bins (e. g. all measurments between 4.5 and 5.5 are put summed up in one bin spanning from 4.5 to 5.5 ).

So, I suggest you to generate from this distribution a large number - let's say 1000 - 'synthetic' natural distributions each consisting of the same number n of measurements as your original natural distribution (using sampling with replacement from the various bins - e.g. probability to obtain a measurement between 4.5 and 5.5 equals the fraction that the respective bin has compared to the sum of all bins). For your 1000 synthetic samples you can now estimate for each bin the average value and also within which values +- this average you find 66.7% of the values (if you are looking for 1 sigma). This should give you an idea about the uncertainty of your natural distribution.

If you are also interested in the uncertainty of your n=30 'observed' values, the answer above from Oleg looks reasonable, I think.

Hello Robbi Bishop Taylor, Your approach is fine. however that variables should not be spatially wide spread. . wher ethe data can be analysed with accuracy.

Your approach is fine. I would do the same.

Thankyou very much for the answers so far. Oleg is correct; I wish to obtain confidence intervals around natural distribution with respect to observed distribution. I expect these confidence intervals to vary depending on the sample size n of the observed distribution.

(I have little to no uncertainty on the values of the natural distribution; I am dealing with spatial data that I can randomly sample a large number of times to obtain a representative sample.)

Jackknifing sounds promising... I however aren't exactly sure how to implement such a technique myself. I have immediate access to Minitab and Excel; could such analysis be conducted within either of those software packages? Will I be required to aggregate my results into specific bin sizes in order to produce confidence intervals for different values of the variable being studied?

Dear Robbi, this task would be a nice learning example on how to write a small program using any progaming language. If you have also immediate access to webspace (e.g. via your university account), you could even implement the necessary alogrithms in PHP and make it accessible to the rest of the world. It is really easy - just contact me if you need advice.

From the Minitab webpage I would assume that this program can also somehow do the job, but I have no experience with it.

I don't know how to implement this without some programing. Though you may aggregate your results into bins, it is not necessary. Look, make a lot of runs, drawing 30 elements from your natural distribution. Let each run be represented as one column. Sort each column ascending. Then elements in each row you have got represents the same quantile of distribution and are related to the specified point in your observed distribution. For each row, find upper and lower values such that 95% of row elements lay in the interval (symmetrically)- this will be your confidence interval at current point. Make this for all 30 rows and you'll have 30 upper values and 30 lower values. Plot these values versus values of observed distribution and you'll get quantile-quantile plot. Such approach is more exact and informative when the goal is to compare distributions.

To obtain bins, just make each column not 30 elements, but N (N

Hi Robbi,

isn't there a flaw in your research question? If your archaeological site depend on the natural surface structure of a geographic area (e.g. nobody would bury a person on a steep hill), the distribution of your sites are most likely not "random".

Late to this, but if I only had Excel I'd use the Chi squared test and generate a test value as follows. Assume you have i bins. For each i, take the difference in bin event amounts between observed and expected distributions, square it, and divide by the sum of those two bin amounts. Sum across all i. Use Excel's chi squared function to check the critical value of interest.

Notice that is comparing two binned distributions. Your original question is more ambitious and seems to seek tests for particular values of the support of that distribution. That feels infeasible due to, inter alia, multiple testing concerns.

I think the only way you could do it is to transform the probability distributions into cumulative distribution functions and then test equality of the distributions above (or below) a particular support value using the K-S test. You could end up saying sth like "for elevations greater (or less) than X, there is a stat sig difference in the distributions with fewer (or more) than expected." Stills feels a bit iffy, because essentially you could end up testing 30 times along the support without adjusting for multiple comparisons. You should end up having a few "positive" results just by chance...

Hello, again, Robbi. I, like Marco, am coming late to your question, but I wonder, something like Oliver, about your question, the limitations on factors that might be influencing the values of measured variables, and from those, whether or not you are seeking a bigger hammer than appropriate for the nail you are trying to strike.

Your question suggested, Oliver offered a comment and example, and if I am presuming correctly, that you are interested in looking at, say, human activity sites of archeological interest. Among such sites it would seem to follow that slopes within a certain range will have been preferred over time. Your question would seem effectively to boil down to what are the slopes (or aspects, or ...) that have been preferred in a given area over the archeological time frame under consideration?

You anticipate that you can conveniently obtain a large number of reliably random measurements from which to determine the distribution and variability of the natural background parameter of interest, so, presumably your natural background sample size will be robust enough to "bin" in almost any fashion you wish. Further, you should be able to obtain a mean and "tight" confidence interval for the whole natural background population and even relatively small fractions of it, e.g., "bins" within it.

On the other hand, the number of measurements for your investigation sites will be more limited. It seems to me that it would be the case that if there was a substantial preference for a certain slope, then there is a reasonable likelihood that such a preference will be immediately apparent through examination of mundane, simple statistical summaries, i.e., comparison of max, min, mean, std. deviation/confidence interval, for the whole natural background data set and investigation sites data sets. It is possible, of course, that the two data sets could have identical basic statistics and still have very different distributions, but that would seem unlikely and can be confirmed by additional analysis, if needed. My point here is that it would seem likely that you should be able to discern relatively quickly and easily whether or not there is anything to look for by doing some simple gross summaries before concerning yourself with more sophisticated statistical approaches.

In the event that you determine that the gross sample data sets are statistically different, then you can proceed to fractionating the sample data sets into appropriate subsets of interest. Initially you might take the "least-work-first" approach and simply bifurcate both data sets into upper and lower halves, then run and compare (investigated to natural) basic stats on each, and repeat bifurcating each half-data-set, then each quarter quarter-data-set, and so on, until you can see the region of most interest/contrast. If necessary, multiple adjacent data subsets that comprise that region can then be reassembled and the approach repeated with selected subsets as seems fit. Each such step will generate means, confidence intervals, etc. Though this approach may not be statistically sophisticated, it is relatively easy, quick, and very importantly, transparent, enabling you and any who would look at your work to see what your data says directly. Also, this approach gets you around the question of trying to determine minimum "bin" size, which would seem to be the point where a single "bin" no longer contains enough data points to be of statistical interest. Using this simple approach, when your compared data subsets get too small statistically, your investigation site and natural background confidence intervals will overlap, i.e., you have subdivided your data set too far.

This approach is not dissimilar in principle from writing a program to bin your data and running histograms with stats on each bin. I just have always found it helpful to have a closer look at the data, and this simpler, relatively efficient, more hands-on approach forces such a closer look.

You, of course, are aware of the possibility that local topography may have changed between now and the time of occupancy of the sites of interest, and that human concerns, attitudes and, hence, preferences, for certain slopes, aspects, etc. may have changed over time. Given those two possibilities, it seems likely that you will not be able to get highly reliable data on, for instance, slope. That is, you may be able to reliably designate at-time-of-site-occupancy topography slopes on a scale of say 0 (level) to 10 (vertical), but not to the nearest degree of angle. It has been my experience that in comparisons of low resolution data, the direct-close-look approach I have described is helpful both as an investigative and presentation tool. Some difficulty can be anticipated if the natural background topography is relatively "flat", but that actually resolves to a resolution problem rather than an analysis problem.