Hello, I need some help on what statistical test I should use for my data analysis. I have a Data set A, which is an array of 5000 numbers, all of which are zero, and a Data set B, which is an array of random continuous numbers that do not necessarily follow a normal distribution. Both data sets can be plotted on a histogram for visual aid. Data set A is my "ideal" and Data set B is my "measured" - I would like to compare the similarity of Data set B to Data set A (ideally it would be a single output figure such as a % similarity). I would then go on to test another Data set C (the same style of array as data set B - it does not have normal distribution and is continuous numbers) and compare its similarity % with Data set A. I would then be able to make a "ranking" on whether Data set B or Data set C was most similar to Data set A. Some of the considerations: - The similarity value has to account for the shape (ie. the histogram of Data set B will rank with a higher similarity % the closer it is to a vertical straight line as shown in Data set A) - The similarity value has to account for x axis distance on the histogram (ie. the further from zero the poorer the % similarity to data set A) - The weighting of each has to be equal (ie. neither the shape or distance on the x axis is more important) - Because the weighting is equal, if data set B was a straight line at -5, it should have the same % similarity to data set A if it had been a straight line at +5. - the order of the values in array B does not matter
I'm essentially trying to rank data sets against the "ideal" data set A (but taking into account non normal distribution, histogram shape similarity, distance etc). I have no idea what test to apply that can give me a % similarity to the ideal under these conditions.
Thank you so so so much.
You can use the Kolmogorov-Smirnov Test, and the D-statistic is pretty much doing what you are asking.
Andrew Robinson thank you for replying. I had considered the KS test d statistic but Im not sure how it would work given the fact that my "ideal" data set is a vertical line at zero - (the data set I'm comparing it with has a similar stack at zero but with some side lobes and outliers). Since the d statistic uses max values I feel it could unjustly give a bad representation of the similarity (for instance if there is a very large outlier in the histogram but all of the other values are at zero then I believe its quite similar to the ideal, however, I think the KS d statistic would show poor similarity since the max value would be used) - I may be misunderstanding it though a i don't have much experience with the KS test
In that case, perhaps what you really want in Kullback-Liebler divergence, good luck making that answer interpretable to others however
Andrew Robinson - thanks again for replying:) I've actually been looking today at the KL divergence and I'm considering it but my experience with it is very limited so I'm a little worried I'm misinterpreting its use and using it wrongly. My understanding is that it is a measure of the difference between two probability distributions. I think if I was to normalise my two histograms then they would be able to be considered a probability distribution? ( bearing in mind one of my histograms is still just a straight line at zero). (Thanks again for your help it really does make a big difference)
Use both KS and KL but also remember percentages and elasticities are possibilities (depending on the type of data)
The quirky thing about KL divergence is that the "distance" changes based on what you reference against, e.g. information is proportional to the degree of surprise. Information theory is both fascinating and weird where essentially the more strongly you expected an answer the less information it conveyed.
You need some advice and are wondering who you should talk to. There’s less to be learned from people whose responses you can predict.
I suppose you can still use KL divergence if you are always referencing the distribution against the same thing (like using data set A to compare data sets B and C in your original question)
Andrew Robinson - thank you that's really great! Looking at the KL divergence, I'm wondering how the features of the histograms are weighted in their importance/contribution to the final KL divergence value? For instance, how are the side lobes and outliers accounted for/what is their influence on the KL divergence value when referencing the same data set A? Especially since the result is any value greater than or equal to zero - I can see why you mentioned the difficulty in making the answer interpretable
- Badges
- Science topic