I'm doing a project, where I scan over a sample recording a spectrum at each position. Although the sample is supposed uniform, there's a large difference in the overall spectral response of each individual position. So, recording a single point does not give a correct representation of the sample. I compare to an average of 1000 points.
I wish to make an analysis on how many points to include in an average before it represents the sample correctly. So far, I have made an analysis in MATLAB where I take n point out of the 1000 point total measurement, find the average spectrum and find the correlation to the 1000 point average. A problem is, that I also observe a difference in correlation depending on what n point I average over. That is, a 10 point average is not the same as a 10 point average another place.
How can I make an analysis including all the data, where I find the lowest amount of points anywhere still resulting in a representative spectrum of the sample? A "representative spectrum" can be a spectrum with a correlation coefficient above 0.9 compared to the 1000 point average.
The central limit theorem might help. As far as I understand, it implies that the distance between two functions, for example measured as the sum of the difference of all points, will approach a normal distribution, when using enough spectra to average. You can test whether that is true for your samples, meaning whether the difference of your spectra to the average spectrum is normaly distributed, or if you already need to average a few spectra together to make their distance to the 1000-spectra mean normaly distributed.
Afterwards you can use the Variance behaviour of a normal distribution, meaning that the variance of a mean will be inversely proportional to the squareroot of number of elements used. Since in a normal distribution the variance tells you exactly how likely a sample is to be within the variance-limits, this should enable you to calculate the number N needed to reach a low enough variance of the mean, so that, say 99% of N-averaged spectra are within a chosen limit.
You can also test this brute-force style and determine a histogram of the difference between all possible subsets of N averaged curves to the 1000-curve mean (as an estimate of the expected value) and see at which value of N this is narrow enough. This should be about the same as for the above calculation.
If you observe different spectra (I assume optical spectra or is it an other type of observation?) for different points (probably focal areas or position of electrical contact) respectively regions at the surface of a sample, there is simply no such thing like a "true spectra" that could be obtained by averaging. You can average over any number of spectra that you like but it will simply represent the respective averaged spectra. Your observation simply means that one or more of the following points are true:
1st: Your sample is inhomogeneous and neither a glass nor a single crystal, since only those two are really homogeneous.
2nd: The surface of the sample is not sufficiently planar (well enough polished)
3rd: (relates to 1st) Your sample is microcrystalline and each of the microcrystals has an other orientation and since observed spectra depend on orientation, you see many different ones. In that case you may be able to derive some values related to the microcrystalline structure of your sample from the variation of the observed spectra.
4th: (relates to 2nd) you surface is dirty or your sample has inclusions near the surface.
I could come up with more help, if you would tell me what sample you are observing with what instrumentation respectively set-up.
With kind greetings
Wolfgang Grill
Thanks for both answer, I appreciate you took the time to give me some input. I didn't see the first answer, so I apologize I never reacted to that. The problem was a part of a master thesis project that has since been handed in and I'm no longer working on it.
I believe Wolfgang Grill has a point in saying that the "true" spectrum is a misunderstood goal to have, as the spectra (and yes, I meant optical) obviously will be affected by the structure of the sample. I don't believe it was contaminated, but it was most likely not completely homogenous in its particles size and orientation. So instead of various post processing routines, that would only be relevant to one single sample, as the structure would be different for another, it is probably wiser to prepare the samples in another way.
In the end I used Pearson correlation coefficient to determine how well a subset of random samples of different sample sizes compared to the total average spectrum. Thereby I could conclude how many individual spectra had to be averaged for it to be "well enough" correlated to the total average. The problem is that the result is only relevant for one specific sample, and as soon as another sample is prepared, the statistics change. So all in all, the general process in the experiment has to be revised before it can be used further.
Again, thanks for you input!
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics