Hello everyone,
I am trying to statistically analyze whether data from 3 thermometers differs significantly. At the moment, because of COVID-19, several control points have come up at the company for which I work. We have been using infrared thermometers to check up on people and to be aware if they have a fever or not. However, we don't own a control thermometer with which we could easily calibrate our equipment, we thought that using a statistical test would be helpful, but at this point, we are lost.
Normally, we would compare our data to our control thermometer and that would be it. Our other thermometers are allowed to have a difference of +-1°C at max when we compare them to their controls; we can't do that now.
What I have been doing is collecting 5 to 10 measurements from each thermometer and compare them through an ANOVA test, and then assessing the results (when needed) by running Fisher's Least Significant Difference test. I don't know if it is right to do so because sometimes the data I collect does not seem to vary a lot (the mean difference is NEVER greater than +-1°C), and even so the test concludes that they differ significantly.
What would be right here? We don't want to work with the wrong kind of equipment or put away operating thermometers without a solid reason, we just want to do what's best to our people.
Could you guys please help me?
Significance testing will not help you here. Your problem is one of informed reliability and instrument validation testing. If you think about it, the question is not that complex and applying ANOVA would just confuse matters.
- You have three thermometers and you don't know if any of them perform entirely accurately.
- You do not know if thermometers provide precise measurements upon retesting - within re-test consistency.
- You don't know if the thermometers perform in a similar fashion in comparison to each other - between method reliability.
The only way to honestly solve your problem is to set an unstandardized tolerance cutoff that can you would then use to determine if it makes sense to continue using these instruments. Cutoff could be ±1 degree C, just as an arbitrary example.
My approach:
- Test each instrument many times on the same standard.
- My favorite statistic is the RANGE. Do any thermometers perform outside of your cutoff? If so, then somebody may be incorrectly told they have COVID, so best to not use it. It really depends on your context.
- There are many wonderful descriptive and reliability analyses than can be done in your situation. Get the median absolute deviation (MAD) of each thermometer and decide if dispersion is acceptable.
- Check histograms of each and look for skew. If it exists, is it acceptable, given the direction?
- Plot the three measurements in a modified Bland-Altman plot with average between the three in the X-axis and SORTED temperatures in Y axis. Preferably you know the temperature reference, so draw a horizontal line in the plot at that point. Otherwise use the mean.
- Visually check the residuals vs the reference. Maybe get a root mean square error of Y values vs reference.
- You may want to repeat the process with another subject or temperature range.
- In the end, make an informed value judgment about the precision and probable validity of the instruments. If the values are too far apart, then you cannot use the thermometers.
This is really the only way to make these kinds of decisions in my opinion. I am open for others' discussion reliability analysis but be wary if others suggest repeated measures ANOVA, two one-sided t-tests, or even Cronbach's alpha. Your goal is to get the most representative assessment of the situation, not p-hack your way to safety!
Following up on @Miky , this is a useful overview
https://en.wikipedia.org/wiki/Bland%E2%80%93Altman_plot
which is one of the most highly cited statistical procedures in the scientific literature
Bland JM, Altman DG (1999). "Measuring agreement in method comparison studies". Statistical Methods in Medical Research. 8 (2): 135–60. doi:10.1177/096228029900800204
Well,
Mainly, it depends on the ACCURACY of the used instrument.
Regards
Hi, We did plenty of statistical analysis in our Research study. You may get some idea from reading this material.
Experiment Findings FACTOR ANALYSIS ON SALES PERFORMANCE OF ENTREPRENEURS IN NOR...
- Badges
- Science topic