To give some background: I have a measurement device which can measure a concentration to plus/minus 1% vol/vol which is NIST traceable ("method A"). We then employed a novel technique to measure concentrations ("method B") which we compared against method A during a transient event. Over the course of the transient we collected several thousand measurements using both Method A and B.
At the end of the trial, how do I establish the accuracy of using the novel method? I am leaning towards using the Root Mean Squared Error for each measurement and the Standard Deviation of the errors (at some confidence interval). Is this appropriate? And how do you incorporate that the comparison is to Method A, which also has a known absolute uncertainty? How important is the mathematical error propagation versus experimental data with so many data points?
Many thanks in advance.
If your data is a time series, you may need to remove trend or seasonal effects first. Then you can check the noise level of each measurement by comparing the Root Mean Squared Error.
One tip apart from statistics: As you indicate that you collected data "during a transient effect", I conclude that you indeed have 2 time series. As the response time constants of methods A and B may differ, directly comparing between the values of the same time stamps might be sub optimal. So in your processing include a check on response time shift. E.g by checking whether there is a time shift value for which the difference structurally minimizes.
Dear Thomas Kirton,
When a new measurement technique is developed we need to estimate and assign all the associated parameters such as resolution, precision, repeatability etc. Accuracy of measurement system is really critical. I personally suggest to use either of two approaches-
1) Estimate all the measurement affecting parameters those may contribute to error. This can be done if all sources of error are known. Consider this as type B. Type A can be estimated by repeated measurements already taken. Then finally overall accuracy can be estimated. But this method becomes difficult if the sources are unknown. You really do not need any reference such as A.
2) Other approach is relatively simple. Calibrate the method by your traceable method A for the full measurement range. Check the maximum deviation in measurement. It is expected that the measurement deviation be more than 1%. Use correction for any systematic errors. Now use it as type B and use type A for overall accuracy.
I think you can do residual analysis. The difference between the measured values of methods A and method B is the residual. Ideally, the residuals should be approximately normally distributed around zero. If the residuals are not centered at zero, there is a bias in methods B measurement with respect to method A. In addition, you can subtract the uncertainty (variance) of method A from the total uncertainty (variance) of the residuals to obtain the standard uncertainty (precision) of methods B measurement. The bias and standard uncertainty will provide a picture of the quality (accuracy) of the method B measurements.
- Badges
- Science topic
- Similar topics
- Solid State Physics
- Devices