It depends on what precision you desire.

Triplicates are relly too few measuremnts to assess the precision of the instrument. I suggest to do an experiment with more replicates.

You can then estimate the variance (V) of the replicates. The variance of the means of n replicates (Vn) is then given by Vn = V/n.

It's a s simple as that. Now it's up to you to find out what Vn is acceptable for your purpose.

Example for your given values (note that just 3 values are actually too less to get a sufficiently good estimate of the variance!):

V = 0.0001443

the variance of your triplicates is then

V3 = V/3 = 0.0000481

and the variance of duplicates is

V2 = V/2 = 0.0000722

You may also express the variability as standard deviation, what are simply the square-roots of the variances.

Calculate your repeatability and judge from that. You know your "average" error variance then and thus you can calculate how your measurement will improve should you increase repeats. 

It depends on what precision you desire.

Triplicates are relly too few measuremnts to assess the precision of the instrument. I suggest to do an experiment with more replicates.

You can then estimate the variance (V) of the replicates. The variance of the means of n replicates (Vn) is then given by Vn = V/n.

It's a s simple as that. Now it's up to you to find out what Vn is acceptable for your purpose.

Example for your given values (note that just 3 values are actually too less to get a sufficiently good estimate of the variance!):

V = 0.0001443

the variance of your triplicates is then

V3 = V/3 = 0.0000481

and the variance of duplicates is

V2 = V/2 = 0.0000722

You may also express the variability as standard deviation, what are simply the square-roots of the variances.

Dear Alvin, thank you for your swift answer. Do you mean something like ANOVA? Would Single-factor ANOVA in MS Excel do the trick? What should my P-value be?

Why not do a power analysis? You have a pre-know variance for a certain sample size. You also should have an idea of the variation you want/need to detect in your experiment (alpha and beta) you should be able to calculate from those values how many replicates you need to take to achieve a certain precision with a given confidence. You will be able to find this kind of analysis in the standard packages of R.

Hope that helps.

Cheers

Andreas

Jochen's reply is very good. Just to add to it.

If you have several sets of 3 measurements on different samples then you can use all of these to estimate the variance of repeat measurements of the instrument. If you fit a one way ANOVA to the data, with a term for each set of three measurements then the resulting residual variance is the one you want. Note that I am assuming that you variance is fixed whatever your actual measurement, e.g. it is not proportional to your measurement for example.

You can then estimate the variance of the mean of 'n' replicates exactly as Jochen states. Assuming normality, 95% of your estimates will lie between roughly +/- 2 times the standard deviation (sqrt variance). You need to decide what level of precision you are willing to accept.

Note that accuracy and precision are not the same thing. Think of throwing a set of darts at a dart board and aiming for the bulls eye. You are accurate on average  if your darts spread around the bulls eye. The overall spread however is your precsision. All close to the bulls eye is precise. All spread out is less precise. You can be precise but not accurate, e.g. all you darts are close to the treble 20 yet you were aiming for the bull. In a lot of ways, accuracy is harder to measure than precision.

Just to add a consideration to the answers of Jochen and Paul; note that your estimate of precision is for the method, not only for the instrument. So for example, each subsample to be read must be equally representative of the sample for you to be measuring the precision of the instrument. Otherwise imprecision in your manipulations will add up to the calculations of your instrument's precision. After you established the precision of your instrument, you don't need to perform duplicate readings. You can just measure your sample one time and report the precision of the measurement.

Note also that you can always achieve a better estimate of your variable using replicate readings on subsamples. That is, using a same protocol, you can average out instrumental variability. So for example, instead of establishing the precision of a single measurement, you could establish the precision of the average of two measurements, and always perform your reading in duplicates. There are mathematical manipulations with which you can calculate the precision of the average of size n, considering you know the precision for a single measurement and the distribution that best fit your instrument variability. As a rule, however, the higher your number of replicates, than the closest the instrument variability can be described by the Normal curve.

As Alexandre said, your error is a combination of machine error and methodology error. It is possible that some of the machine error is variable depending on other uses. You use the machine on Wednesdays, and other people use the machine on other days of the week. However, the samples they process change the machine and influence your error rates. One example would be column performance for chromatography (GC, HPLC). To guard against this, you make a standard sample that is run frequently. You then look for changes in machine output for this standard sample. Understanding the equipment requires an experiment whose sole purpose is to test the equipment.

The problem is, as Jochen states, that three replicates are too few to assess the precision of the machine. This is a standard sampling problem, where you have withdrawn three observations from an unknown distribution. Even if we assume that the population is normally distributed, a single draw of three observations from such a population could be highly distorted. You have no way to assess if your three observations are close to the mean, or if one or more are from the tails of the distribution. The only way to tell is to take more subsamples.

1) I just want to estimate the error, and I don't care how much error to attribute to the machine and how much to my methodology.

     In this case I would take three or four samples. I would then subsample each of them many times -- I have no idea about the costs, but say 100 times each (1000 times would be better). With these data, use a computer to randomly select 2 observations. Have the computer do this 100 times, and each time calculate the standard deviation. You now have 100 observations for the standard deviation. Take the average of these 100 samples. Repeat this process but select 4, 8, 16, 32, and 64 samples (if you did 1000, then you can spread this out more and your results will be more accurate). Plot the data. It is likely that increased sample size will result in decreased standard deviation. With three replicates you can use linear regression to estimate the sample size necessary to achieve a target standard deviation.

    Note: this test shows what the expected outcome should be. Especially with a sample size of 2, your actual results may differ greatly from the expected outcome. As sample size increases your observed results will probably be ever close to the expected outcome.

2) If you want to separate the methodology error from true machine error, try doing the above experiment with four people. Each person does the experiment described above. Person then becomes a variable in the model. If you also had multiple machines then you could figure out the person x machine interaction if there happened to be different models with slightly different designs.

Paul's answer provides another aspect to consider. However, it is "easily" addressed using standards that can be verified using other equipment. Of course, you will need some method to verify that the problem is with your machine and not the standard. You could also make your own standards. Do this multiple times to estimate the error in the standards.

This is all probably more work than you had in mind. So how about running all your samples once. Do the data analysis and look for outliers. Anything that is more than 1 standard deviation from the mean will be resampled two more times. Of course this assumes that storage of the sample does not result in degradation.

Have you examined what might be the sources of your error?  Is there human error?  Decompose the measurement process to determine the source of error.

If the distribution of error appears normal, then it may be okay to go from three to two and accept the increase in variability around your measure given the decrease in the sample.

However, if there are different sources of error, you may want to consider not using the mean but the MEDIAN.  The median is a more robust measure of centrality than the mean when you have multiple sources of error.  Example:  swim timers take three i.i.d times of a swimmers swim time, but they don't take the mean, they use the mean.  There are at least six sources of error:  each timers perception of when the swimmer touched, their ability to hit the timer button the instant they perceive the swimmer hitting the wall, their reading the number out loud off the timer, the recorder correctly recording the time read out loud, the data entry person reading the time, and then the data person entering the time.  Most of this error is normal - but occasionally somebody makes a gross error - and you end up with three times of 21.55, 21.32, and 31.43.  So if your errors are not normal, consider using the median when you have small samples, and you might want to stay at three.

In a way, accuracy relates to the number of measurements (or experimental replication) as previously implied. Look into the Central Limit Theorem and Chebyshev Inequality. There is an example in  S. Ross (1994), "A First Course in Probability" page 393, 8th edition (2010)... 

If you have triplicate expt. compute Cronbatch alpha measure of reliabilty

Hey Hamid, all aspects of repeated measurements, accuracy, repeatability and reproducibility is focused on by industrial statistics and quality control. You will find explanatory material and guiding about how to plan, execute and calculate experiments on different sources of error (material, operator, material) in Douglas C. Montgomery "Design and analysis of experiments".

In R there are some packages which you can use to do so-called gageR&R or GaugeR&R analyses. Some of them even help you to plan randomized experimental designs.

HTH

To generate more-accurate means, one can also objectively eliminate outliers via Dixon’s nonparametric, statistical Q-test:

Hawkins, C.E., and J.H. Niewahner. 1983. Data analysis, graphing and report writing. 1st ed. Mohican Textbook. Loudonville, OH. 92 pp.

Wikipedia. 2015. Dixon's Q test homepage, http://en.wikipedia.org/wiki/Dixon's_Q_test.

-Bob Vadas, Jr.

@Robert,

I can see great abuse of this approach. Ok, so the Q-test in the Wikipedia article only allows rejection of one data point in a data set. However, once one rejects one data point that we don't like, just one more can't hurt, and one more. If rejecting one data point improves accuracy, rejecting two must be better? It is just part of how people are.

It is also important to note that the Q-test assumes an underlying normal distribution. Yet, if you have five data points it is likely that the test for normality will not result in rejection of the null hypothesis. So it is more likely that you can use the Q-test on small sample sizes where the loss of another data point will have a greater impact.

What might be helpful is to give some case where this approach makes sense. I felt the same way about imputing data (the "Do Not Do This" feeling). However, once explained I could see why this is done. I was always taught that the data are the data, and you don't discard inconvient observations.

That said, there are sometimes good reasons for eliminating data. Say you discover that every time the air conditioner turns on there is a power surge that adds about 0.05 to all your readings. I could then monitor line voltage and use those readings to reject specific measurements.

The Q-test is well-established in the literature (Hawkins and Niewahner 1983) and has been regularly used by analytical chemists who are concerned about contamination. Similarly, our WA work on stream-channel widths has revealed that toe (bench) type hasn’t always been consistently measured, so we want to weed out those problematic measurements. If you’ve got  more than 1 outlier, the Q-test normally won’t be significant, so nothing can be eliminated. In cases where the outliers are at both ends, we have less concerns about the arithmetic mean being biased. So this test is valuable when you’ve got “contamination” at one end of your data set.

So with three readings for every data point, there are several choices.

1) take the mean (which is what you are doing now)

2) use the middle value, and discard the other two readings.

3) use the mean, but test to see if one of the observations is an outlier. If it is, then delete that value and recalculate the mean.

Option 3 assumes that there is some unknown event that can distort machine readings. So we need to see if all three readings are close together, or of one is unusual. Deleting the unusual reading will improve overall accuracy.

One could apply this approach for each sample, and delete a measurement if it is greater than one standard deviation from the mean (or something like this). Alternatively, one could calculate the standard deviation for each sample, and then calculate a mean from all measurement that were made to find the expected standard deviation for any observation. One could then go back to each observation, use the global mean (of standard deviation) to identify outliers. This approach would assume a constant machine error rate across the range of observations.

Overall, I suspect that option 2 is the worst choice. Option 3 will be better if there are artifacts in your data, but you don't have enough data to prove that these are artifacts rather than "natural" variability.

Several people have suggested some variation of "calibrate the equipment" and "run standards." Without these methods how do you know that the readings from the machine don't drift. Sample 1 reads 0.136, but if you were able to run it again 2000 samples later you would get 0.175. Or you could get 0.136 in both cases, but in the first case the standard deviation is 0.003 while in the latter it is 0.102. In running many samples there is also human drift. Accuracy improves as one become more familiar with the equipment and internalizes the method, while accuracy declines as one gets tired and bored. The overall result is ??? Understanding these issues may be a better way for you to justify reducing the number of readings from your unknowns.

The simple Q-test doesn't depend on standard deviations, and is definitely more conservative when sample size is minimal (i.e., N is closer to three). Also, if one is concerned about observer fatigue/inexperience or differences among workers, then remeasuring by the same and/or different people makes sense, as we've done for stream-toe widths.

I apologize if my previous post suggested that Q was based on standard deviations. These tests can be made more or less conservative, though usually the probability of making a mistake are unknown. So to make the approach with standard deviations more conservative use two, or three standard deviations -- or hundreds of standard deviations if desired. Of course, the more critical you make the test the more likely you are to retain observations that should have been deleted. One web site had a app that calculated a critical Q through Monte Carlo simulation. One could do the same thing using standard deviations. It might be fun to compare the outcomes.

The Q-test is based on the range of the potential outlier, relative to the nearest observation vs. the total range. So that's why two outliers can't be eliminated via this nonparametric test.

Q=(Xn - Xn-1)/(Xn - X1)

Why could one not generalize to

Q= any (Xi - Xi-1) for all i=1 to n. This will trim off all values where there is a large gap between them and the bulk of the observations. If you have a bimodal distribution this may trim off the minor peak.

Or why not simply reapply the test. I have a normal distribution mean 0 standard deviation of 1 with two unusual values. Xn=2775 and Xn-1=153. Clearly the first value is removed, but if one would reapply the test one could also remove the second value as well. This might happen if the person entering the data forgot to type in the decimal point and 2775 should have been 0.2775. You could also get this with an immortal creature where the bulk of the population is less than 2 years old, but you happened to catch that rare individual that was 2775 years old.

There is an applet for the Q-test (http://www.chem.uoa.gr/Applets/AppletQtest/Text_Qtest2.htm) and a link that describes how the applet works (http://www.chem.uoa.gr/applets/AppletQtest/Q_to_p2.htm). For N>4 they use a Monte Carlo simulation to calculate the critical Q values. If one has enough raw data one could use that to make this a nonparametric test. However, it seems like the applet uses a normal random number generator. Consistent with this, the second sentence of the Wikipedia page states that the Q-test assumes a normal distribution. This would make the Q-test a parametric test.

Thanks for the chemistry web sites. Please observe these statements therein: "This test allows us to examine if one (and only one) observation from a small set of replicate observations (typically 3 to 10) can be "legitimately" rejected or not... In case of the detection and rejection of an outier, Q-test cannot be reapplied on the set of the remaining observations." But you're correct that it's a parametric test, albeit it doesn't depend on estimating the standard deviation. The Q-test is definitely simpler than the mentioned Huber method, which doesn't just focus on 3 key data points for calculation.

Yes, but they do not state why. Why is (Xn-Xn-I)/(Xn-X1) valid and (Xn-1 - Xn-2)/(Xn-X1) invalid, other than for the reason that the table of Q values was not set up that way. With a Monte Carlo simulation, one can develop the exact same test using an pair of sorted values. I am not the first to suggest that other versions of the Q test could also be applied. Rorabacher (1991, Analytical Chem 63(2): on page 141) gives several versions where Q is equation r10. There are also tables for critical values for these variations of Q out to 30 observations/replicates. There is also no numerical reason that the test cannot be applied as often as one has data points to reject. So if one is comfortable rejecting one point, why not two. Is there some law that states people can only make one mistake in gathering data? What makes rejecting one good science but two bad science? Is this one value per treatment or one value for the whole experiment?

Timothy, my educated guess is that statistical distribution is adversely reflected if more than 1 data point is removed. Certainly, if you've got outliers at both ends, they will tend to cancel each other out for effects on mean values. If you've got 2 outliers at 1 end, then perhaps the data set needs to be tossed (or at least tested via resampling).

Calculate the standard deviation of the values and see the variability between the measurements is minimum so to confirm that the instrument is accurate in the measurement.

Accuracy is defined as the difference between true value (Theoretical value) and measured value.

1. Measure of central tendency: Mean can be calculated. Ideally should match to true value or theoretical value, if it is known in advance. Otherwise assuming normal distribution, estimate interval.

2. Measure of spread: Standard deviation.

How about this:

Everyone who is part of this project should mix up 6 sets of standards. I would make sure that for each set started with a clean bench top. I would reset all the pipets, I would use different glassware. I would then measure each set five times today. I would then measure three of the sets every day. I would measure the other three every week. In an ideal world, the 10 ppm solution would measure at 10 ppm for all of the sets, and would measure at 10 ppm every single time until I ran out of solution to test. I could then mix a new batch and it too would read at 10 ppm. The question is then how quickly do the readings depart from this? You might also ask how much error is attributable to the machine versus how much to the operator (you versus any of your three assistants) versus how much to error in making the standards. How fast do standards degrade?