The errors of a model are the devotions of the observed from the predicted values of the model. Variance is an average of the summed squares of these errors.

In a very simple model all the variance of the model is error variance or unexplained variance, but if you have a complex model you 'explain' variance by adding additional parameters that reduce error. Explanation in this context is used in a narrow, technical sense (that the particular value of a parameter reduces the unexplained variance).

More generally the terms error and error variance don't necessarily imply error and can themselves be considered the sum of many unknown sources of variation (measurement error, sampling error and/or thousands of other influences).

James -

We can have modeling error, sampling error, nonsampling error, depending upon context, and the error can be from bias and from variance. 

With a model, if we leave out a part of a model, say a pertinent independent variable, this can technically lead to bias, though if the model were assumed correct, the estimated variance of the prediction error will be inflated. 

There is a distinction to be made between (1) a standard deviation of a population, or in regression, the standard deviation of the residuals (or random factors of the residuals in WLS), and (2) the standard error of an estimated mean or total.  The former is a parameter of a distribution, the latter is reducible by increasing sample size, as long as the increased sample size does not increase nonsampling error (say, measurement error) too much.  The variability of a population (say, the standard deviation squared) is not really an error, but higher standard deviations  lead to higher variance for estimated means and totals, though as I noted, those may usually be reduced by increased sample sizes. 

The mean square error (MSE) is the sum of bias-squared and variance.   It's square root (RMSE) is an important measure of accuracy.  For models, there are three parts to the MSE because variance is broken into two parts, one strictly for the residuals, and one for the coefficients.  When we assume model-unbiasedness, we measure an estimated variance of the prediction error, then assumed to be the MSE.  It has those two error measuring components,  though both are based on the standard deviation from the residuals (or really random factors of the estimated residuals).  I finally realized recently that when you later estimate model-failure by validating the model with test data and/or later obtained information, that the difference is not really the bias-squared, because the estimated variance of the prediction error used an estimated standard deviation of the residuals for a model assuming there was no bias.  At any rate, the real expected prediction error, as it is called, though it is written such that it should be called the expected square of the prediction error, or mean square error (MSE), is made up of the two variance components noted, and a bias-squared component.  The odd thing is that the part of the variance strictly for the residuals is actually from the standard deviation of the residuals or their random factors in weighted least squares (WLS) regression.  This part represents error which cannot be eliminated.  See the first part of page 6 in Galit Shmueli's paper:  https://www.researchgate.net/publication/48178170_To_Explain_or_to_Predict?ev=prf_pub

The term "MSE" may possibly be used more often when not considering regression.  Consider that under "consistency" in the following, it is important for MSE to approach zero with larger numbers of observations:

http://www.ms.uky.edu/~mai/sta321/mse.pdf

The EPE noted in the Galit Shmueli paper, and commonly noted in the literature on "statistical learning" has a nonzero lower limit, the "irreducible error":

http://scott.fortmann-roe.com/docs/BiasVariance.html

Another note here:  I think it is technically incorrect to call a residual an "error."  A residual is the difference between an observed value and its predicted value.  You might call it a prediction error. 

There are all kinds of error, and attempts to measure their overall impact might be referred to as trying to measure accuracy. Variances are a very important part of this, but let's not forget bias.  In Fortmann-Roe's document linked above, he shows a standard way of explaining bias and variance, using a dartboard.  This is an analogy which is generally applicable to all kinds of attempts to measure accuracy.

Cheers - Jim

Article To Explain or to Predict?

Thanks for the answers so far, I note they concentrate heavily on the technical side. Jim kind of hints at my main purpose for asking the question when he says:

'Another note here:  I think it is technically incorrect to call a residual an "error."  A residual is the difference between an observed value and its predicted value.  You might call it a prediction error. '

I agree with Rafael generally that variation due to poor experimental control is error. Consider, however, the variability in sub sampling. this variability might reflect poor experimental control. If however you subsample exhaustively exactly the same parameter is a measure of population variance not a measurement error.

What do people think of Jim's comment that I have quoted?

I'll explain in more detail what has prompted this question but don't want to bias the answers initially.

James there is a lot of discussion around these questions in science.

To your first question first there are a series of guides published by the BIPM (the original and premier physical standards organization) that provide I believe the most comprehensive and consistent view of terminology for scientists.

The VIM (Vocabulary in Measurement) gives the meaning of words like error, variance, uncertainty etc which the GUM (Guide to Uncertainty in Measurement) provides the context for application. While focused on measurement scientists the guides are just as applicable to modelers and provide a common and clear playing field. While these documents will not define every term you may be interested in I believe that if it is defined here we as scientist as beholden to use that definition.

 Regarding your second question "residual" this is outside the ontology of the VIM, but is a term I use daily in my work on measuring sea-level. In this context it is used to mean the difference between tidal modal (an astronomical prediction of the tides and the real sea-level that includes thermal expansion of the oceans, sea-level rise, meteorological effects etc. So in this context the residual is not a "prediction error" it is a measure of non-astronomical contributions to the true sea-level. As such you could argue it is a measurement in its own right.

My point is I half agree with Jim. In my work the prediction is not trying to model the final sea-level, but it is the difference between an observed value and a predicted values.

There is no universal definition so you need to provide one for your context. And like wise, if you decide as a scientist you don't like the BIPM definitions, you are free not to use them. But I would argue you are obliged to declare you are deviating from them and provide a clear statement of how your usage of the word differs.

http://www.bipm.org/en/publications/guides/#vim

http://www.bipm.org/en/publications/guides/#gum

Thanks to Jane for her useful input and references, again touching briefly on my motivation for asking the question.  Perhaps it is I think it is time for me to expand on the background to this question.

Lets say you take several measurements from a sample and calculate the deviance of the replicates from the mean for the replicates.

Are you measuring sample heterogeneity or sampling error?

Does it matter what you label it? I think it does and it will change based on what you are doing.

I inherited work from another group who had failed to get an analytical method progressed from promising PoC to a validate method. Within 2 weeks I realised the sampling approach they took gave a measurement error of 120% of the diagnostic separation of the case and control means.  Several more months work established that 1000x the area needed to be sampled and brought the measurement variability down to 1%.

The original group had measured the variance in sampling and labelled it as error and sought to control it by excluding outliers to leave a core of consistent signals.

Understanding that initial assessment as sample heterogeneity however leads to a very different strategy. The variance is not an error yet - it is a feature of the sample and therefore the analytical strategy needs adapted to capture this - so the sampling area needs expanded to capture the inherent variability.

Only once the analytical method is finalised does the variance in sampling become an error, but only from the perspective of the reliability of the final result. Terms such as uncertainty and variability are more neutral but maybe sometimes it is useful to give a more defined perspective on whether the variability is an inherent property (sample heterogeneity) or a source of inaccuracy (error).

The same applies, as Jane and Jim suggest, to other measures that are often written off as errors, when in fact they may be useful information when viewed from another angle. I'd love to hear people's perspective and examples on this.

James I think for me you have hit at the core of what the GUM is about. "Validation of the measurement". Just because I take a measurement and have a number does not mean that I have information. Until I understand what has influenced that observation I have little more than a random number.

Sadly I don't believe this is taught very well at school/university. We are use to instruments confidently delivering a number on a display that is unwavering. We no longer see the "doubt" in the wobbling/flickering needle and as a result look no further. We trust that the way we are using the tool we have at hand was foreseen by the instrument manufacturer. (Which from my experience is seldom the case, actually it is seldom the case they do what is claimed).

It is the scientist responsibility to go back to the drawing board and think through the problem. What is it that I want to know? What might be effecting my ability to get that information? A simple example, what is the temperature outside? What do I want to know? Lets say it is the average kinetic energy of the gas molecules in a parcel of air 1m3. This might seem an odd definition for air temperature but in reality it is what we are doing. Remember it is nominally defined as the "shade temperature" in most countries. So we don't want to know about radiative energy from the sun, or conducted energy from the brick wall next door to the thermometer etc.

Your sampling issue is the same it is an issue of understanding the experimental design and what might influence the homogeneity or representativeness of the sample. This is even an issue in the temperature example, is the thermometer positioned in a location that represents the volume of air I want to study.

Sample homogeneity is an issue that a lot of fields deal with but which is seldom discussed. Another useful source of old fashion information in this area is Skewhart who did a huge amount on process control and Deming.

Ultimately your sampling inhomogeneity has to be a consideration in the determination of your uncertainty. What it is called is secondary.

Now however we touch on the issue I struggle with the most, normality in the real world. We spend most of our time, for convenience and practicality assuming that the world is normally distributed. However we know that it is not. My question is how non-normal does it need to be before our assumptions no longer work!

Jane, I think that your point about basic assumptions such as normality are indeed related. I think as scientists we need to constantly challenge our assumptions. and never be lazy in applying the same assumptions in new studies. The sampling issue I highlighted was from a group that would be considered world leaders in their field, and their approach had been developed for a very different application where homogeneity was critical. They never revisited nor challenged their assumptions in their QC procedure to assess if it was still relevant to this new application.

In my own science education not enough emphasis was placed on understanding variation, the range of sources it can come from, how to quantify it and its impact on analytical methods. Too much pressure was placed on publishing proof of concept work (which is easy to get accepted as it is novel) and none (worse, this was discouraged)  on method development (proof of concept now published, the stuff is less novel and less easily published).

It wasn't until moving to commercial sector and the rubber hit the road that it became clear just how important these issues are. Not doing the ground work can mean the difference between people establishing careers and losing jobs or between customers continuing to struggle with a lack of knowledge or benefitting from new insight.

I think maintaining an open mind is critical to ongoing development. In the above discussion this means allowing that assumptions that previously helped don't apply in all circumstances and recognising that variation is not always an error and may need handled differently depending on the context.

James, could agree more. I am involved (immediate past President) of an organization called the Metrology Society of Australasia. It was formed 20 odd years ago by people industry who where facing the same issues as you are discussing. And to battle the on slaughter of ISO9000 which at the time in Australia, and to some extent still, was making business think it doesn't need to measure anything.

Measuring things so that the numbers contain real information is hard, very hard. I agree with you that not enough account is taken of variability in science. For me measurement (which for most scientists should be at the heart of what they do) is the process first of understanding what it is you need to know, and then identifying and testing anything that might interact with what you are after. It is why the GUM methodology is so valuable if you use it properly. It challenges you to look at everything and highlights when you are making assumptions. It allows you to make them but you have to do it knowingly (expert knowledge in the context of GUM). As a consequence you can also knowingly go back to them.

Error as many people use it I don't like, it is leaving science in the lap of the gods!

Good luck with your search for an answer, it is an interesting road to walk and as you say careers are both made and brought down based on how well you do it.