It is definitely bad practice to leave them out because conclusions based on the means only are generally misleading. I have seen enough papers claiming 10% improvement or the like, and the error bars would indicate that there is no statistically significant difference between the approaches under test. The problem is that authors simply not know about the pitfalls in simulations (or don't care), and what is worse is that the reviewers are not better, either. But with no one enforcing good practice the overall quality of the publications may drop.

All of this has lead to a quite bad reputiation for simulations studies without any further validations (like experiments, analysis), the trust is (rightly so) low. Some interesting papers on this:

S. Kurkowski, T. Camp, and M. Colagrosso. 2005. MANET simulation studies: the incredibles. SIGMOBILE Mob. Comput. Commun. Rev. 9, 4 (Oct. 2005), 50--61. http://doi.acm.org/10.1145/1096166.1096174

Heidemann, J.; Mills, K.; Kumar, S.; , "Expanding confidence in network simulations," Network, IEEE , vol.15, no.5, pp.58--63, Sep/Oct 2001 http://dx.doi.org/10.1109/65.953234

For wireless networks, there were also comparisons on how well the simulations predicted performance in the real world, with sobering results. Most assumptions made in simulations are quite far off from reality:

D. Kotz, C. Newport, R.S. Gray, J. Liu, Y. Yuan, and C. Elliott. 2004. Experimental evaluation of wireless simulation assumptions. In Proc. of MSWiM '04. ACM, New York, NY, USA, 78--82. http://doi.acm.org/10.1145/1023663.1023679

It depends on the simulation strategy. I mean if the random number generator (used for simulation ) itself was individually verified to meet hypothesis criteria for pseudo randomness (based on say Chi-square test ..there are several other tests too) then i guess it is safe to assume the results to have the same level of confidence as the RNG used. In most cases what happens is people would have used a certain version of kernel (which comes with a RNG) on which some other results have been published who in turn did the statistical tests. so u just trust them!!!

But then if u have developed a prototype in real world for say vanets or video transmission over multihop adhoc network or say a simple routing protocol for Manets. In such cases there are several layers of randomness. First by the nature of losses (influenced by several factors like electromagnetic cavities caused by underground pipes on university campuses Vs what if the experiments were done on a grassy troll?) or nature of mobility (i mean how do quantize mobility??) etc . In such real world experiences you will have to study a large number of tests and make sure that you change only one factor at a time (how do you control node mobility repeatedly..draw lines on the floor so that between two experiments the nodes did move in the same way??? and wireless errors....interference caused by other MIDs.. ). It s going to be one big massive effort!!!!

Finally what if there was a software bug in your implementation itself that caused your protocol to give better/worse results than other proposals.. I don't now if any statistical test can identify this. I mean effect of software bug itself is a constant K (the graph will shift either higher or lower :-) ) and doesn't cause any randomness to impact statistical results.

Manual post-processing (statistical testing) is always worth doing to produce errors estimates or minimum 95%CIs, if the software does not provide this options; tests for normality of distributions are also needed when comparisons are made. I suppose, simply, the any of the above has not been done in many publications because either the authors lack the basics of statistics/probability or have not approached a qualified statistician for advice before submitting the paper, or both! Evidently, the same is valid for the Editors and Reviewers of the journals which, at the end, compromise the credibility of any such published results from such simulations that present only mean or median values.

I guess it depends on what you are simulating. For example, if you are simulating an electronic circuit whose governing equations are well known, it makes little sense to put error bars on the response if the component values are well defined. It _does_ make sense to estimate errors due to word length and operator truncation, but these are usually well defined and as long as convergence of the solution is assured (and tested experimentally), error bars on the solution don't really make much sense.

However, if we are doing an error propagation analysis (because of component tolerances, etc.), then it makes sense, because we are usually looking at acceptable spreads for a circuit response and there will be means and variances associated with the response. Likewise, if we are attempting to characterize noise response or response to random signals, error bars should be included.

In short, if we are describing systems with a stochastic component, error bars make sense. Where the solution is expected to be entirely deterministic, it makes sense to account for error bounds in the solution method and show that these errors tend to zero if certain conditions are met (as in the case of deterministic finite difference and finite element methods, etc).

Just my two cents' worth. You may or may not agree..

Cheers!

Error estimation requires a clear explanation of what kind of error and if there is a benchmark that is acceptable to the community to which you are communicating. It can be difficult to explain what a model is comparing against, and how a reduced error (or improvement against a benchmark) really means your model works better as a result of the research. In climate analyses, for example, errors are often expressed as differences from the previous iteration of the model, not against reality. Errors could be expressed as differences from an accurate spatial observation (for example, atmospheric depth from observations ) or against a time series (error in the ability of the model to capture temporal variability). Many model outputs do not have clear observational correlates that can be used as a benchmark, thus 'errors' are entirely model-dependent, reducing their utility outside of the field.

NASA has a project called the Carbon Monitoring System, that seeks to incorporate environmental observations from satellites into climate models of land, ocean and atmosphere. In this project, the objective is to develop data products that might be useful to policy makers in various fields. The problem that had to immediately be addressed was how to expressing the errors in our models to each other (each discipline has different standards and terminology), and to communities outside of science. 'How good is good enough?' is the question that we seek to address - does the model really provide the information needed to sufficient accuracy? This requires both an understanding of which errors matter in the model (for example, can you simulate seasons in your model accurately) and then conveying these errors simply and clearly. If the community you are trying to communicate with has no idea of their requirements (which errors are important) then coming up with useful measures of model error is very difficult.

Thus to answer the question about error bars - it depends entirely on who you are trying to communicate with and whether you've done the considerable work before hand to determine what your benchmark is. If you know 'how good is good enough' then yes, absolutely, you need to show that your results are 'good enough' to be trusted, cited and emulated or simply used to make a decision. But it is very hard to do right.

Having IEEE papers on VANET in question, I can only conclude that we have a small issue of presenting results of simulation of physical phenomena that are infinite in their contributors against presenting results on statistically limited sets (confidence interval etc.).

Radio propagation for example is expressed as a mean with RMS error, and often this error is truncated as a margin over a minimum requirement for successful communication. This enables creation of coverage maps with mean signal to noise over a prediction that already includes the uncertainty margin.

We have some level of "noise" in both cases, small set statistical, and natural infinite set, only the nature of sets is entirely different.

IMHO the worst of the worst is pushing Gauss distribution reasoning against the limited sets with binary response, say 10 pupils (being autistic or not), whereas it only works properly against the infinite sets.

Error estimation is quite important when the physical process can be modelled as normal distribution.

Otherwise, error estimation should be handled with great care because it could be actually creating false data.

Some colleagues that were working on wind farms power output prediction mentioned in a presentation that many commercial products use least squares to provide data fitting and predict the power output. However, least squares corresponds to the maximum likelihood for normal distributions. Since wind cannot be modelled with a normal distribution, these products had an inherent bias in the prediction, which was unfortunately optimistic. By the way, I think my colleagues were using self-information for predicting the power output, which is independent of the distribution.