Well, seeing that there are no answers yet, consider Gravity Probe B as an example, or the Hubble space telescope. Long in the making, expensive, subjective confidence driving science targets. Results were (at least initially) not as expected, requiring more time, more funding, driven once more by overconfidence and optimism. The outcomes were fairly favourable in the end....with a slight dubious taste in the mouth at the one end and an empty pocket at the other. Without the over optimism, none of these very commendable projects would have been supported in terms of funding. Does the collective memory of these projects influence the outcome of future projects? Probably...

Because we typically fail to consider important ways in the complicated future events, we become unnecessarily confident about future occurrences based on the fewer ways we actually do experience. On a paper, everything appears to be highly supervised and guaranteed, but too far to be optimistic about the real unpredictable world.

I think, hubble mission was one of the early big space borne telescope; we had less experienced people. But now for James Webb space telescope, will we make the same mistake as we did for Hubble (not adequate testing)?(more ways we have already experienced)

The another point to consider is human's self-defense mechanism. We naturally tend to seek support for our viewpoint rather than to search for our deficiencies. The solution is hiring more and more reviewers, competitors and even jealous people! But I can found few people who have a deep capacity to tolerate opposite opinions.

Amir, very true about the self-defence mechanism and intolerance of other opinions. To some extent these can often lead to conflict and an the early demise of many proposed experiments. On the other hand, some experiments only come to fruition through the sheer obstinacy, dedication and continuous efforts of those who are convinced it is a good idea. This oftentimes makes experiments dragged out, not fully supported by all, and leaves an unsatisfactory taste in the mouth. Some of the drivers of these experiments, remain optimistic...and that is the only way in which many experiments survive...obstinacy and optimism. Confidence probably suffers a bit during the process.

Great discussion, Dr. Combrinck,

We cannot also neglect the importance of money and budget. When you are a modest and pragmatic person (ideal researchers), you are supposed to encounter with numerous people evaluating and analyzing your work in order improve it (ideal referees). Therefore you should spend a lot of time which leads to a lot of money. It is not an easy task to endure both budget pressure and countless critics. In this situation, even ideal researchers will eventually prioritize optimism over pragmatism.

Yes Amir, I agree. Funding, spending within a budget (which sometimes is not practical or realistic within the optimistic original point of departure) as well as political 'buy-in' shape and massage the balance between pragmatism and optimism. Sometimes a lot of time is spent on writing proposals, gathering foreign participants and collaborators, developing student projects etc. and still no buy-in occurs, no funding etc. This environment of perceived negativity eventually erodes the optimism, the personal drive to get a particular project of the ground and leads to zero output after a lot of wasted input. How does this affect future over-optimistic and enthusiastic approaches to new science experiments that requires support and funding? Is science not often held back by systemic failures?

as a transverse contribution, I refer you to the article in Optics and Photonics NEws Dec 2013 by Marc Kuchner on Science Maturity Level

It can also be found here:

http://blogs.nature.com/soapboxscience/2013/04/03/whats-your-science-maturity-level

Interesting article Samuel, thanks.

I think there is a potential problem in that if we have two parallel research teams who are both trying to be the first to produce a result, and one team is over-optimistic, and the other team is more careful (and want to spend an extra three months redoing the experiment and checking their results before they publish), then Darwinian evolution will favour the "optimistic" group who perhaps cut a few corners in order to publish first.

This "race to be first" can sometimes lead to some bad science, and when a community desperately wants a result to be verified, that enthusiasm can lead to some work being received very uncritically, when perhaps a little bit of scientific scepticism would be in order.

I was going to mention Adams' work on the existence of gravitational redshifts in starlight, but in fact, for decades some other very well-respected researchers were claiming to have found the first definite verification (or refutation) of a gravitational redshift, and with hindsight, all of those results before about ~1959 seem to have been junk science. The way that the reward structure was set up meant that here seems to have been a tendency to over-claim the significance of results (or for the researchers to genuinely believe their own "optimistic" assessments.

Hi Eric, good answer, yes I am sure this is a case in point where the systemic demands on outputs in the form of publications and return value on funding actually damages science. There have been many of these '"races to be first", or even just races to end up with the target number of completed students or published papers. Today they are a permanent scar on the scientific landscape. I think we all suffer from it, and science as a way of life, an avenue of exploration and creativity, is all too often cluttered and poisoned with system demands.