When searching a solution by molecular replacement or experimental phasing the relative value of Rfree is very usefull because it shows you if you are in the "good direction" when you construct your model molecule into the electronic density. If after each refinement Rfree decreases then this is a good indication to continue. However the absolute value of Rfree is not the only statistic which validate your model. Protein geometry, rotamer analysis, Ramachandran plot are also important. Better have a higher Rfree and a good geometry than the oposite. Theoretically Rfree approximately will have a value depending to your resolution. For exemple if you have 2A resolution then a 20% - 25% would be nice. If you have more than that then Rfree may indicate you that there are problems like twin, pseudosymmetries, or extra contributions from molecules you didn't took into account into the density. Have in mind that there are no rules in this game, each protein cristal can be a new adventure, and the cristallographer has to take into account all of possible statistics from the integration of the data until the final model ready to publish.

When searching a solution by molecular replacement or experimental phasing the relative value of Rfree is very usefull because it shows you if you are in the "good direction" when you construct your model molecule into the electronic density. If after each refinement Rfree decreases then this is a good indication to continue. However the absolute value of Rfree is not the only statistic which validate your model. Protein geometry, rotamer analysis, Ramachandran plot are also important. Better have a higher Rfree and a good geometry than the oposite. Theoretically Rfree approximately will have a value depending to your resolution. For exemple if you have 2A resolution then a 20% - 25% would be nice. If you have more than that then Rfree may indicate you that there are problems like twin, pseudosymmetries, or extra contributions from molecules you didn't took into account into the density. Have in mind that there are no rules in this game, each protein cristal can be a new adventure, and the cristallographer has to take into account all of possible statistics from the integration of the data until the final model ready to publish.

The concept of using an Rfree value is not about "believe" for crystallographers. It is actually a very useful indicator for proceeding in the right direction as explained by Nicolas. On the oher hand, there is a limit to its use. The completeness of the data is reduced in the process and this has an impact on the quality of the electron density maps calculated: 5% of the data or 1000 reflections are set aside for Rfree calculating and thus do not contribute to the quallity of the electron density maps calculated for building the model. This means some details are lost from a certain point of view whilst others will reason that the model is at least not over-interpreted. Especially for a low resolution structure, the "loss" of 5% of the data has quite a strong influence on the maps but these are the structures that benefit most from Rfree calculation.

Dear Jeroen and Nicolas,

Do you have specific examples in which you were able to tell that something was wrong or right based on Rfree behavior? How big was the fragment and what were the specifics of the lattice symmetry; low - high? The generalities and beliefs you cite are well known and accepted. I am rather interested in concrete examples.

What do you mean with "fragment"? I talk about refinement of protein and complexes (macromolecules). If you have a wrong solution in molecular replacement or use the wrong spacegroup, the gap between R and Rfree can be quite large.

Thank you for the clarification. This is one useful direction but the original method was designed to weave out the errors in folding and tracing. I see as many errors in deposited in the PDB structures now as before, so I was wondering about specific people's experiences. My experience (of more than 100 papers) is that it is a spurious source of info that can by supplanted by many other sources. It does not help in weaving either errors nor obvious forgeries and serves as a harassment point from mindless worshipers of lack of common sense. As you noted above if we take 5 or 10% of atoms away (instead of reflections) we would not call a structure finished (even though, many deposited structures miss more than 10% of atoms). Since I have seen the papers with 50% reflections published in prominent journals then what would be the Rfree in reference to such a structure.

You say "...I see as many errors in deposited in the PDB structures now as before," . What is an "error" for you concerning a pdb structure ? I think that there is missunderstanding in what you call "error" and also "missing atoms". You also say "...My experience (of more than 100 papers) is that it is a spurious source of info that can by supplanted by many other sources". Can you make it more clear? Also in my opinio the experience in that case is not how many papers you have but how many structures you resolved by your own ?

Dear Nicolas,

You are indeed opinionated. Almost as myself. However, everything what you say sounds a bit arrogant. You avoid answering my simple question opting instead to erode my credibility as a scientist. Look at my newest paper in Protein Science and you understand what errors mean. I might be a bit critical and demanding about the crystal standards but I have the experience and the record to prove that what I say is what I say. So focus on answering the question.

My statements were not to antagonize but only to turn attention that advertising is not a replacement for value. Before Rfree people solved many structures and vast majority of them were correct. After Rfree (which can be dated to around 1993) we had several cases of major fraud (e.g. Krishna Murthy from Birmingham, AL) and several cases of major blunders (e.g. Chang from Scripps, CA). All of these studies used Rfree. I personally came across many correctable errors in structures published in the last two years. Some of the findings I shared with the IUCR committee on protein validation.

I hope these addresses your concerns. Besides the number of responses to my question is the best testimony to the apprehension of the crystallographic community about this issue. It is best not to touch something sacred. I fully appreciate the role of it but disagree with a sanctity of it and would like to know the practical limitations. Its practical applicability depends on so many elements, like space group, completeness, Rmerge etc. It can be very easily manipulated, as in my hands, in a relatively small data set the selection of ten independent test-sets resulted in Rfree variation around 10%. Its abuse leads to routine under-refinement of water structure and mis-identification of almost 50% small molecule components of the solvent (read the papers by Rupp). I am not mentioning about opportunistic omission of identifiable fragments that may lead to clear biological interpretation with weaker densities that inclusion might lead authors into trouble. I think that we need to stop now and wait for feedback. I wanted the responders to bring all these problems without suggesting them, but I guess ...

I apologize for beeing arrogant. Let's go back to the point. What you have realized about waters small molecules in the density or about the 10% fluctuation of Rfree depending on the Rfree data set, are totally right. We all know that and there are very good articles talking about that ....This a nice one. There are many answers in your questions incide.

http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2631636/

Dear Nicolas,

Thank you for pointing this article which I missed. I know pretty much most of the literature on the subject. I asked the question because I have noticed in my recent investigations that the effort of PDB-Redo is not effective in spotting and removing errors. One such case is described in Prot Sci paper. But I published at least three other papers showing different difficulties with existing structures. I am preparing a mechanistic paper in which almost all the structures concerning a certain enzyme have major defects that preclude the correct conclusions concerning the mechanism.

I proposed, many times in the past, changing the requirements for the refinement standards by just simply collecting and refining all the crystal structures just in P1. Only then the real approximation to the crystallographic symmetry we observe on the stage of the data collection can be revealed. Every crystallographic symmetry is only approximate, as solvent and many elements of the structure that contribute to the Rmerge (usually in the range of 5-10%) must violate the symmetry. Therefore, only refinement in P1 would unequivocally establish the violations. However, in many years talking about this idea I got literally nowhere. The irony is that it is completely doable with modern synchrotron data collection and would serve as an excellent validation tool by removing systematic biases introduced by averaging the data for a particular space group, which is evident in the article by Read and Kleywegt.

I am still interested in somebody answering my question: how a big fragment can be unequivocally detected as incorrect using Rfree (with usual caveats of a particular space group, or a particular level of Rmerge).

Probably the best example of the use of Rfree is Gerard Kleywegt and Alwyn Jones' deliberately mis-traced structure of cellular retinoic-acid-binding protein (see http://xray.bmc.uu.se/gerard/gmrp/gmrp.html) in which the R value could be made as low as 0.21 but the free R-value was 0.61.

In practice I think that using Rfree is a good way to make sure that we do not fool ourselves and others. It requires rather deliberate effort to manipulate Rfree. while it is easy to manipulate R by changing parameter values and parameterization. Consequently the focus on Rfree helps prevent inadvertent or deliberate non-productive adjustment of the model and over-parameterization.

Additionally, the difference between R and Rfree is partly an indication of whether there are too many parameters in the model. There is no need to make sure that these are very close to each other, but a huge gap does suggest that there is something that could be improved in refinement.

Dear Tom,

I followed your entire career and really admired your contributions. I am not sure you share the same experiences with me.

You cite a very old piece of science that is mostly misleading.

I followed most of the developments published in the literature and found most of the claims exaggerated. My experiences are much more modest. Your answer is at best on the elementary level why my question is on a bit more advanced stage. I worked remotely with Alex (Brunger) in the early 90s on the precise definitions of applicability of Rfree. The results of this small project were that Rfree is mostly a measure of the strength of the constraints not the accuracy or correctness of the model. The Rfree difference depending on the selections and the circumstances can vary from around 1 to 12%. As such it becomes almost unusable and not a real science (means predictive, numerical) indicator. It comes from a jackknife or bootstrapping methods used in statistical analysis. However the much softer meaning of crystallographic data makes it much less usable than popularly believed. I wanted to know the examples of clear cut help that has been gotten from the use of it. Everybody I know is just mouthing about it but I do not see much of concrete examples that would justify the trust (especially the referees) put in this numerical parameter. Especially pointless appear the remarks that the difference between R and Rfree is too big. In my opinion most of the structures have under-refined disorder and the solvent structure. The correct refinement of these elements lead to higher (not lower difference in R and Rfree. It is good though that one of the main contributors to the effort of improving the reliability of the protein structures responded to this simple question. A low response rate suggested that this is rather an embarrassing subject that nobody wanted to touch. In reality most of the structures in the PDB have larger or smaller defects that PDB-Redo is unable to deal with. The question is how to move forward. I will gladly share my thoughts on the subject.

By the way you also sidelined my main question: how a big fragment can be reliably detected as erroneous by the Rfree depending on the space group, resolution, Rmerge and a lot of other parameters..

Thanks for the quick reply to my late entry in the discussion! I apologize for an answer that is not at the level you were looking for; I was trying to answer to the community and tried to have a level appropriate for that (I am guessing that this is why some of the other respondents also missed the level that you were looking for).

Yes, I didn't answer your main question directly. This is because I don't think that Rfree is a very good way to detect fragment errors, or even to detect errors at all (from your comments I gather that we would agree on that).

I think that the R value is a perfectly fine way to detect overall errors, while the free R is a reasonable way to make sure that we are not fooling ourselves with the R value. Local measures (agreement with least biased map available) are at present a much better method for detecting fragment errors.

I agree with you that there is too much emphasis on the Rfree values and gap between R and Rfree. This emphasis distracts from the main goal of having the best possible model.

I wonder if you can expand upon your comment that, "...the correct refinement of these elements lead to higher (not lower difference in R and Rfree"?

I like Rfree. I use it every week for structure refinement. I see it as a useful tool to follow refinement progress and as a relatively unbiased fitness function. As you know, a fitness function is an objective function that can summarize, as a single figure of merit, how close a given calculated solution is to a set of data. For me as a working crystallographer, it is extremely helpful to have one number that tells me about an overall match of the data to a model that is less biased that the working R-value. I agree with the points that Tom Terwilliger made from my experience with the structures I have solved. You correctly point out, as an overall quantitative result, Rfree is of course not especially useful or accurate to check any given fragment. Here detailed unbiased electron density maps speak more accurately as they define local detail. However Rfree is telling us (as reviewers, readers or curious researchers looking at PDB structures) something useful about the overall model to data agreement. As you also note, Rfree provides a useful basis to appropriately weight stereochemistry and structure factors during a refinement process. I think manipulating the Rfree without deliberate fraud is more difficult than correctly solving structures, so I see it as a positive and useful criteria to examine model-data agreement. No metric will withstand deliberate cheating or intense manipulation efforts. Yes I solved structures before Rfree that were correct, such as the first Cu,Zn superoxide dismutase structure (Tainer JA, Getzoff ED, Beem KM, Richardson JS, Richardson DC. (1982) Determination and analysis of the 2 A-structure of copper, zinc superoxide dismutase J Mol Biol. 160: 181-217), but I am sure glad to use Rfree now that we have it. I think you might contribute more to the field by providing alternative useful metrics that do better for various issues than spending more time showing that as a single function Rfree is not equally useful for evaluating all detailed aspects of structures. More parameters are better and multiple factors are certainly relevant to judging structures. I look forward to seeing what you might provide in this regard.

Dear John,

Thank you for your answer. We had a very interesting exchange with Tom off-line after his remarks on RG. You slightly misread my remarks. Nobody, including myself, fights against Rfree. It is useful and needed parameter. What is my concern that proponents had for a very long time overstated its usefulness. For the last two years I came across a number of structures that had more than 10% structure defective and conventional structure validation was not as successful as I would expect. I communicated my findings to the IUCR committee on structure validation but have not seen much of a response. It is important that we work on new tools but also better define the validity of the old tools. By the way we are not only in need of new tools but also in need of changing some of our customary ways we treat the structures. For reasons look at my paper in PNAS 2009.

For an example of such a structure that passed validation read my newest paper in Protein Sci. I can gladly provide examples that in my opinion are flawed even at ~1Å resolution.

Another measure that could be helpful in this context in the future are "theoretical R-values", that give the approximate numerical value of an attainable R-value given the experimental data (observed intensities and their standard uncertainties), together with the number of parameters that are going to be amployed in the structure model. See doi:10.1107/S0108767313022514

Our suggestion is to not use TEST and Rfree.

See explanation on the CCP4 forum and in our paper (attachment).

http://proteincrystallography.org/ccp4bb/message37460.html

Dear Jure,

You are missing the point as Dusan is missing too.

The problem is how to avoid gross errors, like misfolding misplacement lack of appropriate solvent molecues and include multiple conformers where needed. Or alternatively how to fix them.

The kick maps method can never deliver this. In opposition to Dusan view, omitting reflections is not such a bad idea as statistical methods teach us, but I agree with him that excluding reflections from the refinement is an absurd idea.

I had private exchanges with several participants and I manged to convince them that it is difficult to find in the PDB structures that someone would not call defective.