Almost all modern refinement programs were inspired by the old ORFLS program written by Busing and Levy. At that time one refinement cycle was an overnight job and any simplification of code saving the computing time was very important. This was the reason why after a data reduction observed intensities were first transformed to F(obs) which can be directly calculated from the refinement model. Some programs divided the process into two or even three program blocks – calculation of structure factors + derivatives, building normal matrix and inversion. As mentioned by Leonardo, calculation on F made this procedure faster. On the other hand, the problem with estimating sigma(F) for weak reflection is big disadvantage. The minimized sum for the refinement on F^2 is built from terms w*(F(obs) ^2-F(calc)^2)^2=w*(|F(obs)|-|F(calc)|)^2*( |F(obs)|+|F(calc)|)^2. The refinement on |F| uses terms w’*((|F(obs)|-|F(calc)|)^2 where w’=w/4F^2(obs). So these terms are related by the factor (|F(obs)|+|F(calc)|)^2/(4F(obs)^2) which goes to 1 if |F(obs)| goes to |F(calc)| as mentioned by David Rae. For systematic errors these differences can overweight the minimized sum so strongly that refinement may explode (Michal Dusek). But similarly as Julian Henn I think that this can be taken as another advantage of the refinement on F^2.

In our system Jana2006 (http://jana.fzu.cz) we are still using as a default the refinement on F to keep some continuity with older versions. But in the new version we shall change it. This replies your question why some people still refine against F: because despite of the problems with estimation of sigma(F) for weak reflections the refinement works well in most cases, yields almost identical results, and there is no real need to change the defaults, except the methodologic reasons.

Another question is why to refine against intensities separated from frames. I think that real future is refinement of single crystal data against data from individual frames – Rietveld in 3d.

If you virtually enforce the answer into the question, why then do you ask the latter ?

Who refine against F nowadays ? At least the SHELXL97 users are encouraged to refine against F^2 : from the user manual, "the use of data corresponding to this format (HKLF 3 i.e. using F) is allowed but is NOT RECOMMENDED, since the generation of Fo and s(Fo) from Fo2 and s(Fo2) is a tricky statistical problem and could introduce bias" (which is the answer you give at your own question, thanks.

Dear Julian,

From practical point of view refinement on F is more stable in situations when some reflections in the data set are absolutely wrong. Typical case is a low angle reflection partially shaded by the beamstop which may appear in the data set when the data reduction software has improperly defined beam stop mask. In such case refinement on F**2 may "explode" why refinement of F works usually well. But I agree that defaut refinement option should be F**2.

Best wishes,

Michal

Counter question:

Why do you refine against F^2 values in a reflection file and not against the raw data on the images?

the first thing that is coming to mind -

SHELX-97, Chap. 2.4:

"Refinement against ALL F**2-values is demonstrably superior to refinement against F-values greater than some threshold [say 4sigma(F)]. More experimental information is incorporated (suitably weighted) and the chance of getting stuck in a local minimum is reduced. In pseudo-symmetry cases it is very often the weak reflections that can discriminate between alternative potential solutions. It is difficult to refine against ALL F-values because of the difficulty of estimating sigma(F) from sigma(F**2) when F**2 is zero or (as a result of experimental error) negative."

and ibid

"Refinement against F**2 also facilitates the treatment of twinned and powder data, and the determination of absolute structure."

Sincerely,

Artem

Thanks for the many quick answers. I just asked because I submitted a paper in which the disadvantages of refinement against F are emphasized after I found a majority of charge density studies performed with refinements against F. I received critics from both referees for this, which led me to withfraw this part from the manuscript as it was not the most important point. I was still puzzled however, whether I missed something important. As Michel Dusek pointed out, the refinement may be more stable when some reflections are affected by unaccounted effects, however this may count as an advantage (I get a result) or as a disadvantage (I get a result despite some measurement issues that would be better treated in some way). Are there other practical advantages?

Armel is right. To the best of my knowledge, the refinement against F is a little faster and therefore it was preferred some decades ago, when the least squares procedure required significant computational effort to be applied even on relatively simple structures. In the era of modern computers, the preferred refinement recipe definitely relies on F**2. Besides, when the error is propagated for extracting F, it turns out that s(F)= s(F**2) / 2|F|, so the new weights directly depend on the structure factor modules - this is not merely a convention, it relies on the assumption that your measured data are subject only to normal noise (questionable). In any case, the least squares procedure is very sensitive to the weighting scheme and weak reflections may introduce some biases.

Personally, in least squares I am wary of using derived quantities as observations when it is unnecessary: as a SHELX user, I always refined against F**2.

The question on F or F^2 refinement is treated in the article which can be found at

http://xray.chem.ualberta.ca/xray/shelxl/fsquared.htm

CRIME [ Crystallographic Imaging using Maximum Entropy ] requires the use of phased and unphased F's, not F**2's. One may indeed elect not to become a

CRIMINAL and enjoy the peace of mind of belonging to the Flock ( or Pack ? ).

This being said, I am also a SHELX user and as such do abide by Georges Sheldrick's rules when using his software package(s).

A recent review 'The use of partial observations, partial models and partial residuals to improve the least squares refinement of crystal structures' A.D. Rae (2013) Crystallography Reviews addresses this question amongst many others. The review includes a criticism of the paper by Hirshfeld and Rabinovich.

F and F squared refinement become the same when the non linearity of a F squared refinement is accounted for by replacing the derivative of Fcalc squared by (Fobs + Fcalc) times the derivative of Fcalc.

The least squares equations depend on an initial model that partitions an observation using an initial model that changes each refinement cycle. This model should include the background and an estimate of the error in the background and crystallographers should start insisting that instrument makers make this information available using their data processing software.

Speaking about SHELX 97, you should consider an update to SHELX 2013. It is available here http://shelx.uni-ac.gwdg.de/SHELX/index.php and free for academic users.

X-rays are diffracted by electrons correspond to electrons of the atoms which form the planes (hkl) of the crystal. The diffracted intensity is proportional to F ^ 2, but the value of the structure factor, F (hkl) is a complex number, F(hkl) = A (hkl) + iB (hkl). Structural determination involves solving this general expression, I = KF ^ 2 (Lp) (Tv) (Abs), where Lp correspond to geometric factors, (Tv) symbolizes the correction factor for thermal vibration, and K is a factor scale. The F(hkl) corresponding to a certain combination of amplitude and phase, while the radiation diffracted by the crystal is proportional to the square of the amplitude, F ^ 2. Although both F and F ^ 2 are valid data, F ^ 2 may be more useful in structural determination to be directly related to the diffracted intensity. Currently existing crystallographic programs usually resolved this dilemma.

Almost all modern refinement programs were inspired by the old ORFLS program written by Busing and Levy. At that time one refinement cycle was an overnight job and any simplification of code saving the computing time was very important. This was the reason why after a data reduction observed intensities were first transformed to F(obs) which can be directly calculated from the refinement model. Some programs divided the process into two or even three program blocks – calculation of structure factors + derivatives, building normal matrix and inversion. As mentioned by Leonardo, calculation on F made this procedure faster. On the other hand, the problem with estimating sigma(F) for weak reflection is big disadvantage. The minimized sum for the refinement on F^2 is built from terms w*(F(obs) ^2-F(calc)^2)^2=w*(|F(obs)|-|F(calc)|)^2*( |F(obs)|+|F(calc)|)^2. The refinement on |F| uses terms w’*((|F(obs)|-|F(calc)|)^2 where w’=w/4F^2(obs). So these terms are related by the factor (|F(obs)|+|F(calc)|)^2/(4F(obs)^2) which goes to 1 if |F(obs)| goes to |F(calc)| as mentioned by David Rae. For systematic errors these differences can overweight the minimized sum so strongly that refinement may explode (Michal Dusek). But similarly as Julian Henn I think that this can be taken as another advantage of the refinement on F^2.

In our system Jana2006 (http://jana.fzu.cz) we are still using as a default the refinement on F to keep some continuity with older versions. But in the new version we shall change it. This replies your question why some people still refine against F: because despite of the problems with estimation of sigma(F) for weak reflections the refinement works well in most cases, yields almost identical results, and there is no real need to change the defaults, except the methodologic reasons.

Another question is why to refine against intensities separated from frames. I think that real future is refinement of single crystal data against data from individual frames – Rietveld in 3d.