The problem you have described is by no means restricted to social sciences. Basically it is a question of the balance between over-fitting and under-fitting. If your model contains too few variables it gives bad predictions because it does not take important variables into account. If the model takes too many variables into account it is easy to fit to existing records to an extend that it tells noting about the future.

Finding the right balance between under-fitting and over-fitting is by no means easy, but in some cases there are methods like cross validation that can be used even in situations where it is difficult to make a precise statistical model. Unfortunately many scientists have an inadequate understanding of the problem and tend to either always advocate for using more complicated models or always advocate for using simpler models.

I fully agree with Peter Harremoës. Often, people would like to know the exact number of model parameters optimally describing the system - some easy rule like: 'it should be between two and four, one is always not enough and more than five is always overfitting'. Unfortunately, this is the worst way, because then people might have some a priori expectations about the optimal number of model parameters. But naturally, each system is very different, requiring different models, even in the same research fields. 

Dear Peter, dear Jan,

I entirely agree with you that the problem of the statistical individual may be the same for other sciences. But when considering over-fitted or under-fitted models, you are supposing that a scientific theory needs to generalize an empirical statistical model. However this one of the two main approaches for human knowledge you are following, while you let the other under silence. Let me explain what I mean about these two approaches.

The first approach, which I think you are following, is the “covering law” approach, introduced by Hume in 1748. For it, scientific explanation consists of deducing facts from law-like generalizations. So that it is a deductive concept of explanation. It tries to verify if “hypotheses” are valid or not. But as any new test leads to change some hypotheses it is an endless task of verification. It had dominated twentieth century social science speaking of testing the validity of a model and rejecting those that fail to agree with the data.

The second approach, which I try to follow here, is the “inductive” approach introduced by Bacon in 1620. For it, scientific explanation consists in discovering principles from a study of its properties, by way of experience and experimentation. So that it is an inductive concept of explanation. It is defined this time as a discovery of principles from the study of their consequences. It was followed by Galileo, Graunt, Newton, and more recently by Einstein. It no more tries to verify “hypotheses”: as Newton said “hypotheses non fingo”. Contrary, it tries to find “axioms” which are the principles governing the phenomena: in order to do that it tries to discover the combination of concepts without which the observed properties would be inconceivable or impossible. I recommend you to read the book edited by Franck (2002), which discusses all these concepts.

While I did not say it enough clearly, I follow here the “inductive” approach. So that I am no more considering over-fitted or under fitted models, but trying to find if the statistical individual may or may not be considered as one of the “axioms” for social science. But as you said, this may also be true for other sciences. For example Newton and Einstein never considered “probability” in the axioms of their space-time. As Einstein said “God does not play dices”. However Born in quantum mechanics introduced probability at the basis of its predictions. See for more details the Chapter of Suppes (2002) on Representations in mechanics. So that the realities they describe seem utterly incompatible. However recently Cowen (2015) a theory of quantum gravity may be able in the near future to join these to approaches.

Please don’t hesitate to ask me if you want more precisions on the “inductive” approach.

References

Bacon, F. (1620). Novum Organon. London: J. Bill.

Cowen, R. (2015). Space, time, entanglement. Nature, 527, pp. 290-293.

Franck, R. (2002). The explanatory power of models. Dordrecht: Kluwer.

Hume, D. (1748). Philosophical essays concerning human understanding. London: Millar.

Suppes, P. (2002). Representations and invariance of scientific structures. Stanford: CSLI publications.

Presumably "Probability/" which I have begun to loath despite being my area of research),  is allegedly the 'logic of science'. Despite the fact that currently, we do not even know what it means.

This it taken to include any concept, collectivist, single case, frequentist or subjectivist, numerical or qualitative that attempts to grade degrees of possibility./plausibility/graded modality/ uncertainty/necessity that comes in degrees, whether epistemic or ontic" Including descriptive mental states where one expresses statements or partial inclinations 'ie thats plausible'

 I calll probability, Graded modal logic I call it. Any halfway house between ontic/epistemic determinism/predictability and in-determinism/unpredictability likewise

Unfortunately without this circular (completely non-understood- as Russell would say) concept we presumably could not do anything (but how we would we qualify that assertion, 'in all probability'.

Terrence Fine make a remark about this and mentions Suppes in a relatively recent power-point presentation). But with it , we are likewise doomed. I see it as the' mother of science' and the 'mother of all our problems in science' as well (where its social science or quantum mechanics).

Incidentally the book by Suppes is quite good 2002. I am not sure if I agree with his bemusement  when P. Suppes makes the statement in" foundations of measurement vol 1'  to the effect that:

' its unclear why unlike mass, for example, that the meaning and axiom-atization of probability has attracted so much controversy".

Perhaps its probability analysed/axiomatized, commonly, as if just is a mass, or a measure of a volume; as even these satisfy the probability calculus but are clearly different.

I guess that the notion/analysis in terms  of 'ratios and ratios of differences', or proportions, are more difficult to make sense, when considering a single case analyses of probabilities alone (using the single case alone), which have become more common in the last 50 years or so. ie that an event A  twice as probable B, occurs twice over on the single case/singular trial (is dutch-book, dominated, twice over?)