Oh boy! A very thorny issue, especially considering all the reports of false, irreproducible results from medicine, psychology and more social sciences. I would generally say, if the statistical method is applied correctly, very reliable; but, if applied as it usually is today... very unreliable.

Oh boy! A very thorny issue, especially considering all the reports of false, irreproducible results from medicine, psychology and more social sciences. I would generally say, if the statistical method is applied correctly, very reliable; but, if applied as it usually is today... very unreliable.

I have not answered that problem: is the observed non-reliability a matter of low arithmetic accuracy or it has a deeper cause? All programs use 16 digits accuracy (the 'good' programs), so every time we have the computation of the pseudo-inverse matrix the solution cracks down...

No, I wouldn't say it's an issue of accuracy, it is more to do with application and whether or not the researcher is applying the method to data that is truly indicative of random behavior.

I'm not an expert, but I think the reliability of any method of analysis is a function of assumptions to be met. In regression we have assumptions too. Assume we measure two variables with very high precision and control any other variables which can interfere. If and if we lower the error term to zero for each paired datum, we can change the statistical model into a mathematical function with negligible uncertainties.

Zero mean of residuals means that the central gravity of residuals coincides the regression line. So a non-zero mean of residuals Says that if the regression line is a cut-off, the residuals are not equally distributed in both sides, so the estimators are not converged.

Ah, but the question is more along these lines: "provided two parameters and a regression line, does that regression line constitute cause-and-effect?" In other words, the issue is that of spurious correlation.

Good Question ! Yes, you are right that linear and non linear regressions have differences and similarities. The similarity is that mentioned by you is that the residuals are random . This assumption is needed to solve the unknown parameters in statistical methods . The supporting theory for it should come from your respective Social or Behavioral Sciences that the theory and model explain the systematic component in the movement of your dependent variable and the residual movements are random which can not be explained by theory.It of course depends on the strength of your theory/ model.

Regarding non-linear regression, I would like to refrain from making comments.

As far as linear regression is concerned, there are the four Gauss-Markov conditions to be satisfied. The first of these four conditions is that the mathematical expectation of the random errors is equal to zero. This postulate is about the population, and not about any individual sample; the word 'expectation' is associated to a population, and not to a sample. In a particular sample, this zero-mean situation need not be true.

No particular probability law is associated with the four Gauss-Markov conditions. For testing of hypotheses, we need a fifth condition, outside the Gauss-Markov conditions, that the errors follow a normal probability law with zero mean. So if we already have a situation that the mathematical expectation, or the population mean, is not equal to zero, then it is already outside the standard linear regression analysis anyway.

It is to be noted that forecasting error is an altogether different thing. Standard books on Econometrics include descriptions of forecasting errors. This must not be confused with the random errors associated with the linear regression model.

It is a common practice that the requirements of Ordinary Least Squares (OLS) regression are not even tested before doing the regression: The researcher just puts the x-y data at two columns and runs a whole set of regressions. Only after looking at the results he/she decides if he/she found something interesting and not a spurious relation.

Combine this lack of a priori testing with the limited accuracy of all those 'regression machines' and you end up with a sui generis situation.

Dear Demetris,

First of all, I am not an expert. Sorry.

Secondary, I don’t trust any applied statistics in Social Sciences. I believe that any result may be booked a priori.

At last, regression technique is only a tool, which should be true to use.

Unfortunately, I have no answer on your specific question, which, as I understand, is only interested for you.

Regarding behavorial and medicinal sciences and their woeful use of statistics, a few remarkable articles:

1] Begley C. Glenn, "Reproducibility: Six red flags for suspect work", Nature, 497, 433–434 (23 May 2013)

2] "Drug development: Raise standards for preclinical cancer researc", Nature, 483, 531–533 (29 March 2012)

3] Ioannidis JPA (2005) Why Most Published Research Findings Are False. PLoS Med 2(8): e124. doi:10.1371/journal.pmed.0020124

As Luisiana already mentioned the 'regression party' is holding in medical sciences: There, if at least 30 patients can be treated, a regression is done with many astonishing results (do you remember?: 'recently research show that if you eat ... then you...' and other 'great announcements') Those results affect directly our daily habits, since we are most likely to adopt something 'scientific', but we have to always remeber that usually it holds: 'scientific'='regression result'. I think Medicine is not a pure social science, since Biology, Chemistry, Physics and other natural sciences are involved. The results of Medicine are that affecting other social sciences.

Merry Christmas everybody: for Christians because it is a Holy Day, for others because it is holiday!

Dear Demetris Christopoulos,

Regression formula:

Prediction for y = mean(y)

+ [x – mean(x)]*r*SD(y)/SD(x)

● If the plot (x vs y) appears random, regression is acceptable

Merry Christmas to everyone!

Social sciences are not exacts ones, so many heuristics must be taken into account.

Merry Christmas and good holidays

Dear Afaq, I think you probably meant: "If the plot (x vs y) appears not to be random...", since then a kind of relation in principle occurs.

I see regression as a tool to learn something about the relationships between a number of variables. I rather focus on the learning aspects and any tool that is effective (be it linear or non-linear regression) is what counts. I have used artificial neural networks (ANN) and an array of tools available via the various toolboxes in MATLAB. Need to be careful with understanding the envelope of validity of the developed models; not to "over-learn" with too much data, when possible translate the mappings to heuristic rules which govern the relationship (leads to deep learning as opposed to the shallow learning usually associated with regression techniques). Use a systems based approach to verify the predictive capability of the models. I have learned to rely on ANN for certain applications and on other cases have found simpler models as the situation demands.

Dear Demetris Christopoulos,

Refer to my previous posting and your query:

The plot which are called residuals. If those residuals appear to be essentially random they resemble a normal ( Gaussian) distribution), then that offers support to that our linear model.

Thanks!

Dear Afaq, I think you wanted to write that the residual plot has to be random noise and not the original y~x plot, it's ok I understood your point.

Dear Kelly, I have also tried a set of heuristic methods in simple problems: they tend to present solutions based on the minimum sum of squares and not based to an overall accepted method for recovering the data. They fail even in the simplest polynomial regression data. Thus the problem of regression still remains open.

*** Regression could be a powerful tool only if it could recover our hidden functional relation with accuracy for the most of the cases, at least.

@Demetris, I had a chance to use Regression as a process of estimation in Medicine. Actually, "The analysis of infant mortality in the period up to 6 months in one district in Serbia" was the subject of specialist postgraduate thesis of my friend. Since he was a doctor, and did not much know statistical tools, I helped him in the study.

The results thus obtained and the conclusions drawn from these regressions were very valid, time has shown. Specific predictions were to be accurate!

Demetris, fellows, two brief comments:

- in the social sciences, as well as in environmental and health studies, the frequently spatial character of the data can lead to falacious conclusions when using traditional regression techniques due to spatial autocorrelation, and the multiple area unit problem (MAUP), as the conditions to valid application of the regression model may not be warranted

- as in all of the scientific field, associations and correlations found in data can be used to advance and to falsify theories, but in most complex analyses (in particular, in those involving humans) drawing conclusions from a 'crude' interpretation of the regression results might be falacious not just because of the spatial character of the data, but starting from the initial selection of the variables etc

Well, maybe these are not exactly comments, they are just reminders...

Another favorite method of regression community is the effort to remedy multicollinearity problem by using Instrumental Variable technique:

http://en.wikipedia.org/wiki/Instrumental_variable

see the section 'Potential problems'.

It is a common practise a Nobel Prize to be given just for a suitable instrumental variable proposal. But if you see the R^2 value of the proposed relation then most of the times is disappointing low.

So, the effort is to do OLS regression by a way that sometimes "hides the problems under the carpet".