Dear R. Carrasco Hernandez ,

Yes, I think you can reformulate an ANOVA as a multiple linear regression, but not in the way you have suggested. I suspect your x1 to x3 variables should be binary with values set according to the origin of the data. You probably also need one less than the total number of options for each group as the value of the last variable can be inferred from the others. You also need to consider whether or not to include interaction terms.

Maybe I got the whole question wrong, but, how many data points do you have for y and all combination of groups?

Hello R. Carrasco Hernandez ,

Yes, ANOVA is a just a special case of OLS regression (which is the poster child for the general linear model).

Yes, your problem could be recast as a regression model.

Yes, Peter Samuels 's suggestion is correct; if what you are really trying to do is add in country as a nominal IV, ethnicity as a nominal IV, and age as a nominal IV, you would have to convert each nominal/categorical IV to a set of k - 1 dummy variates (where k is the number of levels for that IV). As well, you would have to consider whether interaction terms made sense to evaluate and, if so, create them as additional variates in your model.

Yes, Rainer Duesing 's question is pertinent, as you'll need to consider whether you have ample replications of all possible combinations of IV terms. That issue is independent of your choice of anova or ols regression software, however.

As an observation, wouldn't age be better treated as a single, continuous IV than as a categorical IV? As well, one's sex would likely have some predictive value as regards body weight.

Good luck with your work.

David Morse , Peter Samuels . Thank you so much, I had read that answer before... Now my question is why are dummy variables the "right" solution? and why is my proposal not the "right" solution?. Let's step down a bit and go to only one categorical factor (e.g. people's weight from three different countries) Picture a scatterplot of 'weight' values on the Y-axis and the 3 country weight averages on the X-axis (e.g. 70kg, 75kg and 80kg). So the x-axis is now a continous axis. Our model here is stating that: the weight of a person can be predicted from the mean weight of people from the same country... So it is an "interpolatable" and "extrapolateble" model, to whatever other countries; of which we could get to know their mean weights in the future. The scatterplot, and a least-squares regression of it, are also useful for testing significance; because, if there are no differences in mean weights between the countries, or if the within-country standard deviations are large, then, the slope of the regression should become not significant.

Now, I am thinking overfitting could be a problem if you include each "yi" vaue in the caluclation of the group average, specially if you have few oservations. Here's where @Rainer Duesing's question comes into play: If there are a lot of observations per group, then overfitting might not be a problem; but, with few observations, maybe recalculating the group average for each "yi", by removing "yi" from the average caluclation could solve the overfitting problem?.

Hello again R. Carrasco,

I guess I'm still a bit unclear on what your research aim might be. If you're using country averages as data points, then the effective sample N becomes very small, and the precision with which coefficient estimates are computed becomes poor. As well, the observed range of "values" for the IV will be artificially restricted (versus that obtained by using individual case values).

If your interest is that of identifying the specific differences from the grand mean that a specific value of a factor implies (e.g., if case comes from country "X", then we predict that their weight will be 1.3 Kg less than the grand mean), these parameter estimates are frequently available from anova software packages. And, yes, they come from recasting categorical IVs into dummy variates.

If you're suggesting that a multilevel model needs to be applied, then of course, your approach will necessarily be different from that of ordinary anova or OLS regression. Multilevel models are an extended version of regression, wherein coefficients obtained from one level become part of the parameterization for coefficients for subsequent levels. However, three countries won't permit much useful inference about country variation (too small a sampling)..

Good luck with your work.

Linear regression requires a continuous outcome variable and either categorical or continuous predictors. Two of your predictors are nominal variables (country, ethnicity), which means the levels have no intrinsic order (e.g., Spain, Greece, China...). The problem is you are using averages as the levels (which assumes intrinsic order: an average of 80 kg is more than an average of 70 kg), rather than using categories (like Spain) as levels - which is required of a nominal variable.

The model will think you have a predictor (country) with four quantitative levels (65 kg, 70 kg, 75 kg, 80 kg). What you need is a predictor (country) with N (dummy-coded) categorical levels (Spain, Greece, China...). The problem with using an average as a predictor is that it translates (model-wise) into a single quantitative data point with no variance. However, both ANOVA and linear regression are based on assessment of variance.

This makes more sense conceptually, also. Instead of "I can predict your weight given the average weight of people in your country, the average weight of people of your ethnicity and the average weight of people your age" it becomes, "I can predict your weight given your country, your ethnicity, and your age".

David Morse and Blaine Tomkins , thanks for your answers.

The "data" I wrote about is just an hypothetical example, I don't really want to know anything about the "data"... My question is more theoretical, I want to know why is "ANOVA-as-linear-regression" always solved as a multivariate regression of k-1 dummy variables and never as a univariate regression to group averages. Or if there's any literature doing so.

From the answers I have received here, the problem seems related to a small variance on the X-axis?... But let's put another hypothetical example: people's income in 3 countries.

Country 1 Average income: 10 k; Stddev.: 10k, No.of Obs.:100,000

Country 2 Average income: 20 k; Stddev.: 10k, No.of Obs.:100,000

Country 3 Average income: 70 k; Stddev.: 10k, No.of Obs.:100,000

The scatterplot of person's income versus country average may look something like picture 1 attached

In Picture 1, it doesn't look like there is small variation in any of the two axes. It looks like a significant corelation. Plus, it allows me to talk about a more general model, larger than only three countries.

Moreover, if I calculate each "xi" corresponding to each "yi" value as the group average WITHOUT "yi", then I would have internal variation of x values at each group.

R. Carrasco Hernandez Please see the link: https://www.google.com/search?client=firefox-b-1-d&q=Show+me+why+ANOVA+and+Regression+the+same+analysis

Then with a small set of data from a text book run it both ways and get the same answer. That is what Fisher did to check his development of the classical ANOVA based ANCOVA formulae here:https://core.ac.uk/download/pdf/162101428.pdf

See also:https://www.google.com/search?client=firefox-b-1-d&q=Show+me+why+ANOVA+and+Regression+the+same+analysis

and https://pdfs.semanticscholar.org/f195/3678333b7545f5ecaf5f1743fdaa264c56d1.pdf

Bust MOST IMPORTANTLY WORK A SMALL ONE BOTH WAYS IN YOUR FAV. SOFTWARE AND SEE IT FOR YOURSELF. Best wishes, D. Booth

PS then look at this:https://www.researchgate.net/post/What-is-the-differrence-between-Regression-Analysis-and-Path-Analysis and find that the answer is the sames as for ANOVA. Can you see why?

David Eugene Booth thanks!, it's a good idea to run them both; however, the results of my approach and the "dummy-variables" approach are not even comparable in some ways. My approach would yield only one slope per factor, while the "dummy-vars" approach yields k-1 slopes per factor.

I could try to compare residuals, and see what happens... I will do that.

Still, the theoretical interpretations of the two approaches are quite different, yielding models predicting the same response variable but with factors that "mean" different things; and that can be used for different purposes. For example, if you know a formula that correlates individual income to country average income, then you can create a model for future stages introducing changing country averages through time, on the basis of the same formula (of course being cautions about assumptions). Yet, this would be impossible with the dummy variables approach.

Then what you are doing is incorrect. Learn the correct way as Fisher himself did 100 years ago to make sure things were correct. Residuals will show that you are modeling things in correctly still. Perhaps this will help: You can't understand research data until you understand the methodology. Best, David Booth

I do somehow not see the value of this form of analysis, where you use the average value (in this example of income) as a predictor.

1) If you know the (exact) average income of a country, for what do you need the data points within each country? Either you estimate the average from your data points or you will only see how good your sample will represent the "true" value that has been estimated somewhere else. In your example, I would guess, by eye-balling, that in the 10k country, the sample mean value is higher than 10k. And now what? You can say that the sample you gathered deviates from the sample that has been collected to come up with the value of 10k, otherwise you wouldn't know this.

2) The whole (example) analysis does not have any value, if you predict individual income (y) from the average income (x), since you already know (without having other variables/predictors) that the best guess for the individual income of a person is the average income of all persons. So, isnt that circular reasoning? You basically show that the individual income increases, if the average income increases. Wow, would not have guessed this one (kappa ;-) ), if participants are selected randomly.

3) What David Eugene Booth already mentioned, what you typically want to know is how the income differs between countries, and therfore you estimate the average income from your sample and then compare the countries. After that, it may be helpful to find other predictors to explain it, for example political system in that country. Does this predict the differences in average income (as an example)?

4) What do you want wo predict with your regression? You already know that the best guess for the individial income is the average income. For any other country with any other average income (not depicted) your "prediction" will be true. For a country with 50k average income, the individual values will scatter around 50k. So, I do not need a regression equation to show this to me. That is the whole point of an average/mean value. Therfore your statement "if you know a formula that correlates individual income to country average income, then you can create a model for future stages introducing changing country averages through time" makes no sense. You already know that. The average is directly derived from the datapoints.

So, pleases show the value of additional information that your analysis gives and what we are missing here.

P.S.: besides that, the example with income is additionally problematic, since income is heteroscedastic between countries and within not normally distributed, so that the mean may not be a good estimator and therefore the normal OLS regression not suitable. I would expect the residuals to be not normally distributed, but just a side note, which may not be important since it is only an example.

P.P.S.: the high value for R is not impressive since you ordered your predictor a priori and since it is a mathematical transformation from the criterion you basically showed that if you rank values according to their size, the correlation will increase.

Rainer Duesing , Actually, for a country with 50k average income, you do need some statistical analysis showing that they scatter 'narrowly' around 50k, otherwise the correlation may not be significant.

If there were no differences between the group averages and/or if there are very large whithin-group variances, the same regression analysis will show a non-significant slope. Which means it serves the same purpose of an ANOVA...i.e.: Finding significant differences between groups or not!... In other words, ANOVA can also be solved as a linear regression to the group means; but with interpretations differing from the interpretations of the conventional dummy variabes approach.

As I said before, If the analysis showed a significant relation between group average value and individual value, you could then use the known relation to guess your individual value simulating changes in your group's average value. Which would be impossible if the regression is made using dummy variables. On the other hand, if there is no significant relation you can state that "the group average value is not suitable for explaining the individual value".

Now, in my original question, I proposed to further expand this analysis to a multiple linear regression in which each observation corresponds to several grous in different axes. For example: "i" corresponding to Country 1, Ethnicity 2, and Scholar level 4. If you know "if and how" the group average value in each axis correlates to the individual value, you can then create a more "fine-tuned" model for each individual with the variables that resulted significant ... and you could also create more complex simulations with changing groups' averages. Now, this analysis requires a lot less dimensions than the conventional 'dummy-vars' approach and may be more easy to interpret.

Dear R. Carrasco Hernandez please answer the following question:

From where or which analysis did you get the group average that you used as the predictor variable and which you showed on the x-axis?? For example that a country has an average income of 50k?

Rainer Duesing , thanks a lot for your interest!,

The values of the x-axis are the observed group averages of the individual observations on the y-axis. For example (a different example from the one above):

-Country 1-

y1,1 : 10k

y1,2 : 20k

y1,3 : 30k

Mean: 20k

-Country 2-

y2,1 : 1k

y2,2 : 21k

y2,3 : 41k

Mean: 21k

So, I propose to conduct a linear regression on the following x and y values:

y1,1 : 10k ; x1,1 : 20k

y1,2 : 20k ; x1,2 : 20k

y1,3 : 30k ; x1,3 : 20k

y2,1 : 1k ; x2,1 : 21k

y2,2 : 21k; x2,2 : 21k

y2,3 : 41k; x2,3 : 21k

Notice that, in this specific case, unlike my previous example, the relation will not be significant and the linear regression will show a slope non-significantly different from 0, thus serving the same purpose of an ANOVA. In this new example, the analysis shows that the obseeved group average is not better than the observed overall average to predict the individual "y's". While, in my previous example, the observed group average was better than the overall average to predict the individual "y's" (and the analysis showed this).

My proposed approach is different from the conventional dummy-vars approach which would be:

y1,1 : 10k ; x1,1 : 1

y1,2 : 20k ; x1,2 : 1

y1,3 : 30k ; x1,3 : 1

y2,1 : 1k ; x2,1 : 0

y2,2 : 21k; x2,2 : 0

y2,3 : 41k; x2,3 : 0

a linear regression on this model will tell me that belonging to one group or the other affects your individual value. However, as you increase the number of values in each factor and the number of factors, the interpretation becomes more and more difficult; because it implies increasing dimensionality of a multiple regression. It might also become more computationally intensive for large datasets.

For the significance level of all analysis, you may want to choose the p-value of 0.05. This indicates that if the p-value is less than or equal to 0.05, the means of two or more independent groups are significantly different from each other for multi-way analysis, resulting in rejection of null hypothesis that all the group population means of combination are equal. Otherwise (i.e., if the p-value is greater than 0.05), the means are not statistically significant. The p-value could be estimated by the F-ratio (i.e., the ratio of between-group variability to the within-group variability) for testing population mean.

In Minitab 17 the between-treatment mean square and the within treatment mean square are also called Adj MS for the factor and Adj MS for the residual error respectively. Adj MS represents an estimate of population variance. It is calculated by dividing the corresponding sum of squares by the degrees of freedom (DF).

It seems to me that fundamentally this is an issue of clustering of nested data. There are indeed hundreds (probably thousands) of papers on this topic.

First the hypothesis about the average slope for income within countries and the average differences between individuals on income between countries are different hypothesis. These models are necessarily different because they address different Qs. Second, the slope within countries approach isn't ideal because there is clustering within countries (the observations aren't independent).

There are various solutions but I think conceptually you might find that this is best considered as a nested multilevel model. There are many resources for this, so to pick one not quite at random:

http://www.bristol.ac.uk/cmm/software/support/workshops/materials/multilevel-m.html

Thom Baguley , thank you so so much... the answer was in the link you sent me.. My problem is indeed a multilevel problem with a cross-classified structure. I needed that name!.. Thanks again!

Lots of resources on cross classified model here:

http://www.bristol.ac.uk/cmm/software/mlwin/mlwin-resources.html#nonhier