Ehsan, I am not sure if your post is an answer to Giulios question. I think is is not the central problem how to build the model. As I understood, the model is given, and the only question is how to apply it practically, i.e. how to fit it in order to get a prediction for the DV.

Giulio you said you know how the IVs are related to the DV. The solution to your problem -as I understood- is to state a statistical model that will calculate/predict the values of the DV (Y) from the given values of the IVs (X1, X2,...). Such a model can be any kind of function like

Y = f(X1,X2,...) + error

The function will contain some parameters that are searched to minimize the error (the unexplainable deviations from the observed values of the DV and the prediction from the model, also called "residuals").

Your problem will be to find a reasonable function f. The most simple setting is just a weighted sum. The weight factors (also called effects and denoted with the greek letter beta) are then the coefficients or parameters of the model:

Y = β0 + β1*X1 + β2*X2 + ... + error

Because all the coefficients (betas) are untransformed ("linear"), such models are called "linear models" (LMs). Google for that to find more. If you have reasons to think that the distribution of the errors is not symmetric (e.g. when your DV are counts or proportions or concentrations close to zero...) you may have a look at "generalized linear models" (GLMs). The DVs may be transformed to model non-linear relationships between the IV and the DV.

If the relationship can't be formulated as a weighted sum

(example: Y = β0 + β1*sin(β2*X1 + β3) + exp(β4*X2))

then you need to fit such a "non-linear model" (NLMs).

All statistical packages will fit you LMs, GLMs and NLMs, and they will also give you conficdence intervals for predictions.

The problem you have is to find a reasonable model. In the case of LMs, the analysis of the residuals helps to see if the model is seriousely misspecified. Google for "residual analysis", the keys are variance homogeneity ("homoskedasticity") and a (relatively) normal (frequency) distribution of the residuals, further a linear relation of the (possibly transformed) IVs and the DV. The residual analysis for GLMs and NLMs is complicated and I can't really help you there.

Chosing the model based on the data is an exploratory approach. The confidence intervals (for the coefficients as well as for the predictions) do NOT have the properties usually expected. This means that hypothesis tests based on such "selected" models are not valid (this point is frequently overlooked). Still you can report tham as an estimate for the precision. Just be careful not to interpret them in the frequentistic way ("95% of such 95%-confidence intervals will contain the "true" value). The intervals of "selected" models are likely too small. Only when you have one dataset to select a model and then use another, independent dataset to get the confidence intervals, the frequentistic interpretation holds.

There are many many advices and procdures for "model selection" (Ehsan reported an example). This is a big topic. In the end, to my opinion, all these strategies are unsatisfactory. There is only one good way to select a model: THINK! Use your knowledge about known, reasonable, expected, sensible, obvious,... relationships. Include the variables YOU consider relevant, independent of their performance in the model (by reducing the adj,R² or the AIC or whatever measure is taken). If you are not sure about the form of the relationship to the DV (linear, quadratic, exponential...) and none can be favoured for logical reasons, use the simplest ("Occhams razor").

Yes there is. Firstly you should assess the correlation between variable A which is your dependent variable (DV) with all of your independent variables (IVs). In the nex step take out all of the correlated IVs ( i hope there is no non-linear relation between your DV and IVs). In the next step, put all the correlated IVs in a multiple regression model, then use STEPWISE selection method to determine the best regressors.

And when you done check the criteria R-Sqr, Residual plots, VIF and Mallow's Cp respectively.

First, you should have a logic R-sqr and a random scatter plot of residuals with no trends. If they are good enough, then check if the VIF is no more than 10 for each IVs remained in the model, and at last check if Mallow's Cp is equal to the number of IVs remained in the model ±2 and no more or less. then you have a Reliable model to predict A variable using B......Z variables.

I hope it would help

Ehsan, I am not sure if your post is an answer to Giulios question. I think is is not the central problem how to build the model. As I understood, the model is given, and the only question is how to apply it practically, i.e. how to fit it in order to get a prediction for the DV.

Giulio you said you know how the IVs are related to the DV. The solution to your problem -as I understood- is to state a statistical model that will calculate/predict the values of the DV (Y) from the given values of the IVs (X1, X2,...). Such a model can be any kind of function like

Y = f(X1,X2,...) + error

The function will contain some parameters that are searched to minimize the error (the unexplainable deviations from the observed values of the DV and the prediction from the model, also called "residuals").

Your problem will be to find a reasonable function f. The most simple setting is just a weighted sum. The weight factors (also called effects and denoted with the greek letter beta) are then the coefficients or parameters of the model:

Y = β0 + β1*X1 + β2*X2 + ... + error

Because all the coefficients (betas) are untransformed ("linear"), such models are called "linear models" (LMs). Google for that to find more. If you have reasons to think that the distribution of the errors is not symmetric (e.g. when your DV are counts or proportions or concentrations close to zero...) you may have a look at "generalized linear models" (GLMs). The DVs may be transformed to model non-linear relationships between the IV and the DV.

If the relationship can't be formulated as a weighted sum

(example: Y = β0 + β1*sin(β2*X1 + β3) + exp(β4*X2))

then you need to fit such a "non-linear model" (NLMs).

All statistical packages will fit you LMs, GLMs and NLMs, and they will also give you conficdence intervals for predictions.

The problem you have is to find a reasonable model. In the case of LMs, the analysis of the residuals helps to see if the model is seriousely misspecified. Google for "residual analysis", the keys are variance homogeneity ("homoskedasticity") and a (relatively) normal (frequency) distribution of the residuals, further a linear relation of the (possibly transformed) IVs and the DV. The residual analysis for GLMs and NLMs is complicated and I can't really help you there.

Chosing the model based on the data is an exploratory approach. The confidence intervals (for the coefficients as well as for the predictions) do NOT have the properties usually expected. This means that hypothesis tests based on such "selected" models are not valid (this point is frequently overlooked). Still you can report tham as an estimate for the precision. Just be careful not to interpret them in the frequentistic way ("95% of such 95%-confidence intervals will contain the "true" value). The intervals of "selected" models are likely too small. Only when you have one dataset to select a model and then use another, independent dataset to get the confidence intervals, the frequentistic interpretation holds.

There are many many advices and procdures for "model selection" (Ehsan reported an example). This is a big topic. In the end, to my opinion, all these strategies are unsatisfactory. There is only one good way to select a model: THINK! Use your knowledge about known, reasonable, expected, sensible, obvious,... relationships. Include the variables YOU consider relevant, independent of their performance in the model (by reducing the adj,R² or the AIC or whatever measure is taken). If you are not sure about the form of the relationship to the DV (linear, quadratic, exponential...) and none can be favoured for logical reasons, use the simplest ("Occhams razor").

I strongly endorse Jochen's comprehensive answer. The only thing I want to add is that looking for information on "GLM" will turn up far more on the General Linear Model (still linear functions of the weighted sums) than on the GeneralIZED Linear Model (with nonlinear link functions to the outcome, such as logistic).

I don't know that there's a standard initialism for the Generalized Linear Model.

Good to know. I used to think that "LM" is for (general) linear model and GLM is for generalIZED linear models. But there seems to be some confusion in the English litarature. In German there is a clear distinction as "ALM" vs. "GLM" ("Allgemeine Lineare Modelle" vs. "Generalisierte Lineare Modelle"), maybe there is something similar in Italian.

However, a good start to google for LMs specifically is to search for

+"linear model" -"generalized"

and for GLMs specifically is

"generalized linear model"

Good luck.

Jochen, In my training, the G(eneral)LM refers to the general model encompassing ANOVA, ANCOVA, linear regression and so forth. I think that's the standard usage in English-language social-science texts.

We both learned something!

Ok, but if GLM, as you say, is for the general LMs, how are then generalized LMs distinguished (by abbreviation)? I just saw that in Wikipedia, both variants are abbreviated as GLM. Once I saw GZLM for the latter, but I think this is not common. (it is really unfortunate that both words start with "g"! AFAIK even McCullagh and Nelde were unhappy about the wording)

As I said, I don't know that there's a standard (English-language) initialism. It is unfortunate!

It just occurred to me (after I hit return) that it may be SAS's fault. PROC GLM handles general, but not generalized, linear models. It could have spread from there.

Hi Jochen and Patrick, thank you very much for your answers. I am kind of familiar with the fit of general linear models. Here are some doubts that led me to ask that question:

a. My variables are born nonlinear, i.e., they are logarithms; I guess GLM works with that, too. It should be like applying a log-transformation to ordinary data before analysis. Is this correct?

b. I don't have a priori reasons to state the mathematical relationships between variables. I suppose I can only try and extract the beta and look how the model performs; but I stated in my question that I don't know about the mathematical relationship because I'm not particularly interested in that. A black box simulator would be fine for me, if it makes good predictions.

c. The predictors covary. Should I orthogonalize them first? Like with a PCA? Is there a way to use the information I have about the covariance of predictors?

Thank you very much for your suggestion! I was thinking the discussion would lead me somewhere near to bayesian models, but if GLM works, than it's maybe easier.

So I'd say that it still applies what I said:

- set up some model, as simple as possible

- check the residuals

- if they are not ok, transform the predictors or even the response, re-check the residuals. Possibly include interactions between the predictors, re-check the residuals

- when you have co-varying predictors, check the variance inflation factors (VIF). If they are not extraordinary high, everything is ok. If not, PCA is a smart method to get a linear combination of the co-varying predictors as a single, new surrogate predictor (or a set of orthogonal predictors). If the correlation is very strong it might be sensible to use only one of the correlating predictors (and thus make a simpler model).

- since you don't have any "external" guide to build the model you will likely end up with a model that overfits the data. However, this might be the best you can do anyway, so it is ok. But: keep in mind that the confidence intervals will be too small to fulfill the frequentistic expectations.

AFAIK Bayesian methods also do not provide (useful) means for model building/selection.The only possible basis is common sence and scientific reasoning. Data will and can never ever tell you which kind of model is best for future predictions!

Giulio,

If you're ok with a black box, this sounds like a good situation for Partial Least Squares modeling.

Note that there are (at least) two common uses of PLS: One is for simultaneous regression relations, and there is some controversy about its pros and costs. Here, I'm suggesting just using it for a surrogate measurement model, for which it seems an excellent fit.

It will give you a result very similar to the first principal component, but weighted for predictive utility to a criterion, rather than variance accounted for in the components. If you're familiar with canonical correlation, it's the same concept. You get a weighted sum that gives the maximal prediction of the outcome.

This does not address nonlinear relations, but it might be a good last step.