Hi Lorenzo,

Whichever method you use. You should be very very careful that you do not over-fit your data. It is very easy to find subsets of discriminatory variables for a given particular small data set, but these variables may not produce a robust generalizable model. Ideally you should have a repeat expt to validate your reduced variable model with. In addtion if access to data is limited use some sort of cross-validation should be used in the selection process. Often nested or double cross-validation is required - where the variable selection is in the inner loop of the validation.

Personally I have used Genetic Algorithm and Genetic Programming techniques to look for optimal variable subset selection. Also Random Forests &TreeNet (Salford Systems) decision trees can be useful, but generally need much more data than you describe.

Methods described by Matti and Emanuele above also look very interesting.

Lasso Logistic regression is useful - and sometimes forward/backward/stepwise regression, but again I cannot stress enough the need to validate any resulting model. Logistic Regression with more than 4/5 independent variables can be very dubious and unstable.

Good luck.

David.

Some refs:

Isabelle, G., & Andr, E. (2003). An introduction to variable and feature selection. J. Mach. Learn. Res., 3, 1157-1182.

Saeys, Y., Inza, I., & Larranaga, P. (2007). A review of feature selection techniques in bioinformatics. Bioinformatics, 23(19), 2507-2517.

Broadhurst, D., & Kell, D. B. (2006). Statistical strategies for avoiding false discoveries in metabolomics and related experiments. Metabolomics, 2(4), 171-196.

CROSS VALIDATION:

Efron, B., & Tibshirani, R. (1997). Improvements on Cross-Validation: The .632+ Bootstrap Method. Journal of the American Statistical Association, 92(438), 548-560.

Filzmoser, P., Liebmann, B., & Varmuza, K. (2009). Repeated double cross validation. Journal of Chemometrics, 23(4), 160-171.

Smit, S., van Breemen, M. l. J., Hoefsloot, H. C. J., Smilde, A. K., Aerts, J. M. F. G., & de Koster, C. G. (2007). Assessing the statistical validity of proteomics based biomarkers. Analytica Chimica Acta, 592(2), 210-217.

Szymanska, E., Saccenti, E., Smilde, A. K., & Westerhuis, J. A. (2012). Double-check: validation of diagnostic statistics for PLS-DA models in metabolomics studies. Metabolomics, 8(Suppl 1), 3-16.

GENETIC ALGORITHMS

Broadhurst, D., Goodacre, R., Jones, A., Rowland, J. J., & Kell, D. B. (1997). Genetic algorithms as a method for variable selection in multiple linear regression and partial least squares regression, with applications to pyrolysis mass spectrometry. Analytica Chimica Acta, 348(1-3), 71-86.

Jarvis, R. M., & Goodacre, R. (2005). Genetic algorithm optimization for pre-processing and variable selection of spectroscopic data. Bioinformatics, 21(7), 860-868.

Dear Lorenzo,

I think other possibilities are available to perform your variable selection, such as ridge regression and lasso, or Bayesian variable selection methods, like BART (Bayesian additive regression trees). The first two methods allow to shrink some of the regression coefficients to zero, using some penalties, while the latter model, using Bayesian regression trees, allows you to perform variable selection, even before doing model selection. All these methods are implemented in R, in different packages.

This setting sits exactly to my GDO programme (General Diagnostic Optimizer) which is written using Bayesian approach, originally in 1980+ to test the system developed by Francis T de Dombal. My programme uses heuristic process to find a subset of variables that explain the outcome better that all variables together. If you send me the materia in excel I could run the programme and return the results. All Best, Matti Kataja, Tampere, Finland ([email protected]).

what ever method you use for this smaller sample size you cannot fit any multivariate techniques.You should have got some warnings when fitting the model.There should be some problems in convergence of algorithms

Hi Lorenzo,

Whichever method you use. You should be very very careful that you do not over-fit your data. It is very easy to find subsets of discriminatory variables for a given particular small data set, but these variables may not produce a robust generalizable model. Ideally you should have a repeat expt to validate your reduced variable model with. In addtion if access to data is limited use some sort of cross-validation should be used in the selection process. Often nested or double cross-validation is required - where the variable selection is in the inner loop of the validation.

Personally I have used Genetic Algorithm and Genetic Programming techniques to look for optimal variable subset selection. Also Random Forests &TreeNet (Salford Systems) decision trees can be useful, but generally need much more data than you describe.

Methods described by Matti and Emanuele above also look very interesting.

Lasso Logistic regression is useful - and sometimes forward/backward/stepwise regression, but again I cannot stress enough the need to validate any resulting model. Logistic Regression with more than 4/5 independent variables can be very dubious and unstable.

Good luck.

David.

Some refs:

Isabelle, G., & Andr, E. (2003). An introduction to variable and feature selection. J. Mach. Learn. Res., 3, 1157-1182.

Saeys, Y., Inza, I., & Larranaga, P. (2007). A review of feature selection techniques in bioinformatics. Bioinformatics, 23(19), 2507-2517.

Broadhurst, D., & Kell, D. B. (2006). Statistical strategies for avoiding false discoveries in metabolomics and related experiments. Metabolomics, 2(4), 171-196.

CROSS VALIDATION:

Efron, B., & Tibshirani, R. (1997). Improvements on Cross-Validation: The .632+ Bootstrap Method. Journal of the American Statistical Association, 92(438), 548-560.

Filzmoser, P., Liebmann, B., & Varmuza, K. (2009). Repeated double cross validation. Journal of Chemometrics, 23(4), 160-171.

Smit, S., van Breemen, M. l. J., Hoefsloot, H. C. J., Smilde, A. K., Aerts, J. M. F. G., & de Koster, C. G. (2007). Assessing the statistical validity of proteomics based biomarkers. Analytica Chimica Acta, 592(2), 210-217.

Szymanska, E., Saccenti, E., Smilde, A. K., & Westerhuis, J. A. (2012). Double-check: validation of diagnostic statistics for PLS-DA models in metabolomics studies. Metabolomics, 8(Suppl 1), 3-16.

GENETIC ALGORITHMS

Broadhurst, D., Goodacre, R., Jones, A., Rowland, J. J., & Kell, D. B. (1997). Genetic algorithms as a method for variable selection in multiple linear regression and partial least squares regression, with applications to pyrolysis mass spectrometry. Analytica Chimica Acta, 348(1-3), 71-86.

Jarvis, R. M., & Goodacre, R. (2005). Genetic algorithm optimization for pre-processing and variable selection of spectroscopic data. Bioinformatics, 21(7), 860-868.

Thank you all for your help.

Emanuele, thanks for your suggestions. I am trying to better understand how LASSO and LARS work and how they can work in Stata. Otherwise I will need to work it out in R. I have downloaded some references on the techniques and the lars.ado program for Stata.

Matti, I will write back to you to ask a reference for the FT de Dombal work you cite. Just wanted to understand the method before doing any other further step. Thank you for your support.

Vasudeva, you are right about problems in fitting the model. I did not even try to put all the independent variables together. That is why I grouped the variables according to PCA results first. I then ran separate stepdown logistic regressions, excluded all variables not retained by these, used in the final model only the variables resulting from the separate stepdown logistic regressions.

Dear David, thanks for your hints. I am aware of the problem of over-fitting, of robustness and validation. The models I developed appear to be robust. I have tried different approaches and the final results are always quite comparable. Molecules might differ in the end, but results are coherent. I have also tried to test models on controls from other studies, and they hold. My worry is that I might not reach the optimal result. I am afraid I might miss some relevant variables, and wanted to find a more “formal” and robust approach for variable selection. I will look into the genetic algorithms as you suggest. Thank you for the references!

Just another couple of issues.

I am not interested in working with virtual dimensions or variables.

My use of PCA was to group variables into affility groups, not to use PC instead of original variables.

The other issue is that I am trying to avoid normality assumption and linear regression.

I am trying to keep logistic regression as the base for the predictive model.

The easiest way is to perform logistic regression for grouped variables because, apparently by his description they are independent of each other, but predict the outcome differently. Regression with more than 30 covariables there is introduces analytical bias (always).

I'm confused with what you meant a "predictive" model. If your objective was truly to predict probability of having an event given a group of predictors, your approach was incorrect. First, case-control sampling, which is sampling conditionally on the outcome status, won’t allow you to estimate a representative regression constant (alpha or beta null whatever you call it). This regression constant is overall probability (like mu null in ANOVA), which would now depend on your case-to-control ratio (i.e., 1:4 in your study). Note that unlike OR estimation, this constant stays if you are to estimate the probability. Only a random sample that is representative of the study population with respect to all covariate strata is valid. If you start with >100 variables, it’s rarely that you would ever get a representative sample even random sampling is applied. I see this kind of misconception about “developing a predictive model” very often, unfortunately. Second, you would need a much larger sample in order to have a precision at an acceptable level for “prediction” comparing to estimation of “association”. These two points already guarantee that your approach is not going to work. Other minor, but important, points are: a. PCA is not suitable for dichotomous outcomes; b. PCA needs a large sample size in order to get a stable estimate.

Perhaps you are saying that you’d like to find a group of “independent” predictors after “adjusting for” all other covariates in the model. Please refer to many excellent articles discussing causal diagrams (a.k.a. DAGs) and more importantly collider bias.

Hi Hsin-Yi,

I am sorry for the confusion.

My objective is to be able to discriminate between cases and controls on the basis of a profile of molecules.

I am not interested in the probability of an event in the general population.

I am already getting a very good and solid result, adopting a covariates PCA to group variables (without including the outcome variable), and then running separate step-down logistic regressions (with the outcome as dep variable), one for each group of variables. And then putting together only the variables resulting as significantly associated with the outcome in each regression.

This works. I reach 100% sensitivity with 78.85% specificity, in a model with 13 significant molecules.

And trying different combinations I get very similar results. And if I use the final model on other controls from other studies, I get the same % of specificity.

As my objective is to build a diagnostic tool, at this stage I am not interested in causality, confounding and interactions, but only in finding a group of variables that than "predict" the outcome: a diagnostic tool.

I have tried different ways for grouping variables (factor analysis, correlation pca...), but the method I described in the best in terms of quality of results.

The purpose of my question was to search for more formal ways of grouping or prioritizing variables, considering I have 100 variables and not enough cases and controls to support such a number of vars. making sure I am not sacrificing the contribution of any relevant variable.

The problem of p>n is of course a hard one. Some comments: 1. Developing a predictive model at this stage of research may be too ambitious. 2. Anyway simple clinical predictors should be taken into account, both for quantification of additional value of the molecules and as a core in a predictive model. 3. Whatever statistical approach is chosen, internal validation considering the full selection process should be attempted to study model instability and optimism, which may be substantial with low numbers.

Lorenzo,

The problem of having more variables than cases is the norm in initial genomics/proteomics problems and the challenge is how to reduce the dimensionality of the predictors. One approach used by Jill Mesirov are to derive "meta-genes" or meta-variables of correlated subvariables and then use these in trying to forward or other logistic regression approaches. We took a similar approach to this in deriving a genomic classifier for coronary artery disease from 113 genes and a classifier development set of approximately 200 cases and 400 controls (Elashoff et al BMC Medical Genomics 2011); we reduced the 113 genes down to approximately 20 meta-variables which were then input for Ridge regression. As others have noted, using an independent validation set is key (we had an additional 526 patients in that

set, Rosenberg et al Annals of Int Med 2010). I would add that using cross-validation or other techniques for calculating false discovery rates at both the individual classifier component and overall classifier level is often useful to estimate the degree of over fitting but is not a substitute for an independent validation set.

Hope this is helpful.

Steve Rosenberg

Hello

Such as you explains that seems to be good but I will like to propose to you to make an automatic selection (stepwise)si you make an exploratory study, without forgetting to check all the conditions for applications (colinearity, linearity, homoscedasticity…).

Can –be in what you have fact is what you checked the various interactions between variables for the which case it is necessary to make test of sensitivity

be carefuly by the sample et predictor number

Hello

You can try the method of Multifactor Dimensionality Reduction.

Th e first thing that comes to my mind is that this stydy seems to be a "Going fishing study", because of the number of variables, more than 100.

By the other hand, 40 cases is a very small number to apply Logistic regression, based on 100 variables, this situation will lead to numerical porblems, and as a consequence, you will have large standard erros and unstable regression coefficinets.

Meanwhile, it would be great, to know why you deciedd to choose 4 controls by case, ¿Was there any specific reason? ¿Can we know it? Becuase more than 3 control by case, uses to be inefficient, the more controls you ahve per cae, the lower infomation you get, and more inefficiente will be your study.

Based on these consideration, before consider doing any statistical procedure, it will be nnie to know what your objetives are, what was the case definition, and what was base population, where the cases come from, based on what you decided to choose 4 controls by case, and so on.

Gus Bergonzoli.

Hi Lorenzo, besides LR with stepwise variable selection, you could also explore classification and regression trees CART in k-fold cross-validation. This technique would also provide you with understandable rules ( for example differing levels of molecules in cases and controls) and at the same time the stable variables, responsible for separation between the cases and controls, that appear in various sub-divisions of your data. Your data sample is unbalanced, you have more controls than cases, so your feature selection might be biased. You could account for this and verify the results by running your training procedures on selected balanced data sub-samples. Erinija

If your data is highly correlated (i.e. your variables are not completely independent - as is often seen in 'omic data) and your incorporate a projection method in your model (PLS, Neural Net, PC-LDA, PC-CVA, RBF-regression) - OR there are a subset of variables that are very highly correlated (linear or non-linearly) and your signal to noise is very very good then Dana's comment may not hold true. Without some estimate of within group and between group dispersion (e.g. variance) it is impossible to be so certain of sample numbers.

Having said that if you do a power calculation based on a hypothetical expected predictive accuracy, with reasonable confidence intervals then your numbers are quite low unless you are getting a ROC curve with greater than AUC 0.9 - i.e. fairly clear discrimination. That is assuming that your model is very stable.

Remember there is a big difference in proving that there is a significant association between variables than providing an effective biomarker.

No offense intended to Dana, I too am very skeptical of using powerful non-linear methods with small data - but it is not impossible.

D.

Thank you Dana, and thank you David.

I am now is a critical phase. I am waiting for new data on both controls and cases.

I am finally oriented in doing an interated LASSO (a first LASSO to determine the weights and identify the factors to be used in a final Adaptive LASSO model), as explained by Huang, Ma and Zhang (http://www.stat.uiowa.edu/sites/default/files/techrep/tr392.pdf).

Still trying to understand how to correctly use the different R packages in order to do that properly.

I am not quite interested in a biomarker, or in describing bivariate associations.

I do have highly correlated variables. And I can get an AUC higher that 0.95 simply carrying a stepwise logistic regression. But I am not happy with that (no cross-validation, not sure about the robustness, arbitrary choices to avoid overfitting ...).

The problem of doing descriptive analysis is, for example, that if I simply look for factors that are associated with the dichotomous outcome, I get a good set of factors. But then, if I put the factors is a multivariate model, I get variables to play a significant role in the model that where not at all associated with the outcome just by themselves...

Of course this is quite obvious in terms of bivariate vs. multivariate analysis, with interactions and coufounding playing their roles.

My final objective, however, is to build a "predictive" model. I need a pool of variables to tell me finally if a subject has or does not have the disease, possibly with close to 100% sensitivity and high enough specificity.

Finally, I would like to avoid meta-variables as a way to reduce dimensionality. the 100 VOCs I have are already difficult to identify, as I only know them by their molecular weight. and I would finally like to identify them. Thus, I am not interested in virtual variables that do not have any practical meaning.

Lorenzo

Ho Lorenzo.

Your last response does complicate things. Do the principal components show variables strongly representing the component? How many components show high eigenvalues? How much of the variance is covered with them?

Yes I agree with Dana's last comment. Hopefully Lorenzo, you also have a comprehensive database of clinical variables (age, sex, bmi, drugs, risk factors). Adding these into your analysis may dramatically improve the stability of your model. Or at least they are worth investigating w.r.t. each VOC measurement - using something simple like logistic regression. It is possible that as well as correcting for confounders, you may find a clinical variable which is strongly correlated to a key VOC measurement - to the point where one could substitute with the clinical variable, which may be much simpler and cheaper to measure. Just a thought.

Thank you Sandor.

Actually, only the first component shows a high eigenvalue (31) and most factors are associated with this component.

The secondo component has 8 as eigenvalue, followed by 6.5, 4.5, 3.9, 2.8...

The first three components covers 55% of variance. The first 10 cover 79%.

Dana: yes, we will also represent data in a descriptive way. for example, I believe it will be important to graph out the delta of VOCs profiles of cases vs. controls.

And we will present the main characteristics of the distribution of each VOC in cases and controls.

No problem in sharing the data, once we have completed the analysis.

And yes, David, we have collected all relevant (we hope!) variables related to the clinical condition of both cases and controls, including demographics, drugs, aspects of the diseases, etc...

I am for now limiting the inclusion of these variables in the models, just because we are in a preliminary phase. But I can anticipate, for example, that age is significant in the model about Crohn's disease, while it is not significant in the model of Ulcerative Colitis.