@Gant: please check the following references:

http://www.sciencedirect.com/science/article/pii/S0022522306002339

http://www.sciencedirect.com/science/article/pii/S0895435603000477

http://www.unt.edu/rss/class/Jon/MiscDocs/Harrell_1996.pdf

To my opinion, the advantages of bootstrap validation against data-splitting and cross-validation are clear. Also, I think that the argument of credibility is a non-issue, as all these approaches have been theoretically investigated and provide samples from the same underlying theoretical population.

Hi Justin,

the (automated) approach you propose is quite prone to overfitting (especially for small sample sizes), as you'll evaluate numerous possibilities and discard options that do not perform well in the available sample(s). Please consider penalization that is not only based on #included factors in the model, but also on #considered scenarios.

@Thomas: Thanks for the response. I don't believe there is much overfitting at this point as the feature set is small (~20) and their are about 10K training instances.

that is good news, as it simplifies the complexity of the learning task (depending on the quality of your data of course). Outlier detection, filtering etc. is recommended when you think they are a problem in your data. Alternative approaches to deal with noise/outliers may consider non-parametric or semi-parametric learning algorithms, or parametric algorithms with higher tails (eg. Cauchy instead of normal distribution) or (weakly) informative prior distributions on the parameter estimates. Filtering is more an issue in signal processing, where it is typically used to suppress or amplify parts of the data in order to make it more informative. However, when your data is high quality, all these approaches may result in unnecessary loss of information (=> less confidence, depending on the technique (additional) bias can also be introduced). If you consider ensemble methods, try to integrate over all potential sources of uncertainty (sample vs population, feature extraction and parameter estimation) as this may considerably improve the quality of your estimates.

Thanks for the advice. Could you elaborate on what you mean by "try to integrate over all potential sources of uncertainty". Specifically, what does this mean? In other words, what tangible steps should I be taking to do so? Thanks in advance!

I would add a zeroth task to your list above:

0. Data preparation/cleaning/validation

You may have "dirty" data or outliers that need to be removed before model fitting or feature extraction. Also, you may think you have the right data or the right alignment of the data, but this may not be the case. Data usually comes with consistency checks implicitly stated - different files have the same number of lines, each field has only certain values, etc. Get familiar with your data by creating and verifying those consistency checks.

collect data -> pre-processing data -> Feature extraction/selection -> regression/classification

This could be a general procedure. The most effort you can do in former steps the best. But the specific methods and techniques in each step highly depends on the nature of the data and the problem you want to solve.

I don't understand how step 4 ties in with steps 1 through 3. It sounds to me like you seek to implement a different learning algorithm at step 1, assess it at 2 and repeat 1 and 2. In this case, step 2 will have nothing to do with a regression or decision tree - but rather, will require an assessment tool like ROC or LIFT....

with "integrate over all potential sources of uncertainty" I mean the following. Each (discrete) choice your algorithm makes relies on some observations in the data, and the observed data itself represents a sample from the population. This means that selected features, parameter estimates, model complexity (such as nonlinearity terms) etc. etc. may be different in new samples, i.e. each choice has a probability of being the "correct" one. By integrating over all potential sources of uncertainty (i.e. the underlying distributions), you allow the model to account for uncertainty in each choice. Approaches to integrate over this uncertainty may rely on techniques such as (Bayesian) model averaging or bootstrapping.

I try to visualize the data in any way at the very first stage. Since Pattern Recognition is discovering structure in the data, it is good to first know it from different perspectives. You might need for this good ways of visualizing high-dimensional data. Some tools that I use: Self-Organizing Maps, GGOBI, Sammon mapping, scatter plots, histograms.

And in my opinion the first stage you mention (feature extraction) can be done more effectively when you count with the help of an expert in the field where the pattern recognition problem is applied, e.g. if you are building a diagnostic tool for predicting Parkinson, you might better count with the help of a neurologist.

I think as well some people prefer the brute force approach: put everything in the machine as quickly as possible and make all possible combinations till you found a solution good enough from the performance point of view. This might work, but you miss some experience and learning on the way there.

And lastly always reserve some data to test the hypothesis you might be doing over the way (including the training of adaptive algorithms).

@Andres & Miguel: Thank you. It certainly could be that I have not done enough in the pre-processing to ensure the data is precisely as I expect it to be. I will add that to my to do list. :-)

@Kassim: Perhaps you could state the steps that you typically take in the modelling process. I'm not sure that I understand your assessment of the example steps that I put out there; but, regardless, I am more interested in hearing how you approach modelling problems. Thanks.

@Thomas: Thank you again for responding; I value your input. In your opinion, would you say that approaches such as RandomForests that already include bootstrapping do not typically do a good job at accounting for these sources of uncertainty? I want to make certain that I fully understand your points.

Random Forest does a great job of accounting for all sources of variability, while providing a good balance between bias (underfit) and variance (overfit) precisely because the estimate is formed by bootstrapping over-fitted classification trees, the final result being the majority vote. The only downside is that the resulting fit is a weighted average of individual classification trees and as such is more or less a black box rather than providing direct insight into the functional relationship between the outcome and the predictors. It is especially useful in "wide data problems" (few observations but many predictors) . You mentioned you have around 10K training instances and around 20 predictors--however when you start to consider interactions your data is not "tall" by any means--and depending how you limit interactions (limiting to 2 way interactions gives 20 + (20 choose 2) = 210 predictors, 3 way interaction limit gives 210 + (20 choose 3) = 780. Random forests would not limit the number of interactions so you would have possible 2^20 predictors which far exceeds 10K. Each CART would be way overfit, but the estimator is robust because its a bootstrap average over individual overfit CARTS. This is fine if all you need is a means of computing the expected outcome for a given set of predictors--but if you want to know how a given predictor effects the outcome by looking at an equation your out of luck--you can play with starting with a point in prediction space and investigate what happens when you alter one of the predictors, but given the complex nature of the fit, the qualitative behavior will be quite different in different neighborhoods of the predictor space. If you want a functional form than you'll have to go with regression analysis--

logistic if your outcome is binary. If your going in this direction then specify a super model first (all 20 predictors? up to quadratic/cubic in some ? which interactions are possible up to 2nd order?) and then use the difference between model deviances (logistic) or likelihood ratio (normal regression) to conduct backward model selection (omit nonsignificant variables one at a time and test that the smaller model fits as well as the larger model.

To recap the above,

A.

collection and pre-processing --especially measurements taken by laboratory equipment--often the investigators will tell you that the machine comes with software which is widely used by the community and you don't need to worry about the raw data -> analytic data step. This in of itself is a red flag-- typically the methods supplied with the machines are horrible -- MAS5 in the microarray world, as opposed to rma for example. You should at least be aware if and how data are being summarized before they arrive to you.

B. Set aside validation set -- from your above comments it looks like you have a reserved blinded validation set

in fact , institutional structure should be set up in advance to insure that you have never touched the validation dataset.l

C. Train-- either random forest if black box is sufficient or backwards selection . Both of these provide diagnostics which summarize the quality of fit -- RF gives the out of bag model accuracy (meaning that in each bootstrap sample, the training instances not selected are used to compute accuracy of the respective tree, and this is almost as good as a validation data model diagnostic, but not quite), logistic regression has deviance between the fitted and null models, normal regression has likelihood ratio, but both of these are using the data they trained on to assess fit.

D. Model validation -- in all cases we send the trained model to institution B and they run the blinded validation data through it and compute model concordance etcetera

Hope you find some of this helpful

@Aureli: Thank you for the tips. I have not equipped myself well with visualization tools yet. Mostly reviewing summary data because of the environment that I'm working in. I hope to move more of the analysis to R, where I can more easily view the data. I'm open to any advice on visualization.

@Grant: I appreciate the detailed and specific response. The problem at hand is a regression problem over a range of values (not binary). So far, I have used classical linear regression and random forests regression. Segmenting the data has helped put linear regression on par with RF at this point in the process. Again, thanks.

In addition to Grant's post: instead of performing a hold-out validation you may be interested in bootstrap validations. These techniques have been demonstrated to be very powerful, as they allow you to

- use the full data for training

- get an estimate of optimism (overfitting) of the model

Also, the holdout procedure has some additional disadvantages such as: validation results depend on the selection and size of the sample, and does not provide insight into generalizability across populations (but merely reflects how reproducible your model is). Of course, the ideal situation is numerous independent validation samples.

Furthermore, to reply on point C (Train). Instead of deviance diagnostics (which is directly related to the maximum likelihood estimate: deviance = -2Log L if I remember correctly), you may be interested in penalized diagnostics such as AIC or BIC that take model complexity into account. A model with (nearly) zero deviance is likely to be severely overfitted, especially when the model includes numerous covariates. Penalization will allow you to find a good balance between bias-variance tradeoff.

I assumed that you have done all preliminary EDAs and that you have labelled data for supervised modelling. The reason I said step 4 wouldn't fit in with the other steps is because it involves multiple disparate models. If step 4 is confined to, say, decision trees, you could fit multiple models via cross validation, bootstrap or bagging before selecting the "optimal" model. The same applies to multiple neural network architectures or different versions of support vector machines. The assessment I refer to (at your step 2) relates to each of these individual models...

@Thomas: holdout validation is always superior to bootstrap validation at least from the standpoint of credibility if for no other reason--the choice to be made is whether the size of the predictor space relative to the size of the training set permits a leave out validation set or not.

@Thomas: I will have to look more into bootstrapping. I also appreciate the tips on penalization.

@Kassim: Thank you for the clarification.

@Gant: please check the following references:

http://www.sciencedirect.com/science/article/pii/S0022522306002339

http://www.sciencedirect.com/science/article/pii/S0895435603000477

http://www.unt.edu/rss/class/Jon/MiscDocs/Harrell_1996.pdf

To my opinion, the advantages of bootstrap validation against data-splitting and cross-validation are clear. Also, I think that the argument of credibility is a non-issue, as all these approaches have been theoretically investigated and provide samples from the same underlying theoretical population.

I am aware of those publications and of the theoretical results These theoretical results begin with the assumption that the CARTS in the forest are independent which is true under the null hypothesis of no association. However, imagine that the feature space contains a few unknown very strong predictors (and their interactions maybe) and a lot of garbage. The individual CARTs in the tree will always randomize over a small subset (square root n or log n) of predictors to optimize over --this is true and in the direction of lending statistical independence to the CARTs, however, since each CART is grown to complete node homogeneity i.e. perfect fit, on its bootstrap sample, the chance is very great that to each replicate in the dataset there are many CARTs for which it was out of bag but for which the same strong predictors were selected at some stage, the very same strong predictors also are likely to be selected into the trees that this replicate did train, ergo correlation between a tree that the replicate did train and the tree for which that replicate is out of bag for and being counted as a source of independent validation.

So there is a proof in there somewhere. To conclude on a very clear point, can you verify that the above citations do _not_ assume statistical independence between CARTs in the forest?

I have not read all of the responses, so forgive me if I repeat anything already said. If you have a high-dimensional problem, I suggest dimensionality reduction before doing anything else. This usually involves finding a good approximation of a lower dimensional (possibly non-Euclidean) manifold embedded in your high-dimensional Euclidean space. It turns out that Euclidean metrics in high dimensions aren't terribly useful, and thus it's often better to substitute them with non-Euclidean metrics to find the distance between sample features. Inner products are used to measure distance, but there are many different functions that satisfy the requirements to be an inner product. The standard measure of distance in Euclidean space is the 2-norm, but if we replace the 2-norm with a different metric, say a Gaussian Kernel, then we're essentially working in a non-Euclidean space. This specific technique is known as "the kernel trick". Also check out algorithms like Local Linear Embedding, and Laplacian Eigenmaps for very good manifold approximations.

Once you've projected your problem onto a non-Euclidean manifold, you may discover that you've transformed a very hard non-linear problem into a relatively easy linear problem, the solutions to which are plentiful and well-known.

@Grant: you are correct in the sense that bootstrap samples are not independent. If this is not taken into account during validation, the problems you mention can occur. However, there exist an algorithm (I think it was proposed by Harrell) that estimates the optimism based on the difference between the performance in the bootstrap sample and the performance in the training sample. Usually, the performance in the validation bootstrap sample will be larger than the performance in the training sample (I think because the samples are not independent), and this optimism is then used as a correction factor to estimate the performance in a new, unseen sample. In this manner, overfitting can be identified to some extent. The bootstrap procedure in the random forest allows obtaining multiple different samples, with the result that the strong predictor will sometimes come forward. The corresponding probability then reflects the uncertainty about its predictive value.

@Thomas: I would agree with Grant since your statement that bootstrap allows to "use the full data for training" sounds suspicious to me. If you use all the data, you can not estimate he generalization error with new data (independently of any theoretical statements).

@Aureli: please check he references I provided earlier. The bootstrap procedure is well known, and is able to evaluate generalizability despite your suspicions.

http://www.sciencedirect.com/science/article/pii/S0022522306002339

http://www.sciencedirect.com/science/article/pii/S0895435603000477

http://www.unt.edu/rss/class/Jon/MiscDocs/Harrell_1996.pdf

@Thomas O'Neill: Thanks -- I will have to read up on Locally Linear Embedding.

@Thomas re Harrell 'optimism correction' I was not aware of that --and that definitely is the way to go--so the corrected performance should be unbiased! Good luck on your crusade trying to get bone-head clinicians to buy that!

The steps do not, are unique that depends on the problem you are trying to solve, the techniques can be combined and must conform to the real problem you're trying to model, identifcar or classify. For this there are models in which you can support depending on the phenomenon in question.

As mecionas not try to solve the problem could not define any particular but in general there are three steps to follow:

Parameter extraction

Feature extraction

Classification

luck