Unsupervised classification (clustering) is a wonderful tool for discovering patterns in data. I know that it is also an ill-posed problem, but is it thinkable to do cross validation, for example on k-means using internal validity indexes (silhouette, etc.)? For example, a k-fold cross validation is applicable following this schema? I choose a battery of internal validity indexes, I derive k means from the train set then I compute validity indexes on each fold (not on train set as it is usually done). In your opinion, is this approach more consistent with respect the classical one (namely, computing the validity indexes on the whole dataset considered as the train one)? Except the Gabriel model, there exist other cross validation frameworks, and what are preferable?
Hi Antonio,
to me your approach, i.e. deriving quality parameters from folds, sounds good. What you'll get is an idea on the validity of your validity index in a statistical sense.
However, a problem might arise from too few instances or highly imbalanced cluster sizes: I could imagine that the procedure becomes troublesome if not all clusters are contained in each of your folds (i.e. if the probability is high that not all clusters are contained).
Greetings, Sebastian
Thanks, Sebastian, surely I agree with the choice of a validity index.
Let's say that I chose the k-means and the Silhouette score and let consider highly imbalanced cluster sizes (in general, we don't know the size of clusters and more frequently also the number of clusters too).
If not all clusters are contained in each fold this could not have an effect on the Silhouette score. Indeed, I take my fold and I compute the Silhouette score for each "test" unit with respect the solution on the "train" one.
What do you think?
Cross-validations are techniques to measure generalization capacity of any regression against over-fitting or other limitations by comparing several statistical models, which can further be used for better regression by ensemble average.
Conventional Validation works with single partition of sample data with one training set and one testing set, where training set is used to train model and testing set is used to measure the generalization capacity of the trained model. Contrary, cross validation works with multiple such partitioning of the sample data to get more insight about the generalization capacity of any dataset.
- Leave-k-out cross-validation left out k observation at each step and k-fold cross-validation use one out of k subsamples partitioned randomly from original sample at each step.
- Differences between Monte Carlo cross-validation (repeated random sub-sampling validation) and k-fold cross validation is discussed here.
- In linear regression, closed form expression for cross validation is available.
- Cross-validation with data, which are not independent needs dependency related cross-validation like cross validation in time series partitions data in different segments of time series.
- Badges
- Science topic
- Similar topics
- Organizational Psychology
- Training