I'm working with a large dataset (P = 13, N = ~4500) with lot's of nonlinearity. I'm interested in reducing dimensions prior to clustering.
Kernel PCA seems like a good approach. I'm wondering if it makes sense to judge the rough quality of each fit by comparing the sum of the first few eigenvalues (equivalent to total explained variance..) over multiple values for sigma. Working with a Gaussian kernel (for now at least), I'm fitting a kpca multiple times, varying sigma over several orders of magnitude, from .0002 to 20, and retaining the first 10 features of each fit. My hope is to identify the best of my current test values for sigma, and then contract the search range and home in iteratively.
I haven't found a lot of guidance on how to tune sigma that aren't - to be honest - over my head. Reconstruction error seems like a promising alternate metric, but I'm not sure how to implement it in R.
I'm grateful for any thoughts or suggestions.
Send her data to me, hopefully I can learn it and can answer your problem
Dear Rafter,
I have no answer, only questions.
1. I would like to be sure that 'p' means for you variables and 'n' means objects.
And you want to cluster the objects.
And reducing dimmensionality means for you constructing few new features (
Dear Anna,
Thank you for your response.
To answer your questions in turn:
1. a. Yes, I use p to mean variables and n to mean observations.
b. Yes, I want to cluster the objects.
c. I would be interested in dimensionality reduction if it were interpretable, but since it appears that nonlinear DR methods are difficult to interpret, I will let that go in favor of producing a useful clustering.
2. I am using the kpca function from the kernlab package in R (see https://cran.r-project.org/web/packages/kernlab/kernlab.pdf) . It seems to be based on the Scholkopf, Smola, and Muller paper you cite below. I believe it uses the kernel trick. It provides the sorted eigenvalues, which are non-negative.
3. I do know the paper, but it is well outside of my area of expertise so my understanding of it is rather approximate.
I see that the python package scikit-learn has a function to calculate the approximate reconstruction pre-image and thereby tune hyperparameters by minimizing the reconstruction error. I have not worked in python, however, so that would be a hurdle.
Thank you for your time and input.
Warmly,
Rafter
Dear Rafter,
Thanks for the replay.
We are working on a similar problem. For the moment, we have sampled from the big N only n=1000 data vectors. Our p is equal 15.
We wanted to see if kernel PCA can give more as traditional (classic PCA). Our data contain generally 4 subgroups, what is known by us, but of course is unknown both to the PCA and kPCA algorithms.
The results are described in the enclosed manuscript.
Both methods computed principal components, but only the first 3 PC-s
yielded meaningful graphical interpretation on the group structures.
Concerning the reconstruction issues, the classical PCA yields a perfect
reconstruction when using all 15 PC-s: and one can make in steps better and better reconstruction when using 1, 2, 3, ...etc PCs. This is well known.
However, I think I have seen it somewhere in Scholkopf's group papers that
when the PC-s were obtained by the kernel PCA method, this cannot be done (myself, I think, this is impossible if the kernel PCA uses the 'kernel trick').
I would be interested what do you think about our results.
You might write me directly by email.
Wishing much success with your project
Anna
Thank you, Anna. I will read your paper with interest.
The problem of a calculating measure to use to tune kernel hyperparameters remains. For others facing a similar problem, the Python library Scikit Learn allows for the calculation of reconstruction error using approximate pre-images. Unfortunately I don't use Python, and have not found a similar function in R!
- Badges
- Science topic