I am aware of using PCA (principal component analysis) as feature reduction, it reduce the m number of features into n features, where n is number of PCA component. In this case we already have m features.
But in some places I read PCA as feature extraction. I am unable to get this concept. Does PCA in this case produce some new features n adding to previous features m (m+n). Or there are no feature previously and PCA generate some new features. Can some one explain it please?
When doing PCA, and assuming your number of samples > number of features (m), you will get m principal components. To use PCA as a feature reduction method, one would use a truncated PCA representation where the m most important principal components are selected while ignoring the others. The number of selected components m can be chosen such that those components can achieve acceptable reconstruction of a set of images (or any sample set).
PCA does not add any new features, but it provides a small set of important features. A famous example is the eigenface example (you can find illustrations online). For a face classification problem with images of size nxn, the total number of features in the raw data is n^2 (# of pixels). When applying truncated PCA, you end up with a much smaller set of features that are representative. Now a single feature from PCA is not a pixel, but a full nxn image where typically each of them represent some aspect of a face. If you examine the first few principal components, you can easily tell that they carry the basic structure of a face. From this perspective, PCA is considered a feature extraction method.
Checking the images from eigenfaces can really help grasp this idea.
Hope this helps!
Mohamed Baker Alawieh what i get PCA does not extract new features, it just gives the best one from previously existing features
Looks like this point was not expressed clearly from my side. Sorry for that, here is a detailed address of the point.
PCA does provide new features but these features are not independent of the original ones, PCA maps your data into a different space. So if you are doing any learning task, using both original and PCA features does not make sense because they are redundant (if PCA is not truncated).
Each feature of the PCA is a linear combination of the original features, and the most important PCA feature is the one the can best explain the data.
For example if you have a set of 500 128x128 images, and you find the top 20 principal components can achieve 95% reconstruction, now you can train you model with 20 features instead of 128^2, which means you would expect better learning given the limited dataset. These 20 features are combinations of the original ones, however, they carry roughly 95% of the information in your original features.
In short, the PCA features are a better compact representation of your original features. No additional information is provided, but a different representation of original data. When PCA is referred to as feature extraction method, it is this notion of a better compact representation that is in mind.
- Badges
- Science topic