Basically pc''s are the eigen vectors corresponding to the eigenvalues in increasing order. ..ic pc1 corresponding to largest eigenvalue which captures the maximum variance data points. ...where these eigenvalues calculated on the covariance matrix ..which has been formed from the given data matrix.

PC's in PCA are the vectors representing direction of variance of data. PC corresponding to highest eigenvalue is the direction of max variance.

Dear Nima

PCAs and how they're calculated seems clear, and has been explained before.

For your actual question: It is very hard to pinpoint the exact meaning of the actual values on a PC. Because they are calculated from a mix of variables it is possible to get exactly the same PC1-value for different individuals (datapoints) even if they differ in all variables that were used in the PCA. So in my opinion the PC-values per se are not informative. But you still can say something like "individuals with a high PC1-value are rather [enter the most important variables for PC1 (be aware of the direction (negative or positive))]...". But always also report the loadings of all variables on the different PC-axis (as a table or supplementary table).

Just always be aware what you are looking at if you interpret results from statistics.

Hope this helps and good luck with your project.

The definitions you've been told are correct. To put it in simpler terms, each PC is a direction in your feature space that's highly relevant to determine what the output is. Each PC is less important than the last, as each eigenvector represents less variance.

The trade off you get with PCA is exactly what you're asking: your features are way less interpretable, as they are derived from the higher dimesional input space. You could look into each PC and relate it to the original inputs, but then you will be just sorting your input features by variance.

These numbers reflect the participation of the training samples in each PC.

Take the famous eigenface example, suppose each person's image has 1000*1000 pixels, we reshape it into a tall (1million, 1) vector. Suppose we have 5000 people's image, so our dataset is in the shape of (1 mil, 5000).

After centering the dataset, we take full SVD:

x = U * S * V.T

this can been viewed as linear combination of column vectors of U from V.T and scaling by S.

What PCA does is to take first r (n_components) columns vectors of U and truncate the others, and use the linear combination of this r column vectors to approximate the original dataset.

Suppose we do a 2 components PCA, U will be in shape of (1 mil, 2). Pick one data vector R, which is in shape of (1 mil, 1):

P (PC Scores) = U.T * R

(2 by 1 vector)

PC Scores represent the linear combination of of truncated U to approximate R. Once you have truncated U matrix stored, each data point can be represented just by TWO PC scores instead of 1 MILLION numbers, since you can easily reconstruct the approximated data by U * P.

So basically PC scores mean nothing, if the principal components have no realistic meaning. One of the most popular usage of PCA is to do cluster analysis of similar dataset, since PC scores can be easily visualized in lower dimension.

Short answer: in the most general scenario, you can't. In the rotated feature space, values are very hard to interpret in most cases, and you'd be very very lucky to hit a jackpot with getting some PC with clear meaning in the real world.

Longer writeup:

Another huge drawback of PCA is that it is dataset dependent - if you make a pipeline with PCA in it, prepare to deal with eigenvalues order depending on certain classes prevalence in the set [if you intend to use PCA for feature reduction before some classification task] and other nastiness making it bad for engineering.

With all that said, there is a reason it is in every data scientist toolbox, why it is often used at the very start of approaching a new problem and why no 101 class on data processing omits it. That reason is that PCA excels at giving you some insights about the data with minimal effort and, perhaps the most importantly, dimensionality reduction suggests how many independent variables are there to consider in your model. This is also the reason the broken stick model is so popular and why you usually should pay attention to the scree plot (in oversimplified terms, the sharper the dropoff at N is, the fewer meaningful factors you lose when performing the dimensionality reduction). Biplots are a tool to offer some explanations on the meaning of PCs but in my experience, they mostly tell you what you already know and are only good for making nice-looking plots.

PCA is also fit for clusterization tasks in simpler cases, and if you do something like NLPCA or kernel [trick] PCA, you can take it one step further. But all in all, I'd suggest not obsessing over PCA - most insights you can get from it is the number of components and, perhaps, something like K-means on the first few components to roughly tell you what the clusters are.

If you need to come up with the model which is transparent w.r.t. measured and predicted variables and does not have this black box in the middle where the variables are all mixed up, do some feature selection instead; some of the methods are based on PCA or variance in general, in some form - there are very very few other options when considering a general case without any other a priori given information. Be also warned that PCA is fit for the cases where you really know nothing about the data yet and, therefore, must assume normality in the data. Heavy-tailed or, worse, multimodal distributions tend to break it.

It is no silver bullet by any means and usually it won't be the core of the model which can reliably and robustly solve real world problems, but one still absolutely should do it to have a better grasp on how their data looks like and what to expect in general from the problem you're looking at.