StatQuest: Principal Component Analysis (PCA), Step-by-Step

NOTE 1: The StatQuest PCA Study Guide is available! https://app.gumroad.com/statquest
NOTE 2: A lot of people ask about how, in 3-D, the 3rd PC can be perpendicular to both PC1 and PC2. Regardless of the number of dimensions, all principal components are perpendicular to each other. If that sounds insane, consider a 2-d graph, the x and y axes are perpendicular to each other. Now consider a 3-d graph, the x, y and z axes are all perpendicular to each other. Now consider a 4-d graph..... etc.
NOTE 3: A lot of people ask about the covariance matrix. There are two ways to do PCA: 1) The old way, which applies eigen-decomposition to the covariance matrix and 2) The new way, which applies singular value decomposition to the raw data. This video describes the new way, which is preferred because, from a computational stand point, it is more stable.
NOTE 4: A lot of people ask how fitting this line is different from Linear Regression. In Linear Regression we are trying to maintain a relationship between a value on the x-axis, and the value it would predict on the y-axis. In other words, the x-axis is used to predict values on the y-axis. This is why we use the vertical distance to measure error - because that tells us how far off our prediction is for the true value. In PCA, no such relationship exists, so we minimize the perpendicular distances between the data and the line.
NOTE 5: A lot of people wonder why we divide the sums of the squares by n-1 instead of n. To be honest, in this context, you can probably use 'n' or 'n-1'. 'n-1' is traditionally used because it prevents us from underestimating the variance - in other words, it's related to how statistics are calculated. If you want to learn more, see: https://youtu.be/vikkiwjQqfU https://youtu.be/SzZ6GpcfoQY and https://youtu.be/sHRBg6BhKjI (the last video specifically addresses the 'n' vs 'n-1' thing, but the first two give background that you need to understand first).

Support StatQuest by buying my books The StatQuest Illustrated Guide to Machine Learning, The StatQuest Illustrated Guide to Neural Networks and AI, or a Study Guide or Merch!!! https://statquest.org/statquest-store/

Womp womp

What tool or statistical package can I use for PCA

best best....super best video!!!❤

I used to study with your videos when i was a student and even now after working for 4 years in the industry as a ML engineer i still come back to your videos to understand things simply. Thank you for making all these videos. ❤

Again Claude recommended this to me❤

I love you

Thank you a lot fot this amazing video BAM!!.

I couldn't have understood it any better. Amazing BAM!!

This video opened my eyes after 8 years!!!!!!!!

🎯 Key points for quick navigation:

00:00 PCA SVD introduction
00:33 Genes mice dataset
01:04 One gene line
01:37 Two genes clustering
02:15 Three genes 3D
02:53 PCA dimensional reduction
03:32 Center data origin
04:06 Shift center origin
04:41 Fit line origin
05:16 Project measure distances
05:46 Maximize projected distances
06:54 Pythagorean inverse relation
07:25 Maximize squared distances
09:02 PC1 best line
09:36 PC1 slope recipe
10:14 Linear combination variables
11:16 Scale unit vector
12:23 Eigenvector loading scores
12:57 PC2 perpendicular PC1
14:24 Project samples plot
14:52 Eigenvalues variation measure
15:44 Variation proportions PCs
16:14 Scree plot variation
16:44 3D PCA process
18:22 Scree plot proportions
19:40 2D approximation good
20:03 High-D PCA works
20:58 Clusters despite noise

Made with HARPA AI

So far the best video :)

Okay, now I somewhat understand what the linear algebra equations we're doing.

How do you get PC1 and PC2 without being able to plot the n-dimensional graph? That whole jump confused me.