What is PCA (Principal Component Analysis)? [DS Certification Prep]
1. Overview
PCA (Principal Component Analysis) is a technique for summarizing data with many features into "fewer dimensions."
In other words, it is a method for compressing data while losing as little information as possible.
For example, even with data containing 10 features, it is sometimes possible to explain the overall trend quite well using only "2-3 axes (principal components)."
PCA mathematically finds the "direction that best represents the variance of the data."
2. Classification of Learning Models
PCA is classified as "unsupervised learning."
The reason is that it discovers structures by looking only at the "relationships between features (X)" without the presence of correct labels (Y).
Rather than "predicting correct answers" like supervised learning, its purpose is to
organize and compress the structure or patterns of the data.
3. Conceptual Logic
It searches for the "direction in which the data varies the most (the direction where variance is maximized)" and redefines that direction as a "new axis."
Process
Standardize each feature (mean 0, variance 1)
Calculate the covariance matrix (representing the relationships between features)
Calculate eigenvalues and eigenvectors (find the direction of maximum variance)
Sort by eigenvalue in descending order and adopt the top vectors as "principal component axes"
As a result, data is projected onto **new axes (principal components)** created from combinations of the original features, allowing for 2D or 3D visualization or compression of model inputs.
4. Computational Logic
Here, we will break down the process above a bit more mathematically.
PCA creates new axes through an operation called "eigenvalue decomposition of the covariance matrix."
1) Centering (mean to 0)
Subtract the mean from each feature to align the center of the data to the origin.
$$
X_{centered} = X - \bar{X}
$$
2) Create a covariance matrix
Create a matrix that represents the relationships between features.
$$
\Sigma = \frac{1}{n-1} X_{centered}^T X_{centered}
$$
This matrix contains the variance of each feature (on the diagonal) and the correlation between features (off-diagonal).
3) Find eigenvalues and eigenvectors
Decompose the covariance matrix to find the directions where data variance is greatest (principal components).
$$
\Sigma v = \lambda v
$$
Here,
λ (lambda) is the "magnitude of variance (amount of dispersion)"
v is the "direction (principal component axis)"
.
4) Select principal components in descending order of eigenvalues
Since a larger λ (variance) indicates an axis with more information, we select them starting from the top.
5) Project original data onto new axes
Finally, map the data onto the new principal component axes.
$$
Z = X_{centered} \cdot W
$$
W is the matrix containing the selected principal component axes.
5. Use Cases
Visualization: Plot high-dimensional data in 2D/3D scatter plots (to intuitively see cluster shapes)
Noise reduction: Keep only important axes and remove minor fluctuations
Preprocessing: Reduce input dimensions for machine learning models to make them lighter
Feature extraction: Extract "meaningful directions" from multidimensional data
Image compression: Reconstruct images using principal components to reduce file size
For example, in face recognition, a well-known method (Eigenfaces) uses PCA to create "facial feature axes" and calculates similarity using them.
6. Points to Note
-
Always perform standardization
If the scales of the features vary, features with larger values will dominate the principal components.
Therefore, it is necessary to set the mean to 0 and variance to 1 using StandardScaler or similar before PCA.
-
Determine the number of principal components by "information content"
Decide how much to reduce dimensions by looking at the "cumulative contribution ratio".
Generally, a good rule of thumb is to stop where you can explain 80-90% of the total variance.
-
Principal components often lack intuitive meaning for humans
Since new axes are created by linear combinations (weighted sums) of the original features, it may not be easy to interpret "what this principal component means."
7. Advantages
Because it reduces data dimensions, calculations become lighter
It automatically organizes correlations between features
Reduces noise and stabilizes accuracy
Extremely useful for visualization and exploratory data analysis
8. Disadvantages
Principal component axes do not directly reflect the meaning of the original features
If you neglect standardization, the results will be significantly distorted
Because it assumes linear transformation, non-linear structures cannot be captured (t-SNE and UMAP are better at this)
Human judgment is required to decide where to set the threshold for the cumulative contribution ratio
9. Implementation example in Python
Here, using the Iris dataset as an example, we will compress 4-dimensional features into 2 dimensions using PCA and visualize them.
import matplotlib.pyplot as plt
from sklearn.datasets import load_iris
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
# データ読み込み
data = load_iris()
X = data.data
y = data.target
labels = data.target_names
# 標準化
scaler = StandardScaler()
X_std = scaler.fit_transform(X)
# PCA（2次元に削減）
pca = PCA(n_components=2)
X_pca = pca.fit_transform(X_std)
# 結果を表示
print("寄与率（各主成分が説明する割合）:", pca.explained_variance_ratio_)
print("累積寄与率:", pca.explained_variance_ratio_.sum())
# 可視化
plt.figure(figsize=(6,5))
for i, label in enumerate(labels):
plt.scatter(X_pca[y == i, 0], X_pca[y == i, 1], label=label)
plt.xlabel('主成分1')
plt.ylabel('主成分2')
plt.title('PCAによる次元削減結果')
plt.legend()
plt.show()10. Summary
PCA is a fundamental method for "summarizing data with many features into fewer axes." It allows you to grasp the structure of data even without labels, and it works effectively as a preprocessing step before using any model.
However, you must not forget that "compressing information means some of it is being discarded."
PCA is useful, but it is not a panacea.
The key is to understand that and use it effectively across the three phases of exploration, preprocessing, and visualization.