Unsupervised Learning: Clustering and Dimensionality Reduction

Machine Learning Machine Learning 8 min read 1697 words Beginner ExcellentWiki Editorial Team

Unsupervised learning finds hidden patterns in data without labeled outputs. Unlike supervised learning, which requires a ground truth for training, unsupervised algorithms discover structure — clusters, groupings, low-dimensional manifolds — purely from the input data itself. This makes unsupervised learning essential for exploratory data analysis, customer segmentation, anomaly detection, and as a preprocessing step for supervised models.

The fundamental challenge of unsupervised learning was articulated by Geoffrey Hinton: “You do not know what the answers are, and you do not even know what the questions are.” Without labels, the quality of a clustering or dimensionality reduction depends entirely on whether the discovered structure is useful for the downstream task. This makes evaluation inherently subjective — there is no single correct answer, only more or less useful representations.

Clustering

Clustering groups similar data points together such that points in the same cluster are more similar to each other than to points in other clusters. The similarity is typically defined by a distance metric — Euclidean, Manhattan, cosine — in the feature space.

K-Means Clustering

K-Means partitions data into k clusters by minimizing the within-cluster sum of squares (inertia). The algorithm iterates between assigning points to the nearest centroid and recalculating centroids as the mean of assigned points:

from sklearn.cluster import KMeans
from sklearn.preprocessing import StandardScaler
import matplotlib.pyplot as plt

# Scale features first — K-Means is distance-based
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

kmeans = KMeans(n_clusters=5, random_state=42, n_init=10)
kmeans.fit(X_scaled)

labels = kmeans.labels_
centroids = kmeans.cluster_centers_
inertia = kmeans.inertia_  # Sum of squared distances to nearest centroid

Choosing k is the most critical decision. The elbow method plots inertia against k and looks for the point where additional clusters provide diminishing returns:

inertias = []
for k in range(1, 11):
    kmeans = KMeans(n_clusters=k, random_state=42, n_init=10)
    kmeans.fit(X_scaled)
    inertias.append(kmeans.inertia_)

plt.plot(range(1, 11), inertias, 'bo-')
plt.xlabel('Number of clusters (k)')
plt.ylabel('Within-cluster sum of squares')
plt.title('Elbow Method for Optimal k')

The silhouette score provides a more rigorous approach, measuring how similar each point is to its own cluster versus neighboring clusters. Scores range from -1 (wrong clustering) to +1 (dense, well-separated clusters):

from sklearn.metrics import silhouette_score

silhouette_scores = []
for k in range(2, 11):
    kmeans = KMeans(n_clusters=k, random_state=42, n_init=10)
    labels = kmeans.fit_predict(X_scaled)
    score = silhouette_score(X_scaled, labels)
    silhouette_scores.append(score)

best_k = range(2, 11)[silhouette_scores.index(max(silhouette_scores))]
print(f"Best k by silhouette score: {best_k}")

K-Means assumes spherical clusters of similar size. It struggles with non-spherical shapes, varying densities, and outliers. The algorithm is sensitive to centroid initialization — k-means++ initialization (the default in scikit-learn) mitigates this by choosing initial centroids that are far apart.

Hierarchical Clustering

Hierarchical clustering builds a tree of clusters (dendrogram) using agglomerative (bottom-up) or divisive (top-down) approaches:

from scipy.cluster.hierarchy import dendrogram, linkage
from sklearn.cluster import AgglomerativeClustering

# Compute linkage matrix for dendrogram
Z = linkage(X_scaled, method='ward')  # Ward minimizes variance within clusters

# Plot dendrogram
plt.figure(figsize=(10, 5))
dendrogram(Z, truncate_mode='level', p=3)
plt.title('Hierarchical Clustering Dendrogram')
plt.xlabel('Sample index')
plt.ylabel('Distance')

Hierarchical clustering does not require specifying the number of clusters in advance — you can cut the dendrogram at any level. It also produces a full hierarchy, which is useful for understanding data at multiple granularities. The computational cost O(n^3) limits its application to datasets under about 10,000 points.

DBSCAN

DBSCAN (Density-Based Spatial Clustering of Applications with Noise) groups points that are closely packed together, marking points in low-density regions as outliers. It does not require specifying the number of clusters and can find arbitrarily shaped clusters:

from sklearn.cluster import DBSCAN

dbscan = DBSCAN(
    eps=0.5,        # Maximum distance between two points for neighborhood
    min_samples=5,  # Minimum points to form a dense region
    metric='euclidean'
)
labels = dbscan.fit_predict(X_scaled)

n_clusters = len(set(labels)) - (1 if -1 in labels else 0)
n_noise = list(labels).count(-1)
print(f"Number of clusters: {n_clusters}")
print(f"Number of noise points: {n_noise}")

DBSCAN’s main advantage is its ability to handle non-spherical clusters and automatically detect outliers. The eps parameter is critical and dataset-dependent — use a k-distance plot to select it. DBSCAN is not suitable for data with widely varying densities.

Gaussian Mixture Models

GMMs model data as a mixture of several Gaussian distributions. Unlike K-Means (hard assignment), GMMs provide soft assignments — each point has a probability of belonging to each cluster:

from sklearn.mixture import GaussianMixture

gmm = GaussianMixture(
    n_components=5,
    covariance_type='full',  # 'tied', 'diag', or 'spherical'
    random_state=42
)
gmm.fit(X_scaled)

# Soft clustering: probability of belonging to each cluster
probabilities = gmm.predict_proba(X_scaled)
# Hard clustering: assign to most likely cluster
labels = gmm.predict(X_scaled)
# Model selection via AIC or BIC
print(f"AIC: {gmm.aic(X_scaled):.0f}, BIC: {gmm.bic(X_scaled):.0f}")

GMMs can model elliptical clusters of different sizes and orientations, making them more flexible than K-Means. The covariance_type parameter controls the flexibility of cluster shapes. Use BIC to select the number of components.

Dimensionality Reduction

High-dimensional data suffers from the curse of dimensionality: distances become less meaningful, and models require exponentially more data to maintain statistical significance. Dimensionality reduction compresses data into fewer dimensions while preserving as much relevant structure as possible.

Principal Component Analysis

PCA finds orthogonal axes (principal components) that capture the maximum variance in the data. It is a linear, deterministic technique that produces uncorrelated features ordered by explained variance:

from sklearn.decomposition import PCA
import numpy as np

pca = PCA()
X_pca = pca.fit_transform(X_scaled)

# Explained variance ratio
cumulative_variance = np.cumsum(pca.explained_variance_ratio_)
n_components_95 = np.argmax(cumulative_variance >= 0.95) + 1
print(f"Components to explain 95% variance: {n_components_95}")
print(f"Original dimensions: {X_scaled.shape[1]}")

# Apply with chosen number of components
pca = PCA(n_components=n_components_95)
X_reduced = pca.fit_transform(X_scaled)

The first principal component captures the direction of maximum variance. PCA assumes linear relationships and that the directions of maximum variance are the most informative. When these assumptions hold, PCA is a powerful preprocessing step that can dramatically reduce data dimensionality while retaining most of the signal.

t-SNE

t-SNE is a non-linear technique for visualizing high-dimensional data in 2D or 3D. It preserves local structure — nearby points in high-dimensional space remain nearby in the embedding — making it ideal for exploring clusters:

from sklearn.manifold import TSNE

tsne = TSNE(
    n_components=2,
    perplexity=30,     # Balances local vs. global structure
    learning_rate=200, # Typical range: 10-1000
    random_state=42,
    n_iter=1000
)
X_tsne = tsne.fit_transform(X_scaled)

plt.scatter(X_tsne[:, 0], X_tsne[:, 1], c=labels, cmap='viridis', alpha=0.6)
plt.title('t-SNE Visualization')
plt.colorbar()

t-SNE excels at revealing clusters that PCA misses, but it is stochastic (different runs produce different results), computationally expensive (O(n^2)), and distances in the embedding are not meaningful. Use it for exploration, not as a preprocessing step for downstream models.

UMAP

UMAP (Uniform Manifold Approximation and Projection) is faster than t-SNE and better preserves global structure — faraway points in the original space remain far apart in the embedding:

import umap

reducer = umap.UMAP(
    n_neighbors=15,
    min_dist=0.1,
    n_components=2,
    random_state=42
)
X_umap = reducer.fit_transform(X_scaled)

UMAP is becoming the default choice for visualization and can also be used for general-purpose dimensionality reduction. Its theoretical foundations in manifold learning and topological data analysis provide strong mathematical guarantees.

Use Cases

Customer segmentation: group customers by purchasing behavior using K-Means. Target marketing campaigns to each segment.
Image compression: reduce color space by applying K-Means to pixel values and replacing each pixel with its nearest centroid.
Feature extraction: use PCA components as features for supervised models, reducing noise and multicollinearity.
Anomaly detection: points that do not belong to any DBSCAN cluster, or that have low probability under a GMM, are potential anomalies.
Data visualization: project high-dimensional data to 2D with t-SNE or UMAP for stakeholder presentations.

FAQ

How do I evaluate clustering quality without labels? Use internal validation metrics: silhouette score, Davies-Bouldin index (lower is better), and Calinski-Harabasz index (higher is better). These measure cluster compactness and separation. However, no metric replaces domain expert judgment.

When should I use PCA vs. t-SNE vs. UMAP? PCA for preprocessing (dimensionality reduction before modeling), linear feature extraction, and when interpretability of components matters. t-SNE for visualization of well-separated clusters in small-to-medium datasets. UMAP for both visualization and general dimensionality reduction in larger datasets.

Do I need to scale data for clustering? Yes, unless features are on the same scale. Distance-based algorithms (K-Means, DBSCAN, hierarchical) are sensitive to feature magnitudes. Use StandardScaler before clustering.

How do I handle categorical features in clustering? Option 1: one-hot encode categories and use K-Means (works well in practice despite the Euclidean distance on binary features). Option 2: use K-Modes clustering (designed for categorical data). Option 3: use Gower distance with hierarchical clustering.

What if my clusters have very different sizes? K-Means tends to split large clusters and merge small ones. GMMs handle varying cluster sizes better. DBSCAN naturally handles different densities (though not extreme differences). Consider spectral clustering for very irregular cluster sizes.

Internal Links

For supervised alternatives when you have labels, see Supervised Learning: Regression and Classification.
To use PCA as a feature selection step before supervised learning, read Feature Selection: Choosing the Right Inputs.
For a beginner’s overview before diving into these techniques, start with Machine Learning: A Beginner’s Overview and Roadmap.