Feature Selection Techniques in Machine Learning

Machine Learning Machine Learning 8 min read 1498 words Beginner ExcellentWiki Editorial Team

Feature selection identifies the most relevant input variables for a predictive model. Removing irrelevant or redundant features reduces overfitting, improves accuracy, speeds up training, and simplifies deployment.

Why Feature Selection Matters

A model trained on all available features often performs worse than one trained on a carefully selected subset. The reasons include:

Curse of dimensionality — as the number of features increases, data becomes sparse and distance-based metrics break down
Noise amplification — irrelevant features add noise that degrades generalization
Multicollinearity — correlated features inflate variance in linear models
Computational cost — training and inference scale with feature count
Interpretability — fewer features mean simpler, more explainable models
Storage and bandwidth — production pipelines must collect and store only necessary features

Feature selection is distinct from dimensionality reduction (PCA, t-SNE) which creates new combinations of features. Selection keeps the original features, preserving interpretability and requiring no transformation during inference.

Filter Methods

Filter methods evaluate features independently of any machine learning model. They are fast, scalable, and useful as a preprocessing step.

Variance Threshold

Remove features with zero or near-zero variance:

from sklearn.feature_selection import VarianceThreshold

selector = VarianceThreshold(threshold=0.01)  # remove low-variance features
X_selected = selector.fit_transform(X)

# Get surviving feature indices
selected_indices = selector.get_support(indices=True)

Low-variance features contain almost no information. A constant feature (variance = 0) can never help a model — it contributes nothing to predictions.

Correlation-Based Selection

Remove highly correlated features to reduce multicollinearity:

import pandas as pd
import numpy as np

def select_by_correlation(X, threshold=0.9):
    """Remove features with correlation above threshold."""
    corr_matrix = X.corr().abs()
    upper_tri = corr_matrix.where(
        np.triu(np.ones(corr_matrix.shape), k=1).astype(bool)
    )

    to_drop = [column for column in upper_tri.columns
               if any(upper_tri[column] > threshold)]
    return X.drop(columns=to_drop)

Statistical Tests

from sklearn.feature_selection import SelectKBest, f_classif, chi2, mutual_info_classif

# ANOVA F-test (classification, numerical features)
selector = SelectKBest(score_func=f_classif, k=10)
X_selected = selector.fit_transform(X, y)

# Chi-squared (classification, non-negative features)
selector = SelectKBest(score_func=chi2, k=10)
X_selected = selector.fit_transform(X, y)

# Mutual information (classification or regression)
selector = SelectKBest(score_func=mutual_info_classif, k=10)
X_selected = selector.fit_transform(X, y)

# For regression
from sklearn.feature_selection import f_regression, mutual_info_regression
selector = SelectKBest(score_func=mutual_info_regression, k=10)

Mutual information captures non-linear relationships that F-test and chi-squared miss. It is more computationally expensive but often produces better feature rankings for complex data.

Wrapper Methods

Wrapper methods evaluate feature subsets by training a model on each subset. They consider feature interactions but are computationally expensive.

Recursive Feature Elimination (RFE)

from sklearn.feature_selection import RFE, RFECV
from sklearn.svm import SVC

# RFE with fixed number of features
estimator = SVC(kernel='linear')
selector = RFE(estimator, n_features_to_select=10, step=1)
selector.fit(X, y)
print(f"Selected features: {selector.get_support()}")

# RFE with cross-validation (automatic feature count)
selector = RFECV(
    estimator,
    step=1,
    cv=5,
    scoring='accuracy',
    min_features_to_select=5
)
selector.fit(X, y)
print(f"Optimal number of features: {selector.n_features_}")

Forward Selection

from mlxtend.feature_selection import SequentialFeatureSelector

sfs = SequentialFeatureSelector(
    RandomForestClassifier(),
    k_features=10,
    forward=True,
    floating=False,
    scoring='accuracy',
    cv=5
)
sfs.fit(X, y)

Forward selection starts with zero features and adds the best one at each step. Backward elimination starts with all features and removes the worst one. Floating variants can add or remove features at each step for more thorough exploration.

Embedded Methods

Embedded methods perform feature selection as part of the model training process. They are more efficient than wrapper methods because selection is not a separate loop.

Lasso (L1 Regularization)

from sklearn.linear_model import Lasso, LogisticRegression

# Lasso regression (regression task)
lasso = Lasso(alpha=0.01)
lasso.fit(X_train, y_train)
selected_indices = np.where(np.abs(lasso.coef_) > 0)[0]

# Logistic regression with L1 (classification)
logreg = LogisticRegression(penalty='l1', solver='saga', C=1.0)
logreg.fit(X_train, y_train)
selected_indices = np.where(np.abs(logreg.coef_[0]) > 0)[0]

L1 regularization drives irrelevant feature coefficients to exactly zero, performing automatic feature selection. The alpha (or C) parameter controls the strength of regularization.

Tree-Based Importance

from sklearn.ensemble import RandomForestClassifier

rf = RandomForestClassifier(n_estimators=200, random_state=42)
rf.fit(X_train, y_train)

# Feature importances (mean decrease in impurity)
importances = rf.feature_importances_
indices = np.argsort(importances)[::-1]

# Select top k features
k = 10
top_features = indices[:k]
X_selected = X[:, top_features]

# Plot importances
import matplotlib.pyplot as plt
plt.figure(figsize=(10, 6))
plt.title("Feature Importances")
plt.bar(range(k), importances[indices[:k]])
plt.xticks(range(k), [feature_names[i] for i in top_features], rotation=45)
plt.tight_layout()

Permutation Importance

from sklearn.inspection import permutation_importance

rf = RandomForestClassifier(n_estimators=200)
rf.fit(X_train, y_train)

result = permutation_importance(
    rf, X_val, y_val,
    n_repeats=10,
    random_state=42,
    scoring='accuracy'
)

importance_df = pd.DataFrame({
    'feature': feature_names,
    'importance': result.importances_mean,
    'std': result.importances_std
---).sort_values('importance', ascending=False)

Permutation importance measures the drop in model performance when a single feature’s values are randomly shuffled. It works with any model and is more reliable than impurity-based importance when features have different scales or cardinalities.

Feature Selection Pipeline

Combine multiple selection methods for robust results:

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

pipeline = Pipeline([
    ('variance', VarianceThreshold(threshold=0.01)),
    ('select_k', SelectKBest(mutual_info_classif, k=20)),
    ('scaler', StandardScaler()),
    ('classifier', RandomForestClassifier(n_estimators=100)),
])

pipeline.fit(X_train, y_train)

Best Practices

Start with domain knowledge — remove obviously irrelevant features first
Use multiple selection methods and look for consensus
Validate feature subsets with cross-validation
Monitor feature stability across different data samples
Consider feature engineering before selection
For production, use embedded methods for automatic selection
Document why each feature was selected

Real-World Implementation Tips

Production Considerations

When moving from development to production, several factors become critical. Error handling should be comprehensive — every external call (database, API, file system) should have proper error checking, logging, and retry logic where appropriate. Performance monitoring through metrics and structured logging helps identify bottlenecks before they affect users.

Testing Strategy

A thorough testing approach combines multiple levels:

Unit tests verify individual functions and methods in isolation
Integration tests validate that components work together correctly
Edge case tests cover boundary conditions, empty inputs, and error states
Performance tests ensure the system meets latency and throughput requirements

Test data should be realistic but controlled. Mock external dependencies to make tests fast and deterministic. Aim for tests that are independent, repeatable, and fast enough to run on every commit.

Documentation

Good documentation is essential for maintainable code. Follow these principles:

Document the “why” not just the “what” — explain design decisions
Keep examples up to date with the code
Include usage examples for public APIs
Document configuration options and their defaults
Explain error conditions and recovery strategies

Security Best Practices

Security should be considered throughout development:

Validate all inputs at system boundaries
Use parameterized queries for database access
Store secrets in environment variables or secret managers
Keep dependencies updated to patch vulnerabilities
Apply the principle of least privilege

Performance Optimization

Optimize based on measured data, not assumptions:

Profile before optimizing — identify actual bottlenecks
Measure the impact of each change
Consider the trade-off between speed and readability
Cache expensive operations with appropriate invalidation
Use connection pooling for database and network resources

Monitoring and Observability

Production systems need visibility:

Structured logging with correlation IDs for request tracking
Metrics for latency, throughput, error rates, and resource usage
Health check endpoints for load balancers and orchestration
Distributed tracing for request flows across services
Alerts for anomaly detection based on baselines

These patterns apply across all programming languages and frameworks. The specific implementation varies, but the principles remain consistent.

FAQ

Q: How many features should I keep? A: There is no universal answer. Use RFECV to find the optimal count automatically. A common heuristic is to keep features until additional features no longer improve cross-validation performance.

Q: What is the difference between feature selection and dimensionality reduction? A: Feature selection keeps original features (preserving interpretability). Dimensionality reduction (PCA, t-SNE) creates new combinations of features (lost interpretability).

Q: Should I do feature selection before or after scaling? A: Scale before using distance-based or regularization methods (Lasso, SVM). Tree-based methods are unaffected by scaling.

Q: Can feature selection improve deep learning models? A: Yes, but deep learning models are more robust to irrelevant features. Feature selection is more impactful for smaller datasets and simpler models.

Q: How do I handle feature selection with high-cardinality categorical variables? A: Use target encoding or mean encoding, then apply standard feature selection. Alternatively, use tree-based models that handle categorical splits natively.

Q: Is feature selection necessary for Random Forest? A: Random Forest is robust to irrelevant features, but removing them can improve performance and reduce training time. Tree-based feature importance from a trained model can guide selection.

Feature Selection in Practice

Correlation Analysis

import pandas as pd
import numpy as np

df = pd.read_csv('data.csv')
corr_matrix = df.corr().abs()

# Find highly correlated feature pairs
upper = corr_matrix.where(np.triu(np.ones(corr_matrix.shape), k=1).astype(bool))
to_drop = [column for column in upper.columns if any(upper[column] > 0.95)]
print(f"Dropping {len(to_drop)} highly correlated features: {to_drop}")

df_reduced = df.drop(columns=to_drop)

Mutual Information

Mutual information captures non-linear relationships:

from sklearn.feature_selection import mutual_info_regression

mi_scores = mutual_info_regression(X, y, random_state=42)
mi_series = pd.Series(mi_scores, index=X.columns).sort_values(ascending=False)

# Plot top features
import matplotlib.pyplot as plt
mi_series.head(10).plot(kind='bar')
plt.title('Top 10 Features by Mutual Information')
plt.tight_layout()
plt.show()

Recursive Feature Elimination (RFE)

from sklearn.feature_selection import RFE
from sklearn.ensemble import RandomForestClassifier

estimator = RandomForestClassifier(n_estimators=100, random_state=42)
selector = RFE(estimator, n_features_to_select=20, step=5)
selector.fit(X, y)

selected_features = X.columns[selector.support_]
feature_rankings = pd.DataFrame({
    'feature': X.columns,
    'rank': selector.ranking_
---).sort_values('rank')

Regularization for Feature Selection

L1 regularization naturally performs feature selection:

from sklearn.linear_model import LogisticRegression

model = LogisticRegression(penalty='l1', solver='liblinear', C=0.1)
model.fit(X, y)

# Features with non-zero coefficients
selected = X.columns[model.coef_[0] != 0].tolist()
print(f"L1 selected {len(selected)} features out of {X.shape[1]}")