Model Evaluation Metrics: Accuracy, Precision, Recall, F1

Machine Learning Machine Learning 7 min read 1470 words Beginner ExcellentWiki Editorial Team

Model evaluation is the process of assessing how well a machine learning model performs on unseen data. Choosing the wrong metric can lead to deploying a model that appears accurate while completely failing at the actual task. A spam classifier that always predicts “not spam” achieves 99% accuracy on typical email data but is entirely useless — it catches zero spam messages. This is the core trap of relying on accuracy alone.

The discipline of model evaluation draws on statistical decision theory and information retrieval. David M. W. Powers’ comprehensive 2011 survey “Evaluation: From Precision, Recall and F-Measure to ROC, Informedness, Markedness and Correlation” catalogues over two dozen metrics and their appropriate use cases, emphasizing that no single metric captures all aspects of model quality. This guide provides a practical framework for selecting metrics that align with your problem’s business context.

Accuracy

Accuracy is the proportion of correct predictions out of total predictions:

Accuracy = (TP + TN) / (TP + TN + FP + FN)

Accuracy works well only when classes are balanced — roughly equal numbers of positive and negative examples. When classes are imbalanced, a model that predicts the majority class for every input achieves high accuracy while providing no value. For example, a fraud detection dataset with 0.1% fraudulent transactions would yield 99.9% accuracy from a model that never flags fraud.

Accuracy also assumes equal misclassification costs, which rarely holds in practice. The cost of a false negative (missing a cancer diagnosis) is vastly different from a false positive (unnecessary follow-up tests). When costs are unequal, accuracy is the wrong objective.

Confusion Matrix

Before computing any derived metric, build a confusion matrix that shows the full distribution of predictions versus actuals:

from sklearn.metrics import confusion_matrix, ConfusionMatrixDisplay
import matplotlib.pyplot as plt

cm = confusion_matrix(y_test, predictions)
disp = ConfusionMatrixDisplay(confusion_matrix=cm)
disp.plot()
plt.title("Confusion Matrix")
plt.show()

The confusion matrix provides the raw counts — true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN) — from which all classification metrics are derived. Always examine the confusion matrix first; it reveals patterns that aggregate metrics hide, such as a model performing well on one class but poorly on another.

Precision

Precision answers: of all positive predictions, how many were correct?

Precision = TP / (TP + FP)

High precision means few false positives. Optimize for precision when the cost of a false positive is high. In spam detection, marking a legitimate email as spam (false positive) is worse than letting a spam message through (false negative). In medical testing, precision matters when treatments are invasive — you want high confidence that a positive test truly indicates disease before proceeding.

Søren Brunak and colleagues note in their 2019 Nature Reviews Genetics article that precision is often reported alongside recall because optimizing one typically degrades the other — the precision-recall tradeoff.

Recall

Recall (also called sensitivity or true positive rate) answers: of all actual positives, how many did we catch?

Recall = TP / (TP + FN)

High recall means few false negatives. Optimize for recall when missing a positive is very costly. In cancer screening, a missed diagnosis (false negative) can be life-threatening. In fraud detection, failing to flag a fraudulent transaction (false negative) results in direct financial loss.

The tradeoff is that maximizing recall often increases false positives because the model casts a wider net to catch all positive cases.

F1 Score

The F1 score is the harmonic mean of precision and recall:

F1 = 2 × (Precision × Recall) / (Precision + Recall)

The harmonic mean penalizes extreme imbalances — a model with perfect precision but zero recall gets an F1 of 0, unlike the arithmetic mean which would give 0.5. F1 provides a single number that balances both concerns, making it especially useful for comparing models on imbalanced datasets where accuracy is misleading.

F-beta score generalizes F1 by allowing differential weighting of precision and recall:

F_beta = (1 + beta^2) × (Precision × Recall) / (beta^2 × Precision + Recall)

When beta > 1, recall is weighted more heavily. When beta < 1, precision matters more. In practice, F1 (beta = 1) is the standard choice.

from sklearn.metrics import precision_score, recall_score, f1_score, fbeta_score

print(f"Precision: {precision_score(y_test, predictions):.3f}")
print(f"Recall:    {recall_score(y_test, predictions):.3f}")
print(f"F1 Score:  {f1_score(y_test, predictions):.3f}")
print(f"F2 Score:  {fbeta_score(y_test, predictions, beta=2):.3f}")

ROC Curve and AUC

The Receiver Operating Characteristic curve plots the true positive rate (recall) against the false positive rate at every possible threshold setting. The Area Under the Curve (AUC) summarizes model performance across all thresholds:

AUC = 1.0: perfect classifier
AUC = 0.9-1.0: excellent discrimination
AUC = 0.8-0.9: good discrimination
AUC = 0.7-0.8: fair discrimination
AUC = 0.5: random guessing

from sklearn.metrics import roc_auc_score, roc_curve

y_probs = model.predict_proba(X_test)[:, 1]
auc = roc_auc_score(y_test, y_probs)
fpr, tpr, thresholds = roc_curve(y_test, y_probs)

plt.plot(fpr, tpr, label=f'AUC = {auc:.3f}')
plt.plot([0, 1], [0, 1], 'k--')
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.legend()

AUC is threshold-independent, making it useful for comparing models when you have not yet chosen a decision threshold. However, AUC can be misleading for highly imbalanced datasets because the false positive rate becomes very small. In those cases, Precision-Recall AUC (also called Average Precision) provides a more informative picture.

Multi-Class Classification Metrics

For problems with three or more classes, accuracy and F1 extend through averaging strategies:

Micro-averaging: aggregates contributions from all classes to compute the average metric. Treats each instance equally regardless of class. Useful when class imbalance exists but you care about per-instance performance.
Macro-averaging: computes the metric independently for each class and averages them. Treats each class equally regardless of size. Useful when rare classes are as important as common ones.
Weighted averaging: computes the metric per class and averages weighted by the number of true instances. Balances the class-agnostic view of micro with the per-class view of macro.

from sklearn.metrics import classification_report

report = classification_report(y_test, predictions, target_names=['Class 0', 'Class 1', 'Class 2'])
print(report)
# Shows precision, recall, f1-score, and support for each class
# Plus micro avg, macro avg, and weighted avg

Regression Metrics

Mean Absolute Error

MAE measures the average absolute difference between predictions and actual values. It is interpretable in the same units as the target and is robust to outliers:

MAE = (1/n) × sum(|y_i - y_hat_i|)

Mean Squared Error

MSE penalizes larger errors quadratically, making it sensitive to outliers but useful when large errors are disproportionately costly:

MSE = (1/n) × sum((y_i - y_hat_i)^2)

RMSE (Root Mean Squared Error) brings MSE back to the original unit scale for interpretability.

R-squared

R-squared represents the proportion of variance in the target explained by the model. An R-squared of 0.85 means the model explains 85% of the variance:

from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score

mae = mean_absolute_error(y_test, predictions)
mse = mean_squared_error(y_test, predictions)
rmse = mean_squared_error(y_test, predictions, squared=False)
r2 = r2_score(y_test, predictions)

R-squared increases with more features even if those features are noise, which is why adjusted R-squared penalizes model complexity.

Choosing the Right Metric

Balanced classes with equal costs: accuracy is sufficient
Imbalanced classes: use precision, recall, F1, and Precision-Recall AUC
Cost-sensitive problems: weight metrics by misclassification cost (e.g., the cost of a false positive is 10× the cost of a false negative)
Ranking problems: use AUC or Average Precision at K
Regression with normal errors: RMSE
Regression with outliers: MAE
Model interpretability: R-squared

FAQ

Why is accuracy not enough? Accuracy ignores class imbalance and misclassification cost differences. A 99% accurate model on 0.1% fraud data is worthless if it misses all fraud. Always examine precision, recall, and the confusion matrix alongside accuracy.

What is a good F1 score? It depends on the problem difficulty and class balance. In easy problems with clean data, F1 above 0.9 is expected. In hard problems like rare disease diagnosis, F1 of 0.5-0.7 may be state of the art. Compare against published benchmarks and simple baselines.

When should I use macro vs. weighted F1? Macro F1 computes F1 per class and averages them equally, treating all classes as equally important. Weighted F1 weights each class by its support (number of true instances). Use macro when rare classes matter as much as common ones. Use weighted when you want the score to reflect overall performance skewed toward frequent classes.

How do I choose a classification threshold? Plot precision-recall curves or cost curves across thresholds. Choose the threshold that maximizes the business metric (profit, cost savings, etc.). Do not assume the default 0.5 threshold is optimal.

What metrics do I need for multi-label classification? Use Hamming loss, subset accuracy (exact match), and per-class precision/recall. Micro-averaged F1 aggregates across all predictions. Macro-averaged F1 treats each label equally regardless of frequency.