Object Detection Guide: YOLO, Faster R-CNN, Transformers, and...

Computer Vision Computer Vision 7 min read 1489 words Beginner ExcellentWiki Editorial Team

What Is Object Detection?

Object detection combines classification and localization: for each object in an image, the model predicts both its class label and its spatial extent via a bounding box. This is fundamentally harder than image classification because the number of objects varies per image, objects appear at different scales, and objects can overlap partially or fully.

The output of an object detector is a set of detections, each containing: the bounding box coordinates (typically center x, center y, width, height or top-left x, top-left y, bottom-right x, bottom-right y), the class label, and a confidence score. The confidence score represents the model’s estimate that the detection is correct. A post-processing step — Non-Maximum Suppression — filters redundant overlapping detections.

Object detection methods fall into two families. Two-stage detectors first generate region proposals and then classify each proposal, offering higher accuracy at the cost of slower inference. One-stage detectors predict bounding boxes and class probabilities directly from the image in a single pass, offering real-time inference with slightly lower accuracy. The gap between the two families has narrowed considerably with advances in one-stage architectures.

Two-Stage Detectors: Faster R-CNN

Faster R-CNN, introduced by Ren, He, Girshick, and Sun at Microsoft Research in 2015, established the two-stage paradigm that dominated object detection for years. The first stage uses a Region Proposal Network (RPN), a fully convolutional network that slides over the feature map and predicts whether each spatial location contains an object. The RPN outputs a set of candidate bounding boxes — region proposals — with associated objectness scores.

The second stage extracts features for each proposal using RoIPool (or RoIAlign, introduced in Mask R-CNN). RoIPool projects each proposal onto the feature map and pools the features into a fixed-size output. These pooled features are fed into classification and regression heads that predict the final class and refined bounding box coordinates.

The key insight of Faster R-CNN was sharing convolutional features between the RPN and the detection network. The same backbone feature map serves both stages, making Faster R-CNN nearly as fast as earlier single-stage models while achieving much higher accuracy. On the COCO benchmark, Faster R-CNN with a ResNet-101-FPN backbone achieves 42.0 mAP at 5 FPS. Feature Pyramid Networks (FPN), introduced by Lin et al. at Facebook AI Research, enhanced multi-scale detection by building a pyramid of features from a single backbone, significantly improving detection of small objects.

One-Stage Detectors: YOLO

YOLO (You Only Look Once), introduced by Joseph Redmon in 2015, revolutionized real-time detection by framing detection as a single regression problem. The input image is divided into an S x S grid, and each grid cell predicts B bounding boxes with confidence scores and C class probabilities. The final detection tensor has dimensions S x S x (B * 5 + C) — 5 values per bounding box (x, y, w, h, confidence) plus class probabilities.

YOLO was remarkably fast — 155 FPS for the original version on a Titan X GPU — setting a new standard for real-time detection. The trade-off was lower accuracy than Faster R-CNN, particularly for small objects and objects in unusual aspect ratios.

YOLO Architecture Evolution

YOLOv3 (2018) introduced a Darknet-53 backbone with residual connections, multi-scale predictions at three scales, and logistic regression for objectness scoring. YOLOv3 achieved 57.9 mAP at 30 FPS — competitive with two-stage detectors while maintaining real-time speed.

YOLOv4 (2020) by Bochkovskiy, Wang, and Liao was a master class in engineering optimization. The authors systematically evaluated and integrated dozens of training techniques: CutMix and Mosaic data augmentation, DropBlock regularization, CIoU loss for bounding box regression, Mish activation functions, and Self-Adversarial Training. YOLOv4 achieved 65.7 mAP on COCO at 65 FPS.

YOLOv8 by Ultralytics (2023) is the current standard, with an anchor-free design, a decoupled detection head (separate branches for classification and regression), and task-aligned assigners for positive sample matching. YOLOv8n (nano) achieves 37.3 mAP at 800+ FPS on TensorRT, while YOLOv8x achieves 53.9 mAP. YOLO-NAS by Deci AI uses neural architecture search for optimal accuracy-speed trade-offs. YOLO-World enables open-vocabulary detection, recognizing any object described by text.

Transformer-Based Detection

DETR (Detection Transformer), introduced by Carion et al. at Facebook AI in 2020, fundamentally reimagined object detection. DETR uses a transformer encoder-decoder architecture: the encoder processes image features from a CNN backbone, and the decoder uses learned object queries to attend to image features and directly predict a fixed set of predictions. The Hungarian algorithm matches predictions to ground truth boxes during training.

DETR eliminates the need for anchor boxes, NMS, and many hand-designed components of traditional detectors. However, DETR converges slowly (500 epochs) and struggles with small objects due to the quadratic complexity of self-attention. Deformable DETR (2021) addresses these issues by using deformable attention that attends only to a sparse set of key sampling points around a reference point, converging 10x faster and improving small object detection.

DINO (DETR with Improved Denoising Anchor Boxes) and DAB-DETR further improved convergence and accuracy, with DINO achieving 63.2 mAP on COCO — one of the highest reported scores.

Evaluation Metrics

Mean Average Precision (mAP) is the standard evaluation metric for object detection. The computation involves: sorting all detections by confidence score, computing precision and recall at each threshold, integrating the precision-recall curve to compute Average Precision for each class, and averaging across all classes.

Intersection over Union (IoU) measures the overlap between predicted and ground truth boxes. IoU = area of overlap / area of union. A detection is considered a true positive if its IoU with a ground truth box exceeds a threshold (typically 0.5). The COCO evaluation protocol uses mAP averaged over IoU thresholds from 0.50 to 0.95 at 0.05 intervals, providing a stringent measure that penalizes inaccurate localization.

Non-Maximum Suppression

Object detectors produce multiple overlapping detections for the same object. Non-Maximum Suppression (NMS) selects the best detection for each object. The standard greedy NMS algorithm sorts detections by confidence, selects the highest-confidence detection, removes all remaining detections with IoU above a threshold (typically 0.5-0.7), and repeats until all detections are processed.

Soft-NMS decays the confidence scores of overlapping detections rather than removing them entirely, preserving detections of genuinely overlapping objects. Weighted Box Fusion (WBF) combines predictions from multiple models, computing weighted average coordinates of all detections that likely refer to the same object.

Training Best Practices

Training accurate object detectors requires careful attention to data, augmentation, and loss functions. Mosaic augmentation, introduced in YOLOv4, stitches four training images together into a single composite image. This forces the model to learn detection at different scales and in varied contexts within each batch. MixUp blends two images and their bounding boxes linearly, providing a regularizing effect.

Loss functions for object detection combine classification and localization components. Focal loss addresses the extreme class imbalance between foreground objects and background regions by down-weighting easy negative examples. CIoU (Complete IoU) loss considers the overlap area, center point distance, and aspect ratio between predicted and ground truth boxes, providing better gradient signals for precise localization. Multi-scale training randomly varies the input resolution between batches, improving scale invariance.

Anchor box design significantly affects detection performance. K-means clustering on the training dataset’s bounding boxes discovers the optimal anchor dimensions for the target object sizes. YOLOv8 and later models adopted anchor-free detection, simplifying the architecture by predicting object centers directly without predefined anchors.

Frequently Asked Questions

What is the difference between object detection and image classification? Classification assigns one label to an entire image. Detection identifies multiple objects, their classes, and their locations via bounding boxes. Classification answers “what is in this image?” while detection answers “what is where?”

Which object detector should I use? YOLOv8 for real-time applications (30+ FPS needed). Faster R-CNN with FPN for maximum accuracy when speed is less critical. DETR/DINO for research and when you want a clean, anchor-free architecture. YOLO-NAS for the best accuracy-speed trade-off through neural architecture search.

What is mAP and how is it calculated? Mean Average Precision averages the precision-recall area under the curve across all classes. COCO mAP averages across IoU thresholds 0.50 to 0.95 with step 0.05, providing a comprehensive measure of detection and localization quality. mAP@50 uses only IoU 0.5 (more lenient), while mAP@75 uses IoU 0.75 (more stringent).

How do I detect small objects? Use Feature Pyramid Networks (FPN) that combine high-resolution shallow features with semantic deep features. Increase input resolution (if GPU memory permits). Use image tiling — process overlapping image crops at higher resolution. Use specialized small object augmentation by placing small objects on training images.

What is the role of Non-Maximum Suppression? NMS converts the raw output of an object detector (many overlapping proposals) into a clean set of non-overlapping detections. Each detected object should produce exactly one bounding box. NMS filters redundant detections by selecting the highest confidence box and removing highly overlapping competitors.