Video Analysis: Optical Flow, Object Tracking, Action Recognition,...

Computer Vision Computer Vision 8 min read 1558 words Beginner ExcellentWiki Editorial Team

The Temporal Dimension of Computer Vision

Video analysis extends computer vision from static images to the temporal domain. This additional dimension enables capabilities impossible with single-image analysis: understanding motion, tracking objects across frames, recognizing actions that unfold over time, and detecting events in surveillance footage. The temporal dimension also introduces new challenges: processing large volumes of data (30 frames per second), maintaining consistent identity across frames, and modeling temporal dependencies that span seconds or minutes.

Video understanding systems process streams rather than single images. The standard approach maintains a temporal buffer of recent frames and processes them jointly with architectures that incorporate temporal operations. The choice of temporal window length — how many frames to consider simultaneously — involves a fundamental trade-off: longer windows capture more temporal context but require more computation and memory.

Optical Flow

Optical flow estimates the motion of every pixel between consecutive frames. Each pixel is assigned a 2D velocity vector (u, v) representing its displacement in the horizontal and vertical directions. Optical flow is the foundation for video stabilization, motion segmentation, object tracking, and action recognition.

Traditional methods compute flow by assuming brightness constancy (a pixel maintains its intensity as it moves) and spatial smoothness (nearby pixels move similarly). The Lucas-Kanade method, developed in 1981, assumes constant flow in a local neighborhood and solves the optical flow equations using least-squares estimation. Lucas-Kanade is computationally efficient (used in OpenCV for sparse feature tracking) but fails for large motions and at motion boundaries.

The Farneback method computes dense optical flow by approximating each neighborhood with a polynomial and tracking the polynomial displacement. It provides better results than Lucas-Kanade for dense flow but is more computationally expensive.

Deep Learning Optical Flow: RAFT

RAFT (Recurrent All-Pairs Field Transforms), introduced by Teed and Deng at Princeton in 2020, represents the current state of the art in optical flow. RAFT computes a 4D correlation volume between all pairs of pixels in the two input frames. A recurrent GRU-based update operator iteratively refines the flow estimate, starting from zero and progressively improving.

RAFT achieves 1.76 endpoint error on the Sintel benchmark and 4.20 on KITTI, significantly outperforming prior methods. The iterative refinement enables high accuracy on large displacements and motion boundaries. RAFT runs at approximately 20 FPS on a single GPU, making it suitable for offline processing and high-end real-time applications.

Object Tracking

Object tracking maintains the identity of objects across video frames. Unlike detection, which independently finds objects in each frame, tracking associates detections across time to produce consistent identifiers.

Single Object Tracking (SOT)

Single Object Tracking follows one object initialized by a bounding box in the first frame. The tracker must handle appearance changes due to deformation, rotation, illumination variation, and partial occlusion. SiamRPN (Siamese Region Proposal Network) uses a Siamese backbone to compare the target template with candidate regions in the current frame. SiamMask extends this with segmentation mask prediction.

Discriminative correlation filters (DCF) formulate tracking as learning a filter that maximizes the response at the target location. Modern trackers like DiMP and PrDiMP combine correlation filters with deep learning for state-of-the-art accuracy. The OTB (Object Tracking Benchmark) and LaSOT datasets evaluate SOT performance.

Multi-Object Tracking (MOT)

Multi-Object Tracking tracks all objects of interest simultaneously, handling object appearance, disappearance, and re-entry. The dominant paradigm is tracking-by-detection: run an object detector on each frame, then associate detections across frames to form tracks.

DeepSORT, introduced by Wojke et al. in 2017, extended the SORT (Simple Online and Realtime Tracking) algorithm with appearance features. DeepSORT computes a Kalman filter prediction for each existing track, matches new detections to track predictions using the Hungarian algorithm, and considers both motion (Mahalanobis distance) and appearance (cosine distance between appearance feature vectors). DeepSORT runs at 40 FPS and achieves 61.4 MOTA (Multi-Object Tracking Accuracy) on the MOT17 benchmark.

ByteTrack, introduced by Zhang et al. in 2022, improved on DeepSORT by retaining low-confidence detections and matching them in a second association round. ByteTrack demonstrated that a simple tracker with careful handling of all detections (both high and low confidence) could achieve state-of-the-art results: 80.3 MOTA on MOT17. The key insight is that low-confidence detections often correspond to occluded or distant objects, and matching them to unmatched tracks improves track continuity.

Action Recognition

Action recognition classifies activities being performed in video. The standard benchmarks include Kinetics-400 (400 action classes, 300,000 video clips), UCF-101 (101 actions, 13,000 clips), and HMDB-51 (51 actions, 7,000 clips).

3D Convolutional Networks

3D CNNs extend 2D convolutions to the temporal dimension. A 3D convolutional filter has shape (depth, height, width), processing a stack of frames simultaneously. C3D, introduced by Tran et al. in 2015, showed that 3D ConvNets could learn spatiotemporal features directly from video. I3D (Inflated 3D ConvNet), by Carreira and Zisserman at DeepMind, inflated the 2D Inception architecture to 3D and initialized from ImageNet weights, achieving 98.0% on UCF-101.

The Two-Stream architecture processes RGB frames and optical flow streams in parallel, combining their predictions. The RGB stream recognizes appearance-based actions (playing guitar, holding a cup), while the flow stream recognizes motion-based actions (running, jumping). Fusion occurs at the prediction level (late fusion) or through intermediate feature concatenation.

Video Transformers

Video transformers extend the Vision Transformer architecture to video by processing spatiotemporal patches. TimeSformer, introduced by Facebook AI in 2021, applies separate spatial and temporal attention, achieving 80.7% top-1 accuracy on Kinetics-400. VideoMAE, by He et al., masks large portions of video patches and learns to reconstruct them, achieving state-of-the-art results with efficient self-supervised pre-training. VideoMAE achieves 87.4% top-1 accuracy on Kinetics-400 after fine-tuning.

Video Object Segmentation

Video Object Segmentation (VOS) segments a specific object throughout a video sequence. Semi-supervised VOS provides the ground-truth mask for the first frame and propagates it through the video. Unsupervised VOS discovers and segments the primary object automatically.

STM (Space-Time Memory Networks), by Oh et al. in 2019, reads from a memory of past frames with their predicted masks using a space-time memory read operation. The memory stores features from the current frame and selected past frames, enabling the model to retrieve relevant context for segmentation. XMem extends this with multi-scale feature memory for long videos.

Video Summarization and Highlight Detection

Video summarization automatically extracts the most important segments from long videos. The challenge is defining importance — which frames or clips convey the essential content of the video. Supervised approaches train on human-annotated summaries using recurrent neural networks or transformers that predict frame-level importance scores. Unsupervised approaches select diverse and representative frames using clustering or determinantal point processes.

Highlight detection identifies the most exciting or interesting moments in sports, gaming, and surveillance footage. The task requires understanding both visual content and temporal context — a goal in soccer is only meaningful relative to the preceding play. Temporal action localization detects and classifies action segments within untrimmed videos, providing the boundaries for highlight clips. The THUMOS challenge and ActivityNet dataset provide standard benchmarks with frame-level action annotations.

Real-Time Video Processing Pipelines

Production video processing systems must handle high-throughput streams with low latency. A typical pipeline: video capture at 30 FPS, decoding (hardware-accelerated with NVIDIA NVDEC or Intel QSV), preprocessing (resize, normalize, color conversion), batch formation (accumulating frames for efficient inference), model inference, post-processing (NMS, thresholding), and output (overlay rendering, metadata serialization, or storage).

NVIDIA DeepStream SDK provides a complete framework for video analytics pipelines on Jetson and GPU platforms. It handles hardware-accelerated decoding, inferencing with TensorRT, and multi-stream management. DeepStream supports up to 24 simultaneous 1080p video streams on a single Jetson Orin. GStreamer integration enables flexible pipeline construction with CPU-offloaded processing elements. Key optimizations include zero-copy memory transfers between decoding, inference, and display stages.

Frequently Asked Questions

What is the difference between optical flow and motion estimation? Optical flow computes per-pixel motion vectors for every pixel in the image (dense flow). Motion estimation, in the context of video compression (MPEG, H.264), computes motion vectors for image blocks rather than individual pixels, prioritizing compression efficiency over accuracy.

Which tracker should I use? ByteTrack for multi-object tracking with the best accuracy-speed trade-off. DeepSORT when appearance matching is critical (re-identifying objects after occlusion). SiamRPN or DiMP for single object tracking with high accuracy. Bot-SORT for sports tracking with non-linear motion models.

How do I handle tracking when objects leave and re-enter the frame? Maintain track confidence scores. Decrease confidence each frame the track is unmatched. Re-identify objects on re-entry using appearance features (DeepSORT) or by matching detection and track features. Remove tracks after a configurable number of missed frames (typically 30-60 frames or 1-2 seconds).

What is the difference between online and offline tracking? Online tracking processes each frame immediately, maintaining only past information. Online tracking is essential for real-time applications. Offline tracking uses past and future frames to resolve ambiguities — a track can be extended backwards after the object is detected again. Offline tracking is more accurate but requires access to the full video and has higher latency.

What are the main challenges in action recognition? Temporal scale variation (actions unfold over milliseconds to minutes), viewpoint variation (the same action looks different from different angles), background clutter, inter-class similarity (running vs jogging), and computational cost of processing video (30 FPS is 30x more data than images).