The Problem Nobody Warns You About
Speed estimation from video sounds straightforward until you try it. You frame a traffic camera feed, drop in a YOLOv8 model, and expect bounding boxes to translate into kilometers per hour. They don’t. A bounding box tells you where a car is in pixel space. Speed is a physical quantity. Bridging that gap is the entire project.
I built this pipeline during my under graduation, working with traffic footage from fixed cameras mounted on overhead gantries. The goal was simple: detect vehicles, track them frame to frame, and compute speed. The implementation took roughly three weeks of iterative work, and most of that time was spent not on the ML model but on the geometry.
Detection with YOLOv8
I used YOLOv8-nano for detection. It runs at ~120 FPS on a single T4 GPU, which matters when you’re processing 30 FPS video in real time. The model was trained on the UA-DETRAC dataset, which gives you bounding boxes and class labels (car, bus, truck, etc.) for traffic scenes.
The first thing I noticed was that YOLOv8’s confidence threshold was too aggressive for dense traffic. At 0.5, it dropped vehicles in overlapping bounding boxes — two cars side by side where the model merged them into one detection. Dropping to 0.35 recovered most of those but introduced false positives from shadow regions on the road. I settled on 0.4 with a non-maximum suppression IoU threshold of 0.45, which balanced precision and recall for our specific camera angle.
Tracking with DeepSort
Detection alone gives you isolated frames. To estimate speed, you need to know that the car in frame 147 is the same car in frame 148. That’s where DeepSort comes in.
DeepSort combines YOLOv8 detections with a Kalman filter for state prediction and the Hungarian algorithm for data association. I used the deep_sort_realtime Python package, which wraps the whole pipeline. The key parameter I tuned was the max_age — how many frames to keep a track alive without a detection. Set it too low (3-5), and tracks fragment when a vehicle is briefly occluded by another. Set it too high (15+), and you get ghost tracks that linger long after a car has left the frame. I landed on max_age=10 with nn_budget=32, which gave stable tracks across typical occlusion patterns in our footage.
The Geometry Problem
Here’s where it got hard. YOLOv8 gives you pixel coordinates. Speed requires real-world distance over real-world time. The camera is fixed, but the relationship between pixels and meters varies across the frame — objects closer to the camera cover more pixels per meter than objects farther away.
I solved this with a homography-based approach. I calibrated the camera by identifying four known points on the road (lane markings at measured intervals) and computed a perspective transform using OpenCV’s findHomography. This mapped pixel coordinates to a top-down bird’s-eye view where each pixel corresponded to a fixed real-world distance.
The math for speed was then straightforward:
distance_meters = pixel_displacement * pixels_per_meter
speed_ms = distance_meters / delta_time
speed_kmh = speed_ms * 3.6
But pixels_per_meter isn’t constant across the frame. It depends on the vehicle’s position in the image. I interpolated the scale factor based on the vehicle’s y-coordinate in the bird’s-eye view, using a pre-computed lookup table derived from the homography matrix. This reduced speed error from roughly ±18 km/h (using a single global scale) to ±4 km/h.
What Broke and What Fixed It
The biggest failure was temporal. Early versions computed speed between consecutive frames, which worked for stationary cameras but produced noisy results because frame-to-frame pixel displacement is small and susceptible to detection jitter. I switched to a sliding window approach: track a vehicle over 15 frames (0.5 seconds at 30 FPS), compute the total displacement in the bird’s-eye view, and divide by the elapsed time. This smoothed out the noise and gave consistent readings.
The second issue was camera shake. Even mounted gantries vibrate slightly, introducing apparent motion in the scene. I added a static background reference — I extracted the first frame as a background model and computed frame-to-frame affine transformation to compensate for global camera movement before applying the homography. This cut outlier speed readings by about 60%.
What the Numbers Look Like
On a test clip of 12 minutes of highway footage with 847 tracked vehicles:
- Detection precision: 94.2% at NMS IoU 0.45
- Track continuity (track length > 30 frames): 87.6%
- Mean absolute speed error: 3.8 km/h against radar gun ground truth
- End-to-end latency: 33ms per frame (detection + tracking + geometry)
The system runs comfortably in real time on a single GPU. The geometry correction step adds about 2ms per frame — negligible compared to the ~28ms YOLOv8 inference takes.
What I’d Do Differently
If I were rebuilding this today, I’d replace the homography approach with a direct monocular depth estimation model. The homography assumes a flat road surface, which breaks at curves and intersections. A depth-aware approach would give per-object scale factors instead of position-based interpolation, handling elevation changes and non-flat road geometry. But that’s a post-mortem thought — the homography pipeline shipped and worked well enough for the deployment it was built for.
The real lesson was that the computer vision part was the easy part. The hard part was the spatial reasoning: figuring out how to translate what the camera sees into what actually happened on the road.