Building a Four-Modality Driving-Scene Copilot on a T4 GPU
How we fused camera, CAN bus, radar, and cabin audio into a 2 Hz event grid and a frozen vision-language model to answer grounded driving questions.
The problem
Aligning four asynchronous sensor streams with different sample rates and dropouts while preserving causality and running on a free-tier T4 GPU that lacks bfloat16 support and has limited VRAM.
The solution
We grafted new modality encoders and a fusion layer onto a frozen 3B vision-language backbone, resampled all streams onto a causal 2 Hz event grid with validity bits, and used QLoRA with fp16 and torch.compile to keep the trainable surface under 6M parameters.
Outcomes
- The full model achieved 1.00 manoeuvre accuracy and 0.225 numeric MAE on a held-out split with zero hallucination.
- Removing the CAN bus increased numeric MAE by 0.44 (to 0.663) and raised hallucination to 20.8%, showing the telemetry path is critical for grounding.
- Fusion latency on a T4 measured 2.557 ms p50 with torch.compile in fp16, while decode consumed 98.5% of end-to-end time.
+1 more in the write-up
Inside the write-up
- Design choices
- Training pipeline
- Evaluation
- Resource cost
- Open questions and next steps