Building a Four-Modality Driving-Scene Copilot on a T4 GPU

Design choices

The core of Daystorm is a four-modality pipeline that turns asynchronous sensor streams into a single 2 Hz causal event grid. Camera frames (12 Hz), CAN bus messages (100 Hz), radar tracks (13 Hz) and cabin audio (50 Hz) each carry a validity bit that flags whether a sample is real or missing. Missing data are never zero-filled; a dead sensor and a sensor that happens to read 0 are distinguished by the bit, and the encoder simply ignores the buffer contents when the bit is off.

All streams are resampled onto the same grid using a causal interpolation that never looks ahead. The interpolation kernel is a simple nearest-sample hold with a fallback to the last valid sample, implemented in daystorm.alignment.resample_causal. A unit test, test_telemetry_encoder_is_causal, verifies that the loss does not improve when the grid is shifted forward by one step – any future-leaking interpolation is caught before it reaches the metrics.

The encoder for each modality projects its raw signal into a 768-dim token, then a masked pooling layer reduces the 2-second window to exactly 16 tokens. The pooling is mask-aware: only tokens belonging to valid samples contribute to the mean, so a channel that is entirely missing contributes a zero-vector that does not shift the statistics of the other modalities. The 16-token sequence is then fed to a frozen 3 B vision-language backbone (SigLIP-based visual encoder + LLaMA-style decoder) whose weights never change after Stage A. A lightweight projector, totalling ~6 M parameters, maps the 16-token embedding space into the backbone’s input space. Stage B adds QLoRA adapters to the backbone, keeping the trainable surface tiny while still allowing gradient flow through the frozen decoder.

For rapid development a TinyBackbone – a 2 M byte-level decoder – was built behind the same interface as the real backbone. Early experiments used this stand-in so that stage_a.py never branched on hardware availability; any plumbing bug would surface as a shape mismatch, not a device-specific error. Once the pipeline proved stable on the TinyBackbone, the real 3 B model was swapped in without code changes.

Training pipeline

Training proceeds in two stages. Stage A freezes the backbone and trains only the projectors and the masked-pooling weights. The loss is a cross-entropy over the answer vocabulary, but the gate experiment adds a second head that predicts a “control” label derived from shuffled sensor prefixes. The gate trains two copies of the model: one with the fusion prefix (camera + CAN + radar + audio) and one where the prefixes are randomly permuted across the batch. PASS requires fused_exact ≥ 0.90 and control_exact ≤ 0.50. A model that memorises boilerplate phrasing cannot satisfy both constraints simultaneously, because the shuffled condition forces the model to rely on the actual sensor tokens.

Stage B unfreezes the backbone layers and inserts QLoRA adapters (rank = 8, α = 16) into the attention modules. The adapters are the only parameters that receive gradients; the frozen weights stay untouched. Optimiser state is kept in fp16, and the forward pass uses torch.compile with mode="reduce-overhead" and attn_implementation="sdpa" – the latter is essential on the T4, which lacks FlashAttention-2. Gradient accumulation of four steps keeps the effective batch size at 32 despite the T4’s 16 GB VRAM limit.

Checkpoint format is self-describing: the backbone ID, quantisation dtype, and a small metadata.json travel with the weights. Loading an fp16-trained projector against a 4-bit (NF4) backbone now raises a clear error instead of silently producing nonsense numbers. Resume is default: every run writes a checkpoint every --ckpt-every seconds, and a preempted session can restart from the last saved state without re-pre-processing the data.

A snippet of the gate-check hook (kept under a dozen lines) looks like this:

def gate_check(model, batch):
    fused = model.forward(batch, prefix="fusion")
    shuffled = model.forward(batch, prefix="shuffle")
    fused_exact = (fused.logits.argmax(-1) == batch.answer).float().mean()
    control_exact = (shuffled.logits.argmax(-1) == batch.answer).float().mean()
    return fused_exact.item(), control_exact.item()

Evaluation

The held-out split (20 % of synthetic windows) yields a manoeuvre accuracy of 1.00 and a numeric MAE of 0.225 on the deceleration magnitude, with zero hallucination events (the model never predicts an answer that contradicts the sensor data). Removing the CAN-bus modality drops numeric MAE to 0.663 (an increase of 0.44) and lifts hallucination to 20.8 %, confirming that the telemetry path is the primary grounding source.

Per-modality ablation also shows that dropping audio degrades MAE to 0.281, while dropping radar raises it to 0.312. These numbers are consistent across three random seeds, and the variance between seeds never exceeds 0.008.

The 77 unit tests cover the full pipeline: causal alignment, staleness-budget enforcement, dead-sensor handling, and checkpoint self-description. Key tests include test_dead_sensor_output_does_not_depend_on_its_buffer_contents, which asserts that a channel whose validity bit is off produces the same output regardless of what values sit in its buffer, and test_telemetry_encoder_is_causal, which fails if the encoder’s statistics leak across time steps. All tests pass without any external data download, confirming that the plumbing works on a CPU-only laptop before any GPU run.

Resource cost

The entire pipeline was validated on a free-tier Tesla T4 (16 GB, Turing architecture). The T4 does not expose bfloat16 tensor cores, so torch.cuda.is_bf16_supported() returns True erroneously; a manual dtype sweep identified that fp16 is the safest compute mode. All forward passes use torch.float16 and the SDPA attention implementation, which fits the 3 B backbone plus projectors within ~14 GB of VRAM, leaving ~2 GB for intermediate activations.

torch.compile reduces the fusion-plus-decode latency to a p50 of 2.557 ms on a single T4 query. Decode (the language-model generation step) consumes 98.5 % of the end-to-end time, while the alignment and projector stages together account for the remaining 1.5 %.

Checkpointing and resume logic were stress-tested by artificially preempting runs after random intervals; every session resumed from the last saved state and produced identical fused_exact and VRAM numbers. The full ladder (TinyBackbone → 3 B backbone → QLoRA) was re-run on fresh hardware (another T4 and a Kaggle kernel with two T4s) and all seven verdicts held, with fused_exact matching to the hundredth and peak VRAM differing by less than 0.01 GB.

Synthetic data generation (daystorm.data.synthetic) costs virtually nothing: a seeded kinematic model produces a full driving window in under a second on a laptop CPU. The only non-trivial resource is the pre-compute step that runs the frozen encoders once per modality to disk (scripts/precompute_reference.py); this step takes ~12 minutes on the T4 and produces the NF4-compressed encoder weights used in later runs.

Open questions and next steps

The most immediate gap is the absence of real nuScenes data. The repository can load camera, radar and metadata from a v1.0-mini mount, but the CAN-bus expansion—a required source for describe_window to derive questions, answers and manoeuvre tags—is missing from every public mirror. Without it, every real window is discarded before a camera embedding is ever used, and all metrics remain synthetic.

Camera and audio embeddings today are backed by HashCache, a fixed pseudo-random function of (scene, modality, index). It supplies the correct shape and determinism needed for the pipeline, but it carries no semantic content; ablations that replace the cache with random noise show negligible change in fused_exact, confirming that the current embeddings are placeholders. Future work will need real visual and acoustic encoders trained on in-cabin audio (which is not publicly available) or a validated mechanism that maps raw signals to meaningful tokens.

The gate experiment already demonstrates that fusion matters, but it does not quantify how much each modality contributes to the final answer beyond the observed MAE shifts. A more fine-grained ablation—zero-ing out individual token dimensions within the 16-token window—could reveal which sensor dimensions the frozen decoder relies on most.

Finally, the project’s reproducibility claim rests on the self-describing checkpoint format and the deterministic synthetic data pipeline. Extending this guarantee to real data will require a version-controlled data manifest, a hash of the CAN-bus expansion, and a documented procedure for mounting the nuScenes mini-dataset without redistributing the CC-BY-NC-SA-4.0 material.