Daystorm's Per-Channel Staleness Budgets, Enforced in Code

I was looking at the Daystorm logs at 02:14 am when the camera sample arrived 132 ms past its 90 ms staleness budget, and the validity bit flipped from 1 to 0. The event grid marked the frame stale, the downstream causal model received a missing-data sentinel, and the vehicle-speed estimate for the intersection went from 45 km/h to an unexplained 0 km/h. That single frame broke a chain that had been running smoothly for three days. The incident forced me to codify what had been implicit: every sensor channel in Daystorm needs its own staleness budget, and every ingested sample must carry a validity flag that downstream code can read without guesswork.

The Staleness Budget

Daystorm builds an event grid at a fixed 2 Hz cadence. Each incoming sample—camera image, CAN bus message, radar return, cabin audio snippet—is timestamped and tagged with a channel-specific staleness budget. The camera budget is 90 ms; the CAN bus budget is 30 ms; the radar budget is 45 ms. When a sample’s age exceeds its budget, the grid injects a validity bit set to 0, signalling that the data are too stale to be used as-is. The resample_causal routine then pulls the nearest preceding valid sample for each channel, never reaching into the future. This design keeps the grid deterministic and the model input size constant, but it also means that a single out-of-budget sample can propagate a zero-validity flag through the entire pipeline.

I debated whether a single channel’s staleness should quarantine only that channel or kill the whole grid tick. The ADR I wrote (and then ignored for three days) says: quarantine per-channel, because the causal model already handles missing modalities via the zero-filled vector and the underflow flag. Killing the whole tick would drop good CAN and radar data just because the camera hiccuped. The 0 km/h speed estimate was the price of that choice, and it was a price I could measure.

Here’s the budget config in code—right now it’s a hardcoded dict in daystorm/grid.py:

STALENESS_BUDGETS_MS = {
    "camera": 90,
    "can_bus": 30,
    "radar": 45,
    "cabin_audio": 60,
}

You can see the asymmetry: CAN is tight because control messages are time-critical, radar gets a bit more slack because its scans are inherently slower, and camera sits in the middle. The numbers come from empirical observation of typical pipeline latencies, not from first principles. That’s the first thing that bit me.

The Event Grid and Resample Causal

The event grid lives in a NumPy array of shape (channels, time_steps). On each tick, new samples are assigned to the nearest time step; if multiple samples land on the same step, the one with the highest validity bit wins. The resample_causal function iterates over channels and, for each, finds the most recent index where validity == 1. If no such index exists, it returns a zero-filled vector and sets a global “underflow” flag that the model can inspect. The function is written in pure Python, loops over a small fixed set of channels (four in the current configuration), and runs in under 2 ms per grid tick—fast enough for real-time use but fragile when a channel’s budget is repeatedly exceeded.

def resample_causal(grid: np.ndarray, validity: np.ndarray) -> tuple[np.ndarray, bool]:
    underflow = False
    result = np.zeros_like(grid[:, 0])
    for ch in range(grid.shape[0]):
        valid_idx = np.where(validity[ch] == 1)[0]
        if valid_idx.size == 0:
            underflow = True
            continue
        result[ch] = grid[ch, valid_idx[-1]]
    return result, underflow

The loop is tiny, but the fragility lives in the assumption that every channel produces a valid sample within its budget. The camera GC pause proved that assumption false.

For the nerds: the grid is actually a rolling buffer of shape (4, 10) — four channels, ten time steps covering the last 5 seconds at 2 Hz. The validity array is the same shape, dtype np.uint8. When a sample arrives, I compute age_ms = (now - timestamp) * 1000 and set validity[ch, tick] = 1 if age_ms <= STALENESS_BUDGETS_MS[ch] else 0. The resample_causal function then does a linear scan backwards from the current tick. It’s O(channels × time_steps), which is 40 operations per tick — trivial, but the constant factor matters when you’re running on a Jetson Nano alongside the model inference.

The Late Camera Frame

On the night of the incident, the camera pipeline was configured with a 120 fps capture rate, delivering frames every ~8 ms. Normally each frame is timestamped, converted to a normalized tensor, and injected into the grid within the 90 ms budget. That evening, a garbage-collection pause in the Python process delayed the conversion step by 112 ms. The frame arrived at the grid 132 ms after its capture timestamp, crossing the 90 ms threshold. The grid set the validity bit to 0, and resample_causal fell back to the preceding valid frame, which was 210 ms old. The model received a stale image paired with a fresh CAN message, producing a speed estimate that drifted toward zero.

I captured the age distribution of camera samples over a ten-minute window after the incident. Table 1 shows the breakdown:

Age Bucket (ms)	Count	% of Samples
0–30	1,042	86.8%
31–60	112	9.3%
61–90	38	3.2%
91–132	8	0.7%

Table 1: Camera sample age distribution, 10-minute window. The 91–132 ms bucket is the stale region.

The long tail past 90 ms was small in count but catastrophic in effect: one stale frame flipped the speed estimate to zero. That’s the power law of sensor fusion: a single bad modality can dominate the entire estimate if the fusion logic isn’t robust.

Debugging at 2 am

I added a lightweight logger that prints the age of each incoming sample relative to its budget, plus the validity bit state. Running the logger for ten minutes reproduced the failure: the camera age distribution showed a long tail past 90 ms, triggered by occasional GC pauses. I instrumented the conversion function with time.perf_counter() before and after the heavy NumPy operations, confirmed the 112 ms delay, and traced the exact line where the frame was enqueued. I also wrote a small test harness that injects synthetic delays up to 200 ms and checks that the validity bit flips as expected. The test confirmed that the budget logic works, but it also highlighted that the grid had no mechanism to alert the operator when a channel repeatedly exceeded its budget.

# test harness snippet
for delay_ms in [0, 50, 90, 112, 150, 200]:
    inject_delay(delay_ms)
    age, validity = read_grid_sample("camera")
    assert (validity == 0) == (age > 90), f"Failed at {delay_ms}ms"

Here’s the logger code, because it’s useful for any similar pipeline:

def log_sample_age(channel: str, timestamp: float, budget_ms: int) -> None:
    age_ms = (time.time() - timestamp) * 1000
    validity = 1 if age_ms <= budget_ms else 0
    print(f"[{channel}] age={age_ms:.1f}ms budget={budget_ms}ms valid={validity}")
    if validity == 0:
        print(f"[{channel}] STALE: age {age_ms - budget_ms}ms over budget", file=sys.stderr)

The 2 am debugging session was a good reminder that observability isn’t optional in real-time systems. You can’t debug what you can’t see.

The Fix

Three changes were merged into the Daystorm codebase:

Budget-aware enqueue – The camera producer now checks age = now - timestamp before sending. If age > 90 ms, the sample is discarded and a warning is emitted to stderr; the grid receives a placeholder with validity 0 already set, so downstream code does not have to guess.

def enqueue_camera_sample(frame: CameraFrame) -> None:
    age_ms = (time.time() - frame.timestamp) * 1000
    if age_ms > STALENESS_BUDGETS_MS["camera"]:
        print(f"Camera sample age {age_ms}ms exceeds budget, discarding", file=sys.stderr)
        grid.set_validity("camera", 0)  # placeholder
        return
    grid.enqueue("camera", frame.tensor, validity=1)

Validity-bit audit – A nightly job scans the last hour of grid data and reports any channel where the validity bit was 0 for more than 5 % of time steps. This catch-up metric surfaced the camera issue within the first run after the fix.
Interpolation fallback – When a sample is stale, the grid now linearly interpolates between the two nearest valid samples for that channel, rather than returning a zero-filled vector. The interpolation is only applied when the gap between the two valid samples is ≤ 180 ms; otherwise the zero flag stands. This preserves temporal continuity for the causal model while still signalling when data are truly missing.

def resample_causal_with_interpolation(grid: np.ndarray, validity: np.ndarray) -> tuple[np.ndarray, bool]:
    underflow = False
    result = np.zeros_like(grid[:, 0])
    for ch in range(grid.shape[0]):
        valid_idx = np.where(validity[ch] == 1)[0]
        if valid_idx.size == 0:
            underflow = True
            continue
        if valid_idx.size == 1:
            result[ch] = grid[ch, valid_idx[0]]
        else:
            # Use the most recent valid sample, but interpolate if the gap is small
            latest = valid_idx[-1]
            prev = valid_idx[-2]
            gap_ms = (latest - prev) * 500  # 500ms per step at 2Hz
            if gap_ms <= 180:
                # Linear interpolation between prev and latest
                alpha = 0.5  # Could be based on actual time proximity
                result[ch] = alpha * grid[ch, prev] + (1 - alpha) * grid[ch, latest]
            else:
                result[ch] = grid[ch, latest]
    return result, underflow

After deploying these changes, the camera age distribution narrowed: no frames exceeded the 90 ms budget in the following 48 hours. The validity-bit audit reported 0 % stale frames for the camera channel. The model’s speed estimates stabilized around the 44-46 km/h range, and the underflow flag never fired again.

What Still Needs Work

The staleness budgets are currently hard-coded per channel in a Python dict. Adding a new sensor—say, a thermal imager—requires editing the dict and redeploying. I plan to move the budgets into a YAML configuration file read at startup, so the grid can auto-adjust when hardware changes. I also want to expose the per-channel budget metrics via Prometheus, so operators can see real-time age distributions alongside the validity-bit stats. These tweaks will make the system more maintainable without altering the core logic that already prevents stale data from silently corrupting the model.

Specifically, the YAML would look like:

staleness_budgets:
  camera: 90
  can_bus: 30
  radar: 45
  cabin_audio: 60

And the Prometheus metrics I’m eyeing are daystorm_camera_age_ms, daystorm_camera_validity_total, and daystorm_grid_underflow_total. With these in place, the system can tell you when it’s drifting instead of silently producing a 0 km/h estimate at an intersection.