SIA Proto — Building a Local-First System Monitoring Agent with Unix-Socket IPC and Optional LLM Enrichment

Why a local-first agent instead of existing tools

The project started from a concrete frustration: every system monitor that offered structured event analysis either shipped telemetry by default (Datadog, New Relic, Grafana Cloud), required a heavy operational stack (Prometheus + Alertmanager + Grafana + node_exporter for a single box), or exposed only a raw metric stream with no CLI for “show me what happened at 14:23.”

MVP_PLAN.md codified three non-negotiables: no network egress unless explicitly asked, minimal runtime dependencies, and systemd-native deployment. The agent binary (sia-agent) runs as the sia user via a systemd unit with Restart=on-failure and StandardOutput=journal — no Docker, no sidecars, no separate metric database. The only persistent state is a SQLite file at /var/lib/sia/sia.db (schema: events table with id, timestamp, event_type, severity, message, details JSON). Total memory budget defaults to 200 MB; disk quota to 500 MB.

The CLI (sia-cli) talks to the agent over a Unix socket (/run/sia/sia.sock) using JSON-RPC 2.0 — three methods: status, list, show <id>. No HTTP server, no TLS certs, no auth layer beyond file-socket permissions. Optional LLM enrichment calls a local Ollama instance at http://localhost:11434/api/generate (default model llama3.2) only when you run sia-cli show <id> --enrich. If Ollama isn’t running, the command falls back to raw event data — no timeout hangs, no external fallback.

The entire workspace builds in three crates (common, agent, cli) with zero external services required for the MVP. First commit to working systemd service: three days.

Workspace architecture and crate boundaries

The repository is a Cargo workspace with three crates:

sia/
├── common/   # shared types, config, IPC definitions
├── agent/    # sia-agent binary — all side effects
└── cli/      # sia-cli binary — thin RPC client

common (common/src/lib.rs re-exports types, config, ipc) is the contract layer. common/src/types.rs defines Event, SystemSnapshot, Config — the exact structs that cross the Unix socket boundary. common/src/config.rs loads TOML from /etc/sia/config.toml (or $SIA_CONFIG) with environment-variable overrides (SIA_CPU_INTERVAL, etc.), so both crates parse configuration identically without duplication. common/src/ipc.rs declares the JSON-RPC 2.0 request/response enums for status, list, show — the wire protocol is compiled into both binaries, eliminating drift.

agent (agent/src/main.rs) owns every side effect: it spawns the collector loops (collectors.rs), the analyzer (analyzer.rs), the SQLite writer (storage.rs), the Ollama client (llm.rs), and the JSON-RPC server (ipc.rs) listening on /run/sia/sia.sock. No business logic lives outside this crate; the CLI cannot trigger collection or write events directly.

cli (cli/src/main.rs) does three things: parses subcommands via clap, opens a tokio::net::UnixStream to the socket, and pretty-prints the JSON-RPC responses. It has zero knowledge of sysinfo, sqlx, or reqwest — those dependencies appear only in agent/Cargo.toml.

The install.sh script reflects this split: it copies target/release/sia-agent and target/release/sia-cli to /usr/local/bin/, the systemd unit points only at sia-agent, and the socket directory /run/sia/ is created with sia:sia ownership so the CLI (run by any user in the sia group) can connect without sudo.

Collection and analysis loop

The agent runs two independent tokio tasks spawned from agent/src/collectors.rs, each driven by its own interval from Config (cpu_interval=5, proc_interval=10 by default, both in seconds). The CPU sampler calls sysinfo::System::refresh_cpu_all() and refresh_memory(), then constructs a SystemSnapshot containing per-core usage percentages, total/available memory, and a timestamp. The process sampler calls refresh_processes() and emits the top-N processes by CPU and memory (currently N=10, hard-coded). Both samplers send snapshots into a bounded tokio::sync::mpsc::channel (capacity 64) toward the analyzer task.

analyzer.rs receives snapshots sequentially. It evaluates three threshold rules against the latest CPU snapshot: cpu_warn (default 80%), cpu_crit (95%), mem_warn (85%), mem_crit (95%). When a metric crosses a threshold, the analyzer builds an Event (defined in common/src/types.rs) with event_type = CpuThreshold or MemoryThreshold, severity = Warning or Critical, a human-readable message, and a details JSON blob containing the snapshot values that triggered it. Events are forwarded to a second bounded channel (capacity 128) feeding the storage writer.

Backpressure is handled by the channel capacities: if the analyzer falls behind (e.g., SQLite contention), the collector tasks block on send().await, naturally throttling sampling without dropping data. The memory_budget (200 MB) and disk_quota (500 MB) from config are enforced in storage.rs — the writer checks the SQLite file size before each insert and, when the quota is exceeded, deletes the oldest events in batches of 100 until under the limit. No separate vacuum task runs; deletion is synchronous with the insert path.

Storage schema and query patterns

The events table is defined in sql/schema.sql with columns id INTEGER PRIMARY KEY AUTOINCREMENT, timestamp INTEGER NOT NULL, event_type TEXT NOT NULL, severity TEXT NOT NULL, message TEXT NOT NULL, and details JSON NOT NULL. id provides a stable reference for CLI show commands, while timestamp (Unix epoch) orders events for pagination. The details column stores the triggering snapshot values as a JSON object, enabling flexible enrichment later (e.g., LLM context).

storage.rs uses sqlx::SqlitePool to execute three core patterns. Insertion is performed with a prepared statement:

let _ = sqlx::query(
    "INSERT INTO events (timestamp, event_type, severity, message, details) VALUES (?,?,?,?,?)"
)
.bind(&event.timestamp)
.bind(&event.event_type)
.bind(&event.severity)
.bind(&event.message)
.bind(serde_json::to_string(&event.details).unwrap())
.execute(&self.pool)
.await;

The list method supports pagination via limit and offset parameters:

let rows = sqlx::query_as::<_, EventRow>(
    "SELECT id, timestamp, event_type, severity, message, details FROM events ORDER BY timestamp DESC LIMIT ? OFFSET ?"
)
.bind(limit)
.bind(offset)
.fetch_all(&self.pool)
.await;

show retrieves a single event by primary key:

let opt = sqlx::query_as::<_, EventRow>(
    "SELECT id, timestamp, event_type, severity, message, details FROM events WHERE id = ?"
)
.bind(event_id)
.fetch_optional(&self.pool)
.await;

Disk-quota enforcement follows the disk_quota config default of 524 288 000 bytes (≈500 MiB). Before each insert, the writer checks std::fs::metadata(&path).unwrap().len(). If the size exceeds the quota, it deletes the oldest rows in batches of 100 using:

sqlx::query("DELETE FROM events WHERE id IN (SELECT id FROM events ORDER BY id ASC LIMIT ?)")
.bind(batch_size)
.execute(&self.pool)
.await;

This synchronous deletion keeps the SQLite file within the limit without a background vacuum task. Occasionally a VACUUM; command is issued to reclaim fragmented space, ensuring the file size stays close to the quota and preserving fast insert performance.

IPC protocol and CLI contract

The agent exposes a JSON-RPC 2.0 interface over a Unix domain socket at /run/sia/sia.sock. The protocol types live in common/src/ipc.rs so both crates share identical request/response definitions without serialization drift.

Request/response types

#[derive(Serialize, Deserialize)]
#[serde(tag = "method")]
pub enum Request {
    Status,
    List { limit: Option<u32>, offset: Option<u32> },
    Show { id: i64, enrich: Option<bool> },
}

#[derive(Serialize, Deserialize)]
pub struct Response<T> {
    pub jsonrpc: String,
    pub id: Option<Value>,
    #[serde(flatten)]
    pub result: Result<T, RpcError>,
}

RpcError carries a code (−32600 for invalid request, −32601 for method not found, −32000 for internal errors) and a message string. The enrich flag on Show triggers the optional Ollama call; the agent returns the event with or without the llm_analysis field populated.

Server loop (agent/src/ipc.rs)

The agent binds the socket, sets SO_PEERCRED to verify the caller’s UID matches the sia user, then enters a tokio::net::UnixListener accept loop. Each connection spawns a task that reads framed JSON lines, deserializes to Request, dispatches to the storage or analyzer state, and writes the Response back. Errors in deserialization or dispatch return the appropriate RpcError without closing the socket.

CLI mapping (cli/src/main.rs)

The CLI uses clap subcommands that map 1:1 to RPC methods:

Subcommand	RPC method	Parameters
status	status	none
list	list	--limit, --offset
show <id>	show	--enrich / --no-enrich

Each subcommand opens a UnixStream, sends the framed request, reads the response, and pretty-prints the result field. On show, the output includes the raw details JSON and, when enrichment succeeded, the llm_analysis block. Non-zero exit codes surface RPC errors to the shell for scripting.

LLM enrichment as an optional layer

The agent’s llm.rs module implements on-demand event enrichment via a local Ollama instance. When the CLI invokes show <id> --enrich, the IPC show handler calls enrich_event() before returning the response.

Request construction

enrich_event() builds a prompt from a static system instruction plus the event’s structured fields:

let prompt = format!(
    "System event: type={}, severity={}, message={}, details={}",
    event.event_type, event.severity, event.message, event.details
);

It sends an HTTP POST to http://localhost:11434/api/generate with JSON body:

{
  "model": "llama3.2",
  "prompt": "<system prompt>\n\n<prompt>",
  "stream": false,
  "options": { "temperature": 0.2 }
}

The model name is configurable via [llm] model = "llama3.2" in /etc/sia/config.toml (or SIA_LLM_MODEL env override). The endpoint URL is similarly configurable ([llm] endpoint).

Timeout and fallback

The request uses a 30-second reqwest timeout. On any error — connection refused, non-2xx response, JSON decode failure, or timeout — the function returns Ok(None) rather than propagating an error. The IPC layer then returns the event with llm_analysis: null. This keeps the agent functional when Ollama is stopped, upgrading, or absent.

Response handling

A successful response extracts the response field from Ollama’s JSON, trims whitespace, and stores it in the event’s llm_analysis field. The CLI prints this block after the raw details JSON. No caching occurs; each show --enrich triggers a fresh request.

Configuration example

[llm]
enabled = true
endpoint = "http://localhost:11434"
model = "llama3.2"
timeout_secs = 30

Setting enabled = false or omitting the section skips the HTTP call entirely.

Installation, service management, and dev ergonomics

The install.sh script performs a full system deployment in one pass. It creates a dedicated sia user and group (useradd -r -s /usr/sbin/nologin sia), then builds both binaries in release mode (cargo build --release --workspace). Binaries land in /usr/local/bin/ (sia-agent, sia-cli). Configuration installs to /etc/sia/config.toml with the defaults from config/default.toml (memory_budget=209715200, disk_quota=524288000, cpu_interval=5, proc_interval=10). The script creates /var/lib/sia/ (owned by sia:sia, mode 750) for the SQLite database and /run/sia/ (same ownership, mode 755) for the Unix socket. The systemd unit file installs to /etc/systemd/system/sia-agent.service.

The unit runs as the sia user with Restart=on-failure and RestartSec=5. StandardOutput=journal and StandardError=journal route all logs to journald; env_logger picks up RUST_LOG from the environment. No socket activation is used — the agent binds /run/sia/sia.sock itself on startup. After installation, systemctl daemon-reload && systemctl enable --now sia-agent starts the service.

For development, QUICKSTART.md describes a two-terminal workflow: terminal one runs cargo run --bin sia-agent (loads config from config/default.toml, writes socket to /run/sia/sia.sock, database to ./sia.db); terminal two runs cargo run --bin sia-cli -- status or list against the local socket. No sudo required; directories are created under the project root.

uninstall.sh reverses the install: systemctl stop sia-agent && systemctl disable sia-agent, removes the unit file, deletes /usr/local/bin/sia-agent and /usr/local/bin/sia-cli, removes /etc/sia/, /var/lib/sia/, /run/sia/, and finally deletes the sia user and group (userdel sia; groupdel sia).