Ingesta — An Import Wizard, and the Workspace That Makes Its Mistakes Fixable

The gap nobody builds

Getting a spreadsheet into a database is easy once and impossible to trust. You parse a workbook, infer a schema, create a table, insert the rows — and then the data is wrong in eleven ways you cannot see yet. Dates that came in as text. A column that is half numbers and half thousands separators. The same customer keyed three ways.

The usual answer is to skip the middle and fix it in the database, with SQL you run by hand against a table you already committed to. That is unreviewable, it is hard to take back, and nobody writes down what they changed.

So Ingesta has two halves, and the interesting one is the second.

One shape, one set of types

The import wizard is one-shot and boring on purpose: upload workbooks, preview the sheets, pick a connection, create a table, insert the rows, done. The browser holds the file list and the run history; the server opens a short-lived connection per call, does one job, and closes it.

Everything past the parser speaks exactly one shape, declared in lib/types.ts:

GridColumn { name, type, nullable, isPrimaryKey?, defaultValue? }
Grid       { columns: GridColumn[], rows: unknown[][] }

The important decision is in type. It is always semantic — VARCHAR(100), DECIMAL(10,2), DATE, BOOLEAN — and never an engine’s name. No layer above lib/db/dialect.ts knows whether the destination ends up being PostgreSQL, MySQL, SQL Server or SQLite, and adaptTypeForDatabase does the mapping at the moment the DDL is built. Adding an engine means touching one file rather than every operation that happens to mention a type name.

Rows are positional, so row[i] belongs to columns[i] and a Grid has no way to express a misaligned row. Every operation that changes columns rebuilds the rows in the same pass, which is a little more work and removes an entire category of bug.

Operations are values, not actions

The reshaping layer is a discriminated union of plain serialisable objects: filter, derive, rename, drop, keep, reorder, cast, fill, dedupe, sort, trim, replace, limit.

applyOperation(grid, operation) in lib/operations.ts is a pure function. It never mutates its input; it returns { grid, effect, findings? }. And effect is measured while the operation runs — rowsIn, rowsOut, cellsChanged, columnsIn, columnsOut are all counted, which is what the history list shows you instead of a shrug.

Making operations values rather than closures is what lets the whole thing serialise. A step is something you can store, show, re-run and diff.

Rollback is replay, not undo

A dataset is not a grid. A dataset is a base grid plus an ordered list of operations — which means the grid you are looking at is the result of replaying them.

That sounds like a roundabout way to do undo. It is the opposite. Undo requires the engine to have kept enough state to reverse itself, and reversing a dedupe or a fill is genuinely hard. Replay requires nothing but the ability to run the same list again, and it gives you the history, the diff and the audit trail as a side effect of the storage format rather than as a feature bolted onto it.

The boring part is the guardrails

Because operations are values, they can be classified before they run. lib/guardrails.ts sorts them into safe, caution and destructive, and Table Studio will not apply a destructive change to a live table without a snapshot taken first and a plan you have read.

The value of a snapshot is not that you will use it. It is that the moment somebody asks “what did you just do to the customers table”, the answer is a restore button rather than an archaeology project.

Checked by scripts, not vibes

There are seven Bun scripts under scripts/check-* — pipeline, workspace, data, table-ops, tables, sqlite, import-execution — and npm run check runs all of them. They exist because the interesting failures in an app like this are not crashes; they are a cast that quietly produced NaN, or an import that created a table with the wrong nullability.

The docs are unusually honest about their own rules too. ARCHITECTURE.md, WORKSPACE.md and TRACKER.md are written as decisions with reasons attached, including the measured contrast ratios for the theme tokens, so a colour change has to argue with a number.