Pre-1.0 — active development
Twill DB is under active development (currently 0.x). Interfaces, the SQL surface, on-disk and storage formats, and behaviour may change between releases — backward compatibility is not guaranteed until the 1.0.0 release. Pin an exact version and review the release notes before upgrading.
Where the project stands
Every capability is implemented and gated by tests; the storage seam has never moved, so each one was purely additive. The project grows further by composition.
Shipped
Embedded library ✓ Shipped
libengine + engine.h + @twilldb/bun + LocalFileStorage
The engine as an in-process library bound into Bun via bun:ffi, backed by LocalFileStorage (a plain file:// database, no network). SQL → MVCC snapshot isolation → crash-safe WAL durability with deterministic replay. The C ABI is frozen here and reused unchanged by everything built since.
Gate: basic SQL correctness + MVCC snapshot isolation; durable-after-ack with torn-frame recovery. Implementation map → · Engine Core spec
Object-storage backend ✓ Shipped
ObjectStorage (LSM page store + S3-CAS commit log)
A second Storage impl makes the database disaggregated and scale-to-zero — while staying embedded. Flip the connection string from file:// to s3:///r2:///gs:// and the same binary durably bottoms out on object storage, with the local cache keeping S3 latency off the read hot path.
Gate: commit-latency floor, group-commit throughput, and crash-safety (every acked commit survives kill -9). Implementation map → · Object-Storage spec
Server mode + pgwire ✓ Shipped
engine-server (Postgres-wire subset)
The same engine wrapped in a Postgres-wire listener. Ordinary Postgres clients — psql, Bun.sql, pgbench, PostgREST — connect with no bespoke driver. The server links the engine's Rust API unchanged and adds exactly one thing: the listener.
Gate: serves Bun.sql and pgbench over the wire; contention behaviour reproduces in server mode. Implementation map → · Server Mode spec
Branching & lifecycle ✓ Shipped
BranchStorage + durable single-writer lease + twill-controller
Copy-on-write branching as a storage-seam concern (a parent read-through below the fork LSN plus a private write overlay), a durable single-writer lease fenced by a monotonic CAS epoch, and a scale-to-zero lifecycle controller (cold-start, idle reaper, lease heartbeat, thundering-herd admission). All additive: STORAGE_TRAIT_VERSION and ENGINE_ABI_VERSION bumped to 2.
Gate: scale-to-zero + cold-start with the fence re-acquired by exactly one writer; O(1) branch creation with write isolation. Implementation map → · Lifecycle spec
Vector search (in-core) ✓ Shipped
in-core vector(N) type + HNSW + distance operators + KNN · composition
Vector search built into the engine: a vector(N) type, an HNSW access method (CREATE INDEX … USING hnsw), the distance operators <-> / <=> / <#>, and top-k nearest-neighbour answered by the index. Because the index rides the same WAL/replay path as the rows, it branches and scales-to-zero with the database — an agent can fork its memory. Interface/service capabilities (better-auth, PostgREST, DuckDB) are composed around the core, never welded in. Additive: ENGINE_ABI_VERSION → 3, STORAGE_TRAIT_VERSION stays 2.
Gate: HNSW top-k matches brute force; WHERE-filtered + MVCC-correct KNN; the index branches and rebuilds from the WAL on restart. Implementation map → · Capabilities spec
Roadmap
Vector hardening Planned
deferred items from the vector-search scope boundaries
- MAY add a page-laid-out vector index as a cold-read optimization (the current index is WAL-derived in-memory state, rebuilt on open — correct and branch-safe, but warms on every cold start).
- MAY support incremental index maintenance under heavy delete churn (deletes are tombstoned today; full-vs-incremental rebuild is an open question).
- SHOULD expose richer recall tuning — HNSW is approximate by design, governed by
ef_search; surface and document the recall/latency trade-off.
Detail in Capabilities: Build-in vs Compose and the vector-search implementation map.
Exploratory / parallel tracks Exploring
- MAY add a WASM build track (Cloudflare Workers + R2) as a parallel deliverable — a port, not a recompile, and off the native critical path. See Deployment Targets.
- MAY decide NAPI vs FFI for a single Bun + Node package if that becomes a hard requirement.
- SHOULD make the exact pgwire subset boundary explicit — the minimum for
Bun.sql/ PostgREST /pgbenchmay be smaller than full Postgres compatibility. - MUST keep the per-tool go/no-go decision attached to rollout, routing write-heavy hot-row outliers to coupled Postgres rather than blocking a gate. See Hot-Row Contention.
Milestone table
Each milestone is independently shippable and adds exactly one capability, because the seam below it never moves.
| Milestone | Headline deliverable | Gate(s) | State after |
|---|---|---|---|
| M1 — Embedded ✓ | libengine + engine.h + @twilldb/bun + LocalFileStorage | Basic correctness + MVCC snapshot | In-process persistent DB, zero infra |
| M2 — Disaggregated ✓ | ObjectStorage (LSM page store + S3-CAS log) | Commit latency, group commit, durability | Disaggregated + scale-to-zero, still embedded |
| M3 — Server ✓ | engine-server + pgwire subset + pooler guidance | Group-commit + contention in server mode | Multi-client; PostgREST + Bun.sql for free |
| M4 — Controller ✓ | Lifecycle state machine + branch-on-LSN + CAS fencing | Cold read + thundering-herd | True scale-to-zero + instant clones |
| M5 — Capabilities ✓ | Vector-in-core (HNSW); compose better-auth / PostgREST / DuckDB | HNSW top-k matches brute force; MVCC-correct KNN | Platform grows by composition |
How releases are versioned
Releases are versioned from the Cargo workspace version (Cargo.toml → [workspace.package] version). CI tags a release automatically on main once code quality, security, complexity, and unit tests are all green; the /release skill is the manual equivalent. Each release publishes the prebuilt libengine and the matching engine.h.