Architecture Overview

Purpose & scope

This document is the top-level map of the system. It fixes the three architectural slots, the responsibilities of each, the protocols that cross between them, and the read/write data flows that traverse all three. Every later spec (02–08) owns the interior of exactly one slot or one inter-slot boundary; this page is the contract that holds them together and the place to resolve disagreements about where a responsibility lives.

The governing design tension, and its resolution, is the reason the architecture has the shape it does:

Resolution. Keep the engine a library, but make its storage backend pluggable and point it at the network instead of at a local file. The seam is a storage interface inside the library, not a server boundary. Server mode is then merely "the same library, wrapped in a wire-protocol listener." Reference shape: "SQLite that forgot where its file was and found it on S3." Closest shipping analog: libSQL/Turso.

Responsibilities & non-goals

In scope for the architecture as a whole

• Define the three slots (Interface, Engine, Storage) and the seam between Engine and Storage.
• Define the four protocols at the slot boundaries and which ones are new.
• Define the canonical read path (GetPage@LSN) and write path (Append(WAL) → durable ack → materialization).
• Define process and trust boundaries for embedded vs server deployment of the same binary.
• Fix the cross-cutting invariants every slot must uphold (durability floor, single-writer-per-DB, embeddability-depends-on-cache).

Explicit non-goals (owned elsewhere or out of band)

• SQL surface, planner, executor and MVCC internals — Engine Core.
• The exact Storage trait signatures and versioning — Storage Interface (this page shows the canonical shape only).
• LSM layer format, compaction, GC, and the S3-CAS commit log — Object-Storage Backend.
• Multi-writer coordination beyond single-writer-per-DB. The architecture caps at one writer per database by design (SQLite-lineage ceiling); a coordination layer is out of scope. See Hot-Row Contention.
• Analytical/columnar execution (HTAP). The row engine stays small; OLAP is composed over shared storage — Capabilities.

System context & the three slots

The system is a vertical stack of three slots. An application sits above Slot A; durable bytes bottom out below Slot C. The hot path lives in Slot B (the local cache); the network is only crossed on a cache miss (reads) or on commit (writes).

Slot A — Interface

Embedded

FFI / native addon. The engine runs inside the host process (Bun, Node via NAPI, any C-ABI caller). No socket, no wire protocol; the application calls the engine's C ABI directly. See Bun Integration.

Server

The same engine wrapped in a network listener speaking the Postgres wire protocol (pgwire) or HTTP/WS. See Server Mode & Wire Protocol.

Pooler

Server mode only. PgBouncer / pgcat in transaction mode, fronting the listener to absorb serverless connection bursts. Absent entirely in embedded deployments.

Slot B — Engine (the library)

The engine owns: the SQL parser; logical and physical planning; the executor; the MVCC / transaction manager (snapshot isolation via LSN-stamped versions); WAL generation; the local cache that keeps hot pages in-process; and the pluggable Storage interface — the single most important artifact in the system. The cache is not an optimization here; it is load-bearing for embeddability (see Cross-cutting NFRs). Detail in Engine Core and Local Cache.

Slot C — Storage (disaggregated)

Log service

Durable, totally-ordered commit log. A commit is acknowledged only on a quorum/CAS ack. Durability bottoms out on S3 conditional writes (compare-and-swap), giving atomic ordered appends and single-writer fencing without a separate Raft/Paxos cluster.

Page store

LSM layers versioned by LSN; materializes pages on demand. In-memory layer → delta layers (key, LSN) → image layers (full page snapshots), with compaction and GC past the PITR window. SlateDB is the model; RocksDB if a local cache tier is wanted underneath.

Durability floor

Any S3-compatible object store (AWS S3, Cloudflare R2, MinIO). Reached only on a cache miss for reads, and on commit for writes.

Protocol matrix

Four boundaries, four protocols. Exactly one of them is new, and it is internal. The rest are direct function calls or existing standards.

Engine ↔ Storage — canonical interface shape

The engine calls this narrow trait instead of touching disk. The full, versioned signatures live in Storage Interface; the canonical shape is fixed here so every reader shares one mental model:

/// The seam. The engine knows only this; it never names a file or a bucket.
pub trait Storage: Send + Sync {
    /// Read path. Return the page as of (≤) the given snapshot LSN.
    fn get_page(&self, page_id: PageId, lsn: Lsn) -> Result<Page, StorageError>;

    /// Write path. Durably append WAL records; return the assigned commit LSN.
    /// MUST NOT return Ok until the records are durable (see durability rule).
    fn append_wal(&self, records: &[WalRecord]) -> Result<Lsn, StorageError>;

    /// Force any buffered state to the durability floor.
    fn flush(&self) -> Result<(), StorageError>;
}

PageId

Opaque page identifier (database + relation + page number). Stable across versions.

Lsn

Monotonic log sequence number. Totally orders all commits in a database; also the version stamp for MVCC reads and the branch pointer for clones.

WalRecord

A single redo record produced by the executor for a committing transaction. The unit the log service appends and durably orders.

Page

A fixed-size page image materialized from the page store at or before the requested LSN.

Two reference implementations sit behind this trait and are selected by connection string, not by recompilation:

• LocalFileStorage — a single .db file. Pure embedded, zero network. Dev, offline, edge. (file://./local.db)
• ObjectStorage — the disaggregated backend of Slot C. (s3://bucket/mydb)

Read path data flow

Reads resolve a page as of a snapshot LSN through a cache hierarchy. The network (object storage) is touched only on a full miss. This hierarchy is what keeps S3's hundreds-of-milliseconds latency off the hot path and is the reason the engine stays embeddable.

  executor needs  GetPage(page_id, lsn)
        │
        ▼
  ┌───────────────┐   hit   ┌──────────────────────────┐
  │ shared buffer │ ──────▶ │  return page (in-process) │   ◀── hot path, ~µs
  │  (in memory)  │         └──────────────────────────┘
  └──────┬────────┘
         │ miss
         ▼
  ┌───────────────┐   hit   ┌──────────────────────────┐
  │ local file    │ ──────▶ │  promote → shared buffer  │   ◀── warm, local NVMe
  │ cache (LFC)   │         └──────────────────────────┘
  └──────┬────────┘
         │ miss
         ▼
  ┌───────────────────────────────────────────────────┐
  │ Slot C page store: pick image layer ≤ lsn,         │
  │ replay delta layers (key, LSN) up to lsn           │
  └──────┬────────────────────────────────────────────┘
         │ layer(s) not resident
         ▼
  ┌───────────────────────────────────────────────────┐
  │ object storage GET (S3 / R2)  ◀── cold, ~100s of ms │
  │ materialize page, populate LFC + shared buffer      │
  └───────────────────────────────────────────────────┘

1. Executor requests get_page(page_id, lsn) where lsn is the transaction's snapshot.
2. Shared buffer (in-memory page cache) — hit returns in-process at function-call latency.
3. Local file cache (LFC, local NVMe) — hit promotes the page to the shared buffer.
4. Page store resolves the version: select the image layer at or below lsn, then replay delta layers up to lsn.
5. On miss at the layer level, issue an object-storage GET, materialize the page, and populate both caches on the way back.

Write path data flow

Writes append WAL records to the durable, ordered commit log; the commit is acked only after those records are durable; page materialization happens later, off the commit path. This ordering is the durability rule made concrete.

  COMMIT (txn produces WAL records)
        │
        ▼
  ┌───────────────────────────────────────────────┐
  │ Engine: append_wal(records)                    │
  └──────┬────────────────────────────────────────┘
         ▼
  ┌───────────────────────────────────────────────┐
  │ Slot C log service: append to ordered commit   │
  │ log; durability via S3 conditional write (CAS) │   ◀── network round-trip
  │ → assign commit LSN, single-writer fencing     │
  └──────┬────────────────────────────────────────┘
         │ durable ack (and only then)
         ▼
  ┌───────────────────────────────────────────────┐
  │ Engine: return commit LSN to caller  ── ACK ──▶ │   ◀── client sees success
  └──────┬────────────────────────────────────────┘
         │  (asynchronous, off the commit path)
         ▼
  ┌───────────────────────────────────────────────┐
  │ Page store: apply WAL → delta layers → image   │
  │ layers; populate cache; GC past PITR window    │
  └───────────────────────────────────────────────┘

1. The committing transaction's redo records are handed to append_wal(records).
2. The log service appends them to the totally-ordered commit log; durability bottoms out on an S3 conditional write (CAS), which also assigns the commit LSN and enforces single-writer fencing.
3. Only after the durable ack does the engine return the commit LSN to the caller. The client never sees success before the WAL is durable.
4. Page materialization (apply WAL → delta layers → image layers, populate caches, GC) runs asynchronously, off the commit path.

Process & trust boundaries

The defining property: embedded and server are the same binary. Storage choice is config (a connection string), not a rebuild. The difference between the two deployments is only whether a network listener exists in front of the engine.

Component responsibility matrix

Each slot and boundary is owned by exactly one downstream spec. This page owns the contract between them; it does not own any slot's interior.

Cross-cutting non-functional requirements

Three invariants bind every slot. They are restated here because no single downstream spec can enforce them alone — they are properties of the whole.

Durability rule

A commit MUST NOT be acknowledged before its WAL records are durably stored. Caching may hide read latency completely; it must never hide commit latency. Violating this yields acked-write loss on crash, which is disqualifying regardless of latency numbers (validated by Experiment 4).

Single-writer-per-DB

Without an added coordination layer, one writer at a time per database (a SQLite-lineage ceiling), fenced by the commit log's CAS token. This is excellent for "many small databases / dozens of tools" and wrong for one giant high-concurrency database. The lever for write parallelism is many small DBs (independent lanes), not multi-writer in one DB. See Hot-Row Contention.

Embeddability depends on the local cache

Embeddability is not a property of the API alone — it depends on the local cache. Without it, every read is a network call and the "in-process function-call latency" promise collapses. The cache is therefore a correctness-adjacent component for the embeddability NFR, not merely a performance tweak. See Local Cache.

Configuration

Architecture-level knobs that select deployment shape and storage. Per-slot knobs live in their own specs; these are the ones that change which slots exist.

connection_string

Selects the storage backend. file://<path> → LocalFileStorage; s3://<bucket>/<db> → ObjectStorage. Same code path; no rebuild.

mode

embedded (no listener; engine in host process) or server (network listener + optional pooler). Same binary.

listener

Server mode only. pgwire | http | ws, plus bind address/port. Ignored in embedded mode.

idle_timeout

Lifecycle: how long an idle engine stays up before scale-to-zero teardown. Trades cold-start frequency against at-rest compute cost. Owned by 06.

group_commit

Enable/queue parameters for batching the durable-append round-trip (the W1 lever). Owned by 04 / measured in 09.

// Embedded, pure-local — zero network
const h = db.engine_open(Buffer.from("file://./local.db\0"));

// Embedded, disaggregated — same code path, flip the string
const h = db.engine_open(Buffer.from("s3://bucket/mydb\0"));

// Server — same binary, listener wraps the lib (see 07)
//   engine-server --mode=server --listener=pgwire --bind=0.0.0.0:5432 \
//                 --connection=s3://bucket/mydb

Failure modes & edge cases

Behaviour & algorithms — the boundary invariants

The architecture's behaviour reduces to a small set of ordering and versioning invariants that every slot must preserve.

• LSN total order. Every commit in a database receives a strictly monotonic LSN from the log service. The LSN is simultaneously the commit order, the MVCC snapshot stamp (get_page resolves "≤ lsn"), and the branch pointer for clones.
• Durable-before-ack. The only synchronous step on the commit path is the durable append; everything downstream (page materialization) is asynchronous and re-derivable from the WAL.
• Immutable layers. Page-store layers are append-only and versioned by LSN. Because they never mutate in place, a branch is a new LSN pointer over the same shared immutable layers (copy-on-write) — instant clones with near-zero marginal storage until divergence.
• Fencing via CAS. Single-writer-per-DB is enforced by the commit log's compare-and-swap token, not by a separate consensus cluster. Losing the CAS means losing the fence; the loser must stop.
• Cache is a hint, never a source of truth. The shared buffer and LFC may be dropped at any time (scale-to-zero, eviction) without affecting correctness; they only affect latency.

Normative requirements

• MUST The engine MUST reach durable storage only through the Storage interface; no slot above Slot C may name a file path or bucket directly.
• MUST A commit MUST NOT be acknowledged to the caller before its WAL records are durable at the durability floor.
• MUST The same compiled binary MUST support both embedded and server modes; the storage backend MUST be selectable by connection string without a rebuild.
• MUST The Engine↔Storage RPC (Append(WAL), GetPage@LSN, Flush) MUST remain internal and MUST NOT be exposed to applications.
• MUST Writes to a database MUST be serialized to a single writer, fenced by the commit log's CAS token.
• MUST GetPage@LSN MUST return the page version at or before the requested LSN (snapshot-consistent reads).
• MUST NOT Caching MUST NOT be used to hide or shortcut commit latency; an in-memory buffer MUST NOT be the basis of a commit ack.
• SHOULD The embedded hot path SHOULD resolve cache-hit reads as in-process function calls without crossing a socket.
• SHOULD Server mode SHOULD sit behind a transaction-mode pooler to absorb serverless connection bursts.
• SHOULD The write path SHOULD support group commit to amortize the durable-append round-trip across concurrent committers.
• MAY A deployment MAY use LocalFileStorage for fully offline/edge operation with zero network on the data path.
• MAY A deployment MAY ship the engine as a NAPI addon to share one package across Bun and Node.

Acceptance criteria / definition of done

The architecture as a whole is "done" when:

• MUST One binary embeds via FFI and serves via pgwire, with the backend chosen only by connection string (demonstrated end-to-end on both file:// and s3://).
• MUST Crash-injection at both adversarial points (after CAS-append/before ack; after ack/before materialization) shows every acked commit present and no torn state on restart — Experiment 4 passes unconditionally.
• MUST A branch is created as a new LSN pointer over shared immutable layers, with measured marginal storage ≈ 0 until divergence.
• SHOULD Single-commit latency floor (Experiment 1) and the group-commit throughput curve (Experiment 2) are characterized with p50/p99/p999, same-region and cross-region.
• SHOULD Scale-to-zero round-trip (idle teardown → cold start → warm) is measured (Experiment 5) and feeds idle_timeout tuning.
• SHOULD The contention wall (Experiment 3) confirms N separate databases scale as N independent write lanes.

Dependencies / existing pieces to start from

Build outputs that realize the slots: libengine.a / .so / .dylib (embeddable library), engine.h (the stable C ABI boundary), and engine-server (the listener wrapper). Language is Rust — FFI-friendly, links into anything, emits both cdylib and staticlib.

Open questions & risks

• Maturity. Object-storage-native OLTP is the active frontier (SlateDB, S3-CAS log designs, the libSQL rewrite). Fewer battle-tested guarantees than coupled Postgres; expect the page-store and commit-log internals to move.
• Write firehose. Sustained high-rate writes pay the durable-append round-trip per group. Read-heavy/bursty workloads barely notice; sustained writers must be benchmarked explicitly (Exp 1/2) before commitment.
• WASM fork. A WASM target (Cloudflare Workers) is a port, not a recompile: no native fs, single-threaded isolate, storage via the Worker's binding rather than a raw S3 SDK. The engine-form decision precedes everything else for that substrate — see 11.
• Cold-start tail under thundering herd. Many simultaneous cold starts (e.g. 1,000 at once) may saturate spin-up; the per-cold-start measurement (Exp 5) needs an N-concurrent extension.
• Storage-trait stability. The trait is the one fixed point the whole system depends on; getting get_page/append_wal semantics and versioning right early is the highest-leverage decision. Detailed in 03.

Related specifications