Scale-to-zero & lifecycle

The idea

Scale-to-zero rests on one invariant: compute is stateless — all durable state lives in object storage behind the Storage seam. A warm instance is just a cache plus a CPU; destroying it loses nothing durable, so the controller is free to stop the engine whenever a database is idle and reconstruct an equivalent instance on the next connection.

The lifecycle controller

The twill-controller crate is a thin, stateless supervisor. It composes the engine's existing primitives rather than reimplementing them: opening a Database acquires the writer fence and replays the WAL — that is the cache warm — and dropping it releases the fence. On top of that the controller adds the state machine, an idle-timeout reaper, a lease heartbeat, and thundering-herd handling.

  Cold ──start──▶ Warming ──opened──▶ Active ──no work──▶ Idle ──timeout──▶ Stopping
   ▲                 │  (open fails)       ▲     │                              │
   │                 ▼                     └─────┘  (new connection re-activates)│
   └─────────────────┴───────────────────────────────────────────────────────────┘
                                  (Stopping always lands back in Cold)

Cold start

A cold start is exactly: process start + cache warm + fence acquire + WAL replay. In the controller these collapse into one step — Database::open(url) acquires the single-writer fence and replays the durable WAL, which is what warms the instance. The dominant tail term is cache warm: the first reads after a cold start miss the local cache and fall through to object storage, so a large random working set is the worst case.

Idle reaper

A background reaper runs every reap_interval. For each warm instance with no active leases it moves Active → Idle, and once an instance has sat Idle past idle_timeout (and keep_warm is off) it tears the handle down — Stopping → Cold — releasing the fence. The lifecycle and heartbeat threads live in the controller, deliberately not in the embedded engine core, which stays thread-free so embedders own their own scheduling.

Single-writer lease heartbeat

The writer lease is durable and fenced by a monotonic CAS epoch. The reaper heartbeats it for every warm instance (Active or Idle) by calling Database::renew_lease(); if a renewal fails — the instance has been fenced by a newer writer — the controller treats it as fatal, drops the handle, and returns the instance to Cold. Split-brain is impossible by construction: only one CAS epoch wins each append, so a stalled writer's appends are simply rejected.

Keep-warm & thundering-herd admission

Two mechanisms bound the cold-start tail under load:

• MUST dedupe: N concurrent start calls for one cold database trigger exactly one Warming transition; the rest wait on that single warm rather than each spawning a process.
• SHOULD admit under a cap: a bounded warm-admission semaphore (max_concurrent_warms) limits how many distinct databases warm at once, so a herd of many cold databases cannot saturate CPU or the object store's request budget.
• MAY keep warm: with keep_warm on, idle instances stay resident past idle_timeout to cut post-idle latency for latency-critical, low-traffic databases.

The cost trade-off

Scale-to-zero trades a cold-start tail for zero idle compute. The single knob with the most leverage is idle_timeout:

Too short an idle_timeout pays the cold-start tax repeatedly on bursty-but-recurring traffic; too long wastes compute and defeats scale-to-zero. Tune it against each database's inter-arrival distribution, and reach for keep_warm only where the cold-start tail actually hurts.

Related