Twill Bench CLI

Purpose & relationship to the validation plan

Unlike a generic load generator (k6, Locust, Vegeta), Twill Bench understands the engine: it can attribute latency to the commit round-trip, watch a database scale to zero and back, and assert ACID invariants over the result set. It exists to answer three questions on every run — is the database fast, is the data still correct under stress, and how efficiently did the serverless architecture use compute and storage?

Goals & design principles

• MUST capture all latency as a distribution (p50/p90/p95/p99/p999 via an HDR-style histogram); mean-only reporting is prohibited (inherited from spec 09).
• MUST validate data correctness alongside performance — a fast run that loses an acked write or an update is a failure, not a fast success.
• MUST emit both human-friendly terminal output and a machine-readable JSON record for archiving, plotting, and CI gating.
• SHOULD be zero-config for the common scenarios and reproducible (pinned engine + storage SHA, region, instance type, seed — per spec 09).
• SHOULD be safe to run against production for read-only scenarios, and refuse to mutate unless explicitly opted in.
• SHOULD be extensible via benchmark profiles without recompiling the driver.
• MUST NOT pull heavyweight or non-auditable dependencies into the engine core to satisfy a reporting feature; CLI-only concerns (YAML profiles, Prometheus export) stay in the twill-bench crate, feature-gated, and never cross the storage seam or the thread-free engine boundary.

Command structure

The brief proposes a twill bench <scenario> [options] surface. The binary is invoked as twill-bench <command> --url … [flags]; both are sub-command driven. The request-mix scenarios, the correctness profiles, the lifecycle scenarios, the custom profile loader, and compare below all ship; what remains is the optional Prometheus export, not a re-architected entry point. (There is no separate twilldb CLI — twill-bench is the driver's own binary, as the brief intends.)

twill-bench read-heavy --url file://./b.db      # shipped
twill-bench write-heavy --url file://./b.db     # shipped
twill-bench mixed-oltp --url file://./b.db      # shipped
twill-bench counter --url file://./b.db         # correctness profile (shipped)
twill-bench bank-transfer --url file://./b.db   # correctness profile (shipped)
twill-bench inventory --url file://./b.db       # correctness profile (shipped)
twill-bench document-editing --url file://./b.db # correctness profile (shipped)
twill-bench compare --baseline base.json --candidate cand.json   # shipped
twill-bench burst --url file://./b.db           # shipped (issue #79 — closed-loop rate driver)
twill-bench scale-to-zero --url file://./b.db   # shipped (spec 09 Exp 5 cold read)
twill-bench custom --profile workload.yaml      # shipped (issue #81 — feature-gated YAML loader)
twill-bench exp1 --url s3://b/db --max-stall-ratio 50   # V-1 stall gate + region/cold-connection variants
twill-bench exp2-sweep --url s3://b/db --sweep-max 64    # V-2 group-commit-window sweep + plateau knee
twill-bench exp3-shard --url s3://b/db --databases 16    # V-3 N-database sharding + cross-DB CAS verdict
twill-bench herd --url file://./b.db --concurrency 64    # V-4 thundering-herd cold-start knee
twill-bench boundary --dir bench-archive --gate         # V-5 W1/W2 boundary tables from archived records

Benchmark scenarios

Scenarios are named workload shapes. The three implemented experiments map onto the request-mix scenarios; the lifecycle scenarios (burst, scale-to-zero, long-run) are new and depend on the controller being in the loop.

The custom profile schema (issue #81)

The named scenarios make the op ratios data; custom makes the whole shape data — duration, connection count, working-set size, seed, and the op mix — so a workload we don't want to name in the binary becomes a file, not code. The deliberate scope line: a profile expresses exactly the mix-driver vocabulary (the four op kinds' weights plus the run knobs). There is no custom SQL in v1 — the value over a hypothetical --mix flag is making the whole shape data, not turning the profile into a mini query language. Anything outside the documented field set is a hard parse error, not a silent ignore.

# workload.yaml — a custom twill-bench workload profile
label: saas-oltp          # free-form tag recorded in the output (optional)
url: file:///tmp/bench.db # backend URL (optional; a CLI --url overrides it)
duration_ms: 5000         # timed measurement window (required, > 0)
warmup_ms: 500            # discarded warm-up window (default 200)
connections: 8            # concurrent writers (required, >= 1)
rows: 10000               # pre-seeded working-set size (default 1000)
seed: 1                   # fixed PRNG seed → reproducible op stream (optional)
mix:                      # per-op weights (arbitrary positive ints; ≥1 non-zero)
  select: 70
  insert: 20
  update: 8
  delete: 2

The numeric knobs carry the same millisecond / count vocabulary as the CLI flags, so a profile reads like the flags it replaces. The run is driven through the same transport-agnostic mix driver as the named scenarios (so custom runs on both the embedded and pgwire transports), and the report adds a mix section — the configured weights beside the realized op counts — so a run proves the driven shape tracked the requested ratios.

Metrics

The brief groups metrics into five families. The query family ships today; the compute and scheduler families are the new, lifecycle-dependent surface and must be sourced from the controller/server, never invented inside the engine.

Query metrics IMPLEMENTED

Total / successful / failed requests, requests/sec, and the latency distribution (median, p90, p95, p99, max). The driver records per-op latency into an HDR-style histogram and reports p50/p90/p95/p99/p999 (plus min/max/mean) alongside an ok / retried-conflict / failure split — both in the human summary and the JSON record.

Storage metrics PARTIAL

Commit latency is measured today (it is the experiment). Snapshot/WAL read & write counts, storage-fetch latency, and cache hit/miss ratio require the engine/storage to expose counters through a read-only stats surface — additive, but a new seam-respecting introspection path (no backend internals leak into the Storage trait). That surface is now designed: see Observability design decision (#53), Decision 1 (Storage::stats() → StorageStats).

Compute & scheduler metrics DESIGN RESOLVED

Cold/warm starts, average start time, worker-reuse rate, compute active/idle duration, scale-to-zero events, peak workers; queue depth, scheduling delay, allocation/placement time. These come from twill-controller and the server, and overlap directly with a future observability/OTLP export. Twill Bench consumes them; it does not generate them — by pulling a ControllerStats snapshot, never scraping logs or pushing from the core (Observability design decision (#53), Decision 2).

Network metrics NEEDS DESIGN

Client RTT, TLS handshake, request/response transfer — meaningful only on the pgwire transport; measured client-side in the bench driver.

Observability design decision (#53): metric source & read-only stats surface

The lifecycle scenarios and the serverless-efficiency report need numbers the bench cannot make up — cold starts, worker reuse, scale-to-zero events, storage reads, cache hit/miss, compute active/idle time. Issue #53 asks three questions before any of that can be built: what read-only surface exposes those counters without leaking backend internals across the seam; how does the bench observe controller lifecycle events; and which latency segments are observable today versus blocked. This section resolves all three and settles the metric vocabulary once, so the future observability/OTLP exporter emits the same names the bench reports.

Decision 1 — the read-only stats surface (per tier, pulled, seam-safe)

Four snapshot accessors, each owned by the tier that already tracks the signal, each a cheap relaxed-atomic load returning a Copy value struct (no allocation, safe to poll on a hot loop):

Decision 2 — how the bench observes controller lifecycle events

Pull the controller's snapshot; do not scrape logs and do not require a collector. Three candidate mechanisms were weighed:

• REJECTED log scrape — couples the bench to a log format, needs a parser, races with rotation, and carries no histograms. Brittle for a number that gates a placement decision.
• REJECTED (for the bench) push / emit from the controller — that is the future OTLP exporter, and it would force the bench to stand up a collector to read its own run. The exporter reads the same snapshot; it does not replace it.
• CHOSEN pull a snapshot — Controller::stats() in-process for an embedded/controller-driven run; SHOW twill.stats over pgwire for a deployed server. The bench samples it at start and end of a scenario (and on an interval for long-run), and the per-scenario delta becomes the report. Cumulative counters make deltas exact; gauges (peak_workers, committed_lsn) are read as-of.

This keeps lifecycle generation where it belongs — the controller and server own the events; the engine core stays thread-free and metric-free — and gives the bench a stable, typed, histogram-carrying read with no new runtime dependency.

Decision 3 — latency-breakdown attribution: observable now vs. blocked

The full request path is receive → auth → scheduler/admission → compute spin-up → metadata load → storage read → execution → commit → response. What each segment costs, and where the number comes from:

So the breakdown ships in two steps: a client-side + commit breakdown is available now; the storage / scheduler / spin-up segments unlock together the moment Decision 1's accessors land, and the spin-up/metadata split is reported only on the cold path the scale-to-zero scenario drives.

The settled metric vocabulary (shared with the future OTLP exporter)

One namespaced signal set, defined here once. The bench reports these names in its JSON record; the live exporter emits the same names. Families map onto the snapshot accessors above.

The efficiency family is pure arithmetic over the others — no new instrumentation, only the snapshot deltas a controller-driven run already yields. That is the report issue #53 asks for, and it falls out of Decisions 1–2 for free.

Build order this unblocks

1. 1 ✓ The stats surface is landed — the three in-process accessors (Storage::stats(), additive, STORAGE_TRAIT_VERSION bumped 2→3, C1–C8 green; Database::stats(), folding the backend snapshot pulled through the seam; Controller::stats()) plus the SHOW twill.stats pgwire surface (the server intercepts it and answers from the live engine + storage snapshot, returning (metric, value) rows under the settled names). Each has a test that moves the counters, including an end-to-end pull over the wire. The controller's compute/scheduler rows fold into SHOW twill.stats once a controller drives the deployment (step 2).
2. 2 ✓ scale-to-zero (Exp 5) cold-read scenario is landed — the twill-bench scale-to-zero subcommand drives query → idle past the controller's reaper → query for --cycles cold-boot samples (--idle-ms sets the reaper window; spec-09 Exp 5 uses a long one on a real deployment), pulls the ControllerStats snapshot at the run boundaries, and reports the cold-boot percentile distribution against a controller-driven deployment. The compute_active/idle_us, admission_wait_us, and lease_renew_total signals landed in ControllerStats alongside it. Controller-driven and in-process (the --server/pgwire form is rejected — a deployed server runs its own controller); a test drives the full cold path and asserts no durable loss across teardowns.
3. 3 ✓ The serverless-efficiency report is landed — the scale-to-zero run now emits the full derived family from the pulled snapshot deltas: utilization = active / (active + idle), compute_seconds_per_query, storage_reads_per_query (the backend page-read delta pulled from EngineStats.storage per cold read — 0 on the in-memory file:// path, nonzero on an object store with a cold cache), scale_to_zero_count, and avg_worker_lifetime = total resident time / cold starts, alongside the twill_worker_reuse_ratio and twill_storage_page_reads_total source signals. Reported in both the human summary and the JSON record under the settled names; the derived arithmetic is pinned by a unit test (exact + zero-safe) and the full cold path by the scenario integration test.
4. 4 ✓ The long-run soak scenario is landed (issue #80) — the interval sampler this decision anticipated (“…and on an interval for long-run”) pulls a stats() snapshot plus a Linux /proc/self resource probe (RSS, open fds, threads — degrading to zeros where /proc is unavailable, never a crash) every --sample-interval-ms into an in-memory time series, decoupled from a steady-state read load over a pre-seeded working set. A least-squares slope over the post-warm-up window fits memory, fds, and p99; a metric is flagged as a leak/drift only when its projected growth crosses both the relative --drift-threshold (default 10%) and a per-metric absolute floor, so short-run jitter never trips it. The JSON record gains a soak section (per-metric first/last/slope/peak/growth + a PASS/FAIL drift_pass verdict); a detected leak/drift fails the run with the correctness exit code (2). Embedded-only (it samples this process's own resources, like scale-to-zero). Unit-tested on the sampler cadence, the /proc parser, and synthetic rising-vs-flat series; an integration test proves a seeded monotonic growth exits 2 while a flat control passes.

Data-correctness validation

Performance is only valid if correctness is preserved. Twill Bench asserts, over the workload it just drove, that none of the classic anomalies occurred.

• MUST detect: missing rows, duplicate rows, lost updates, dirty reads, non-repeatable reads, phantom reads, serializable violations, and transaction-consistency breaks.
• MUST exit non-zero (code 2) when any invariant is violated, regardless of latency.

All four named, result-checking workload profiles now ship (counter, bank-transfer, inventory, document-editing): each drives a contended fixed-work load, then reads the durable state back and asserts an invariant, setting a correctness verdict that the CLI turns into exit code 2 on violation. They build on what exp3 proved informally — first-committer-wins conflicts are counted and retried (over pgwire the conflict surfaces as SQLSTATE 40001), and the conformance and group-commit suites prove durability and isolation at the crate level — and lift it into an explicit pass/fail assertion over the result set. Because all four are fixed-work (each writer does exactly --ops operations), the expected result is known exactly, so the assertion is a precise equality rather than a heuristic. A test-only --inject-fault lost-update hook deliberately drops one acked write so the negative test can prove the checker itself bites — a seeded violation must exit 2, not pass.

Workload profiles

Reporting

Latency breakdown DESIGN RESOLVED

Rather than a single total, attribute time across the request path — receive, auth, scheduler, compute spin-up, metadata load, storage read, execution, commit, response. The spin-up / metadata / storage segments require the same controller + storage introspection as the compute/storage metrics above; until those land, the breakdown is partial (client-side + commit only). The per-segment attribution and what is observable today versus unlocked by the stats surface is settled in Observability design decision (#53), Decision 3.

Serverless-efficiency report SHIPPED

Unique to Twill DB: compute-active vs idle time, scale-to-zero count, average worker lifetime, compute-seconds/query, storage-reads/query, average compute utilization. This reframes a run around operational cost, not just latency — the most differentiated piece of the brief, and entirely controller-sourced. It is pure arithmetic over the snapshot deltas a controller-driven run yields; the signal set is fixed in the settled metric vocabulary, and the build order it falls out of is listed there. The scale-to-zero scenario emits the full family (human + JSON) under those names: utilization, compute_seconds_per_query, storage_reads_per_query, scale_to_zero_count, and avg_worker_lifetime, sourced from the pulled ControllerStats + EngineStats deltas.

Release comparison IMPLEMENTED

twill-bench compare --baseline base.json --candidate cand.json [--threshold 0.10] diffs two archived JSON records (throughput, p50/p99/p999) into a PASS/regression verdict — a regression (throughput down, or a tail latency up, beyond the threshold) exits 1. It is pure post-processing over the records the driver already emits (each carries the git SHA and the full percentile set), so it needs neither the engine nor a transport, and is naturally CI-friendly. Cold-start and memory deltas join the diff once those metrics are sourced from the controller.

Validation campaign — sweeps, gates & boundary tables (issue #91) SHIPPED

The single-lever experiments (exp1/exp2/exp3) became the falsifiable boundary the platform routes by. Each sweep drives the public commit path and hands the resulting spec-09 curve to a pure, unit-tested analysis layer (crates/bench/src/analysis.rs) for the verdict; the gates are report-only by default and map to a non-zero exit under --gate for CI.

Output formats

• IMPLEMENTED — terminal summary + one-line JSON record (experiment, transport, backend, SHA, throughput, percentiles, and the correctness verdict for profiles).
• IMPLEMENTED — --json emits only the one-line JSON record on stdout (for scripting / CI), suppressing the human summary.
• NEEDS DESIGN — --prometheus export for Grafana/CI. Keep the exporter dependency CLI-only and feature-gated; it must never reach the engine.

Exit codes

IMPLEMENTED — the binary now follows this table exactly: 0 success, 1 benchmark failed (or a compare regression), 2 a violated correctness invariant, 3 a configuration/usage error, 4 a connection failure. The mapping is pinned by the scenario tests (crates/bench/tests/scenarios.rs).

Verification summary — brief vs. what ships

Where each brief capability stands against the current crates/bench driver and the project's architecture rules.

Constraints & open questions

• MUST NOT let any reporting feature (YAML, Prometheus, OTLP) add a dependency to crates/engine or move a backend concept into the Storage trait — these live in the bench/server/controller crates only.
• MUST NOT generate lifecycle/compute metrics inside the thread-free engine; consume them from twill-controller and the server.
• RESOLVED — yes, the compute/efficiency metrics share one signal vocabulary with the future observability/OTLP exporter. The set is settled in Observability design decision (#53): each tier exposes a pulled stats() snapshot, the bench reports those names, and the live exporter emits the same names from the same snapshots.
• SHOULD resolve: what is the safe-against-production guard? Read-only scenarios run anywhere; any mutating scenario requires an explicit opt-in flag and a non-production URL by default.
• MAY defer (future enhancements per the brief): multi-region runs, fault injection, network-latency simulation, storage-backend comparison, flamegraphs, continuous mode, a historical benchmark DB, HTML reports, and an interactive TUI.

Related specifications