Storage backends — overview
Twill DB's durable storage is pluggable, and you select it with nothing but the scheme of the connection string. The engine, the SQL, the C ABI, and your application code are identical for every backend — only the URL changes. This page is the shared model; each backend then has its own page.
At a glance
| Scheme | Backend | Durability floor | Network | Scale-to-zero | Page |
|---|---|---|---|---|---|
file:// | LocalFileStorage | Local disk (CRC-checked WAL) | None | n/a (local) | file:// |
s3:// | ObjectStorage | S3 / S3-compatible (MinIO) | Object API | Yes | s3:// |
r2:// | ObjectStorage | Cloudflare R2 | Object API (zero egress) | Yes | r2:// |
gs:// | ObjectStorage | Google Cloud Storage | Object API | Yes | gs:// |
The promise
Move from a laptop file to global object storage by changing one string. There is no migration of application logic, no second engine, no different SQL. Durability is WAL-centric in every backend: a commit returns only after its records are durable.
What's shared across all backends
These hold no matter which scheme you choose — the backend pages only cover what differs.
- Selection is the URL.
open_storagedispatches on the scheme; unknown schemes are rejected, never silently defaulted. - Durability is WAL-centric.
append_walreturns the commit LSN only after the records are durable — never an ack from a buffer. - MVCC read floor. A read at an LSN returns the greatest version at or below it; readers never block writers.
- Branching & the lease are seam concerns. Copy-on-write branching and the durable single-writer lease are implemented at the storage seam, so they work the same way across backends (a sibling file for
file://, a child key-prefix for object stores).
Credentials never go in the URL
Secrets live in the environment
Connection strings in code, tests, and examples should use file:// only. Real bucket credentials are environment configuration (provider variables), never the connection string or the repo. A secret-scanning hook and CI check exist to catch slips — treat a hit as a stop. Each object-storage page lists the exact variables for that provider.
Choose a backend
file:// — local
Zero network, fastest hot path. Dev, single-node, offline/edge, tests.
s3:// — S3 / MinIO
Disaggregated on AWS S3 or any S3-compatible store. Scale-to-zero.
r2:// — Cloudflare
Disaggregated with zero egress — best for cold-read-heavy workloads.
gs:// — Google Cloud
Disaggregated on Google Cloud Storage.