Client Runtimes & Language SDKs

Purpose & scope

This spec expands the “adding a client is additive” claim of 18 — Scaffolding CLI into a buildable, normative design for clients beyond Bun. It defines, for each runtime, which seam it binds, the binding mechanism, the package it ships through, and how its public surface stays identical to @twilldb/bun. It owns none of the engine, the ABI, the wire protocol, or the storage seam — those are 02 / 07 / 03; this page binds to them and MUST NOT redefine them.

The client matrix

Two binding mechanisms, chosen per runtime by what that runtime can do — not by anything in the engine:

Responsibilities & non-goals

Responsibilities

• MUST bind one of the two existing seams unchanged: the C ABI via the runtime's FFI for embedded, or pgwire via the runtime's Postgres driver for server.
• MUST preserve the ABI ownership contract at every binding: marshal NUL-terminated const char* arguments, copy out borrowed returns before the owning object advances or is freed, and free every engine-owned resource exactly once (08's ownership table is normative).
• MUST map engine status codes + engine_last_error into the runtime's idiomatic typed error, distinguishing retryable (CONFLICT / transient STORAGE) from terminal.
• MUST verify the embedded ENGINE_ABI_VERSION at load time and fail fast on mismatch rather than calling a stale symbol.
• SHOULD keep the public surface (open / exec / query / prepare / transaction / branch / close) one-to-one with @twilldb/bun, so application code is portable across runtimes.

Non-goals

• MUST NOT define or alter the C ABI (02), the wire protocol or pooler (07), or the storage seam (03).
• MUST NOT implement SQL parsing, MVCC, WAL, caching, or storage in any binding — each is a thin, faithful shim.
• MUST NOT require an engine rebuild, an ABI bump, or a storage-seam change to add a runtime. Adding a client is purely additive.

Convergence: one surface, many loaders

The Bun wrapper (08) already separates a loader (the FFI symbol table) from an ergonomic surface (the typed Database / Statement / EngineError). A new embedded client reuses the surface verbatim and supplies only a new loader:

  application code  (open / exec / query / prepare / transaction / branch)
        │  identical across runtimes
        ▼
  ergonomic surface  ── Database · Statement · EngineError · param encoding
        │
        ▼
  loader  ─────────── bun:ffi   (Bun)
                      koffi      (Node + Next.js/Astro/Nuxt/…)
                      FFI::cdef  (PHP + Laravel/CodeIgniter)
                      ctypes…    (Python/Ruby/Go — roadmap)
        │
        ▼
  libengine  (engine.h)  ──  storage seam  ──  file:// | s3:// | r2:// | gs://

• SHOULD mirror the parameter encoding exactly — the one-character type tag (i/f/s/b/n/v) from 02 — so a query written for one runtime binds identically on another.
• SHOULD expose deterministic disposal in the runtime's idiom (JS using/close(); PHP explicit close()/finalize() on long-lived workers), and never report a commit before engine_commit returns durable.

Node — koffi over the C ABI

Node has no bun:ffi, but koffi is a prebuilt, ABI-stable FFI library that loads the same libengine with no compile step. @twilldb/node is the Bun wrapper with a koffi loader: the public surface is byte-for-byte the Bun surface, so an app moves between them by changing the import.

import { open } from "@twilldb/node";        // same API as @twilldb/bun

const db = open("file://./local.db");          // backend chosen by URL scheme
db.exec("CREATE TABLE notes (id INTEGER PRIMARY KEY, body TEXT)");
db.query("INSERT INTO notes (id, body) VALUES (?, ?)", [1, "hello"]);
const rows = db.query("SELECT * FROM notes");
db.close();

Frameworks (Next.js, Astro, Nuxt, Remix, SvelteKit, Vite SSR)

A Node-based framework renders in a Node server process; the engine embeds in that same process. The integration rule is the same everywhere:

• MUST open one Database per server process and reuse it across requests (a module-level singleton). The engine is a single writer per database; opening per-request thrashes the WAL.
• MUST keep the database on the Node.js runtime, not the Edge runtime — FFI is unavailable on Edge. In Next.js, mark the route export const runtime = "nodejs".
• SHOULD guard the singleton on globalThis so dev hot-reload does not double-open.

See clients/node/examples/: nextjs-route.ts (App Router handler) and astro-endpoint.ts (the same shape for any Vite-SSR framework).

PHP — the FFI extension over the C ABI

PHP 8 ships a built-in FFI extension, so PHP embeds the engine directly — no native extension to compile. twilldb/twilldb declares the engine.h subset with FFI::cdef and wraps it in the same ergonomic surface.

use Twill\Database;

$db = Twill\Database::open('file://./local.db');
$db->exec('CREATE TABLE notes (id INTEGER PRIMARY KEY, body TEXT)');
$db->query('INSERT INTO notes (id, body) VALUES (?, ?)', [1, 'hello']);
$rows = $db->query('SELECT * FROM notes');
$db->close();

• MUST require PHP ≥ 8.1 with FFI enabled (ffi.enable=1). PHP's FFI decodes a const char* return into a copied PHP string (and a NULL pointer — a SQL NULL cell — into null), satisfying the “copy out before free” rule for read paths automatically.
• MUST throw Twill\EngineError (carrying the numeric EngineStatus and a retryable flag) on every non-OK status.

Frameworks (Laravel, Symfony, CodeIgniter) — two paths

Embedded

Open one Twill\Database in a service/singleton and reuse it on a long-lived worker (FrankenPHP, RoadRunner, Swoole). Under classic per-request php-fpm, the open + WAL-replay cost is paid per request, so the embedded path favours a persistent worker.

Server (PDO)

Run engine-server and point the framework's pgsql connection at it (host=127.0.0.1 port=5433 sslmode=disable). No FFI, no native library in the PHP process — Laravel's config/database.php pgsql connection, or CodeIgniter's Postgre driver, just works. See clients/php/examples/server-pdo.php.

Polyglot via server mode (today) & embedded (roadmap)

For any other language, the server path already works: connect to engine-server with the native Postgres driver (psycopg, Ruby pg, Go pgx, JDBC, …). The embedded path for each is the same additive pattern — a loader over engine.h behind the shared surface — and is sequenced on the roadmap, not blocked by anything in the engine:

Packaging & distribution

The piece that makes a polyglot matrix tractable is building libengine once per platform/arch and fanning the same artifacts out to every ecosystem (18). No client rebuilds the engine.

• MUST resolve the engine binary for the host platform+arch, and fail with an actionable error when none matches (naming the missing package and the TWILLDB_ENGINE_PATH override).
• SHOULD reuse the existing @twilldb/engine-{os}-{arch} per-platform npm packages for both Bun and Node (Node lists them as the same optionalDependencies, resolved by os/cpu).
• SHOULD distribute through each ecosystem's own registry — npm for Bun/Node, Composer/Packagist for PHP, PyPI/RubyGems/crates.io later — none of which has a maturity gate.

  libengine.{so,dylib,dll}   (built once per platform/arch)
        │
        ├─ npm:       @twilldb/engine-{os}-{arch}   ← @twilldb/bun, @twilldb/node
        ├─ Composer:  twilldb/twilldb loads the matching binary via FFI
        └─ PyPI/Gems/crates.io                      ← roadmap, same binaries

Scaffolding integration (CLI)

The single extension point in 18's scaffolder is the template set behind --client. Node and PHP are now generated clients, not roadmap placeholders:

twilldb new web --client node        # Node embedded starter (@twilldb/node)
twilldb new api --client php         # PHP embedded starter (twilldb/twilldb)
twilldb new search --client node --vector   # + an HNSW vector starter

• MUST emit a runnable embedded starter per client — package.json/app.ts for Node, composer.json/index.php for PHP — selecting the backend purely by the connection string it writes.
• MUST print the correct install + run commands for the chosen client (npm install/npm start; composer install/composer start).
• SHOULD keep a roadmap-aware message for clients not yet generated (currently rust).

Failure modes & edge cases

Acceptance criteria / definition of done

• MUST run the full embedded suite (exec / parameterized query / prepared statement / transaction commit+rollback / SQL-NULL / typed error / copy-on-write branch / lastLsn) on Node over koffi and on PHP over the FFI extension, against a file:// database. DONE — clients/node/test/embedded.test.ts, clients/php/test/embedded_test.php.
• MUST keep the public surface identical to @twilldb/bun so application code is portable. DONE — Node mirrors the Bun Database/Statement API; PHP mirrors it idiomatically.
• MUST generate runnable Node and PHP starters from twilldb new --client, with correct per-client install/run steps. DONE — crates/cli templates + tests.
• SHOULD open the same code against an s3:// URL with no source change (storage chosen by scheme only). DONE by construction — no loader branches on backend.
• SHOULD document framework integration for Next.js / Astro / Vite (Node) and Laravel / CodeIgniter (PHP), including the Node.js-runtime and singleton rules.

Open questions & risks

• NAPI vs koffi for Node — decided (EX-2 / #101): stay FFI-only. koffi already gives cross-runtime Node support over the same C ABI with no compile step; a napi-rs addon is added only on demand if a measured per-call overhead on the embedded micro-loop (09) justifies it. See the decision section above.
• PHP worker model. The embedded path shines on persistent workers (FrankenPHP/RoadRunner/Swoole) and is weakest under per-request php-fpm. Should the PHP package detect the SAPI and nudge toward server mode there?
• Published .ts vs prebuilt JS. Shipping .ts (like @twilldb/bun) keeps one source but pins a minimum Node; a prebuilt dist/ widens reach at the cost of a build step. Pick per ecosystem norms.
• Surface drift. As the engine grows (streaming cursors, new capabilities), all loaders must move together. A shared conformance suite run across runtimes guards this.

Decision — NAPI vs FFI for a single Bun + Node package (EX-2 / #101)

Decision: stay FFI-only. Ship Bun via bun:ffi and Node via koffi over the same frozen C ABI; add a NAPI addon only on demand. This is a MAY decision/spike, not committed build work, and it does not touch the C ABI (engine.h) — both bindings consume the same FFI boundary, so there is no engine-core or storage change either way.

Why FFI-only wins now

The premise that made NAPI attractive — "cross-runtime needs a compiled addon" — no longer holds: cross-runtime is already achieved without NAPI. @twilldb/node binds the same libengine through koffi, mirroring @twilldb/bun's bun:ffi surface one-for-one, so open / query / transaction have identical semantics in both runtimes against one prebuilt platform binary — the exact "if cross-runtime is chosen, prove one triple identical in Bun and Node" exit criterion, met with zero addon code.

The on-demand trigger

Revisit NAPI only if a concrete need appears: (a) a measured per-call overhead on the embedded micro-loop (09) that FFI marshalling cannot meet for a real workload, or (b) a target runtime that cannot load a raw cdylib through an FFI library but does host Node-API. Until one of those is real, a NAPI addon is per-platform compile + distribute + maintenance cost buying reach we already have. The shared cross-runtime conformance suite (run against Bun, Node, and PHP) is what guards surface drift — not a second binding mechanism.

Related specifications