Skip to content

Design decisions

The choices that shape everything else in the engine, recorded in a compact decision-record format: the context at the time, the decision, and the consequences that followed - including the trade-offs accepted, which are as much a part of the record as the benefits.

These records are point-in-time. Each states what was decided and why; the internals references describe how the resulting subsystems work today, and take precedence wherever detail has evolved.

# Decision
001 Arrow is the data plane, not an integration
002 Embedded-first: the engine is a library before it is a cluster
003 State is an open dataset, not an internal format
004 Jobs deploy as compiled plugins
005 The SQL optimizer is hand-rolled, not adopted
006 Incidents replay deterministically from captured input
007 State directories are namespaced by topology generation
008 Rescale one operator at a barrier, without stopping the job
009 One declaration describes a type everywhere
010 The extension contract is a declared surface, checked completely
011 The public API is tiered, and 1.x promises source compatibility on the Stable tier
012 The exactly-once protocol is a machine-checked specification, and the engine's traces are validated against it