Design decisions¶
The choices that shape everything else in the engine, recorded in a compact decision-record format: the context at the time, the decision, and the consequences that followed - including the trade-offs accepted, which are as much a part of the record as the benefits.
These records are point-in-time. Each states what was decided and why; the internals references describe how the resulting subsystems work today, and take precedence wherever detail has evolved.
| # | Decision |
|---|---|
| 001 | Arrow is the data plane, not an integration |
| 002 | Embedded-first: the engine is a library before it is a cluster |
| 003 | State is an open dataset, not an internal format |
| 004 | Jobs deploy as compiled plugins |
| 005 | The SQL optimizer is hand-rolled, not adopted |
| 006 | Incidents replay deterministically from captured input |
| 007 | State directories are namespaced by topology generation |
| 008 | Rescale one operator at a barrier, without stopping the job |
| 009 | One declaration describes a type everywhere |
| 010 | The extension contract is a declared surface, checked completely |
| 011 | The public API is tiered, and 1.x promises source compatibility on the Stable tier |
| 012 | The exactly-once protocol is a machine-checked specification, and the engine's traces are validated against it |