Protocol and format compatibility

Every place where two clink builds, or a clink build and its own older artefacts, must agree on bytes. Each domain below evolves independently and carries its own version and its own policy; there is deliberately no single "clink format version", because the domains change for unrelated reasons and a shared number would force unrelated bumps.

Two global rules hold everywhere:

• Incompatible input fails before side effects. A frame, sidecar, snapshot or plugin that this build cannot honour is refused where it is read, before any state mutation, sink commit, or job admission it would have caused.
• Additions are compatible; changes are versioned. Every encoding here has a defined way to add information that old readers skip (a tail field, an unknown key, a new frame kind); anything that changes the meaning of existing bytes needs its domain's version bumped, and the old version either handled or refused by name.

The frozen-bytes fixtures in tests/fixtures/ (exercised by tests/test_format_fixtures.cpp and, for the SQL catalog, tests/test_sql_catalog_fixtures.cpp) pin the persistent and wire encodings from outside the codebase: the fixture files were written by one build and must stay readable by every later one, so an encode/decode pair cannot silently co-evolve past its own history.

This page is about bytes. The API-level promise (which headers, C symbols and SQL statements a 1.x release holds stable) is the published Compatibility page and design record docs/design/011-public-api-tiers.md.

Domain inventory

Domain	Current	Minimum compatible	Where enforced
Cluster control protocol	3	1	include/clink/cluster/protocol.hpp (kClusterProtocolVersion), three handshake sites
Data plane (operator wire frames)	unversioned by design	n/a	include/clink/runtime/network/wire.hpp
Checkpoint metadata sidecar	1	1	include/clink/state/checkpoint_integrity.hpp (kCheckpointMetaVersion)
Checkpoint directory layout and COMPLETED markers	unversioned; marker keys additive-only	every layout since the generation directories (v0.7.0) restores	include/clink/cluster/state_location.hpp (resolve_restore_source), tests/test_state_location.cpp
State snapshots and savepoints	1 (clink.format_version)	1 (absence reads as 1)	docs/internals/state-snapshot-format.md
Job plan (JobGraphSpec JSON)	unversioned, additive-only	n/a	JobGraphSpec::from_json, src/cluster/job_graph.cpp
Connector offsets and committables	ride the snapshot format	n/a	per-connector state encodings, docs/internals/sink-committer-framework.md
Plugin ABI	kAbiVersion + declared-surface fingerprint + toolchain identity	exact match on all three gates	src/cluster/plugin_loader.cpp, scripts/plugin-abi-surface.txt
Embedded C ABI	2 (CLINK_EMBED_ABI_VERSION)	caller-checked; grows by appended functions and struct_size-gated fields	include/clink/embed/clink.h; symbol manifest scripts/libclink-abi-symbols.txt
Incident capture (.cap)	2 (kCaptureVersion)	header-gated	include/clink/runtime/record_capture.hpp
Capabilities manifest JSON	1 (schema_version)	additive-only within 1	include/clink/connectors/capability.hpp
Derived record codec (described types)	1 (layout specified in the header)	1	include/clink/core/derived_codec.hpp; fixture derived-codec-v1.bin
State shape fingerprints	1 (kind-tag table in fields.hpp)	additive: absence gates nothing	clink.state_fingerprints metadata key; fixture state-fingerprints-v1.txt
Persisted SQL catalog (--catalog-dir JSON)	1 (unversioned, additive-only)	readers ignore unknown keys	Catalog::to_json / from_json, src/sql/catalog.cpp; fixtures catalog-{table,function,model}-v1.json
Protocol trace events (CLINK_PROTOCOL_TRACE_DIR)	vocabulary in formal/trace/events.txt; fields additive-only	a trace module reading an older trace ignores unknown fields; an unknown event name fails validation	include/clink/cluster/protocol_trace.hpp; scripts/check-protocol-trace-events.py holds code, vocabulary and formal/trace/TraceExactlyOnce.tla in agreement

Cluster control protocol

kClusterProtocolVersion = 3, kMinCompatibleClusterProtocolVersion = 1 (include/clink/cluster/protocol.hpp). Peers exchange both numbers in the handshake messages and are compatible when the ranges overlap, so versions may differ as long as each side can speak something the other accepts.

• Enforcement, all three directions: the coordinator refuses an incompatible worker at Register; the worker refuses an incompatible coordinator at RegisterAck ("the coordinator can read me" does not imply "I can read the coordinator"); the coordinator refuses an incompatible client at HelloClient. Each refusal increments the protocol_mismatch metric, so a cluster half-refusing its peers is visible to monitoring.
• Compatible additions are tail-appended fields: decoders treat an absent tail as the field's v1 default, and a peer that declares nothing reads as version 1 (pre-versioning peers still decode). Bump kClusterProtocolVersion only for a change old peers would misread. Where a frame's fencing epoch is read as its last field (Deploy, FinalCheckpointAssigned, and every other frame that carries one), a new field goes after the epoch, not before it: an older peer reads the epoch where it always did and stops. A field placed before the epoch shifts the bytes an older peer takes for its epoch.
• Deploy tail, after the fencing epoch. A v0.10.0 Deploy ends with adaptive_barrier_mode (u8) and then the fencing epoch (u64), and a v0.10.0 worker reads no further. Every field added since follows the epoch. From the epoch on, in this order: the epoch (u64; absent, 0, unfenced); the per-task succession, for each task in task order succeeds_first and succeeds_count (two u32s; absent, every task succeeds itself, kRestoreFromSelf and 1); then the run's checkpoint-id floor, a presence byte and the id (u8, u64; absent, unset, and a 2PC sink keeps its older rule at restore point 0). A decoder reads each in turn while bytes remain, so a frame from any sender since v0.10.0 decodes with the fields it carried and defaults for the rest, and a v0.10.0 worker reads the leading bytes it always read. tests/fixtures/deploy-msg-v0.10.0.bin, written by v0.10.0's own encoder, pins both directions (FormatFixtures.DeployMsgV0100FrameStaysDecodableAndPrefixesTheCurrentFrame), and WireProtocol.DeployTailDecodesWhereverASenderStopped decodes each prefix of the tail.
• Version 2 capability: worker heartbeats carry a tail sequence and a v2 coordinator returns HeartbeatAck. The worker uses acknowledgements as a bidirectional control-session lease, detecting one-way partitions that do not produce TCP EOF. The new frame is sent only when the registered worker declares v2; a v2 worker paired with a v1 coordinator falls back to EOF, so the minimum remains v1 for rolling upgrades.
• Version 3 capability: a v3 worker reads the Row sidecar's second layout (the typed timestamp codes ts_ms and tstz_ms, below). Nothing changes on the control wire; the version is the gate the coordinator uses to admit that layout per deployment. v1 and v2 peers remain compatible and never receive it.
• Visibility: the coordinator keeps the version each worker registered with, and GET /api/v1/workers and GET /api/v1/cluster report it per worker as protocol_version (0 for a peer that declared none), so a mixed-version cluster during a rolling upgrade shows in the read API and not only in the coordinator's logs. Pinned by ProtocolVersioning.TheReadApiReportsTheVersionEachWorkerRegisteredWith.
• Known limitation: client enforcement is one-directional. The coordinator sends no hello ack on success, so a CLI never learns the coordinator's version.
• Tests: tests/test_protocol_versioning.cpp (both refusal directions end to end, pre-versioning decode, refusal legibility); fuzz/fuzz_cluster_frame.cpp with its committed-reproducer replay; tests/fixtures/register-msg-v1.bin pins the handshake bytes, and tests/fixtures/deploy-msg-v0.10.0.bin the last Deploy that ends at its epoch.

Data plane (operator-to-operator wire frames)

Deliberately unversioned per frame (include/clink/runtime/network/wire.hpp). Both ends of a data channel are subtasks of one job, deployed by one coordinator to workers that were each admitted through the versioned control handshake above, so the data plane inherits the control plane's negotiation and never talks to a peer of unknown vintage.

• Compatible additions are new Kind values: framing is length-prefixed, so an unknown kind is skippable, and a payload-layout change gets a NEW kind rather than a changed one (WatermarkIdle = 6 and ArrowBatch = 7 are the precedents; Data = 0 is retired in place as documentation).
• Robustness is structural, not versioned: kMaxFrameBytes is enforced at both read sites before allocation; ArrowBatch payloads are self-describing Arrow IPC validated with ValidateFull before use, and the receiver checks the embedded schema against its registered batcher.
• Row sidecar layout, admitted per deployment. A Row frame's Arrow sidecar is described by row-schema codes. The first layout is every code a worker has always read; the second adds ts_ms and tstz_ms, which a worker from before protocol v3 reads as utf8, so a typed column sent to it would fail the channel ("batch parse failed"). The coordinator admits the second layout for a deployment only when it has a layout-bearing operator and every worker hosting it registered at v3 or later (admit_row_layout, include/clink/cluster/row_layout_admission.hpp), and decides again on every whole-job deploy: submit, restart, the replan rescale and HA recovery. A deployment with no layout-bearing operator can never send a typed column, so it runs the first layout on any workers and is never held to the second layout's placement rules. It says so by adding row_layout=2 to the params of the operators whose factory parses a layout-bearing schema (json_string_to_row_columnar, the three window types, equi_join_row), in the Deploy frames only; a factory without the stamp resolves second-layout codes to their first-layout types. A hot cutover never changes a job's layout (see fault tolerance and rescale). Every upgrade order is safe: an older coordinator never stamps, an older worker is never admitted, and a downgrade silently runs the first layout.
• Tests: tests/test_network_channel.cpp (oversized frames), fuzz/fuzz_data_frame.cpp with the decode-reencode round-trip property; tests/test_row_layout_admission.cpp (admission, the stamp, and the coordinator's Deploy frames on submit, restart, the replan rescale and a hot cutover) and the admission cases in tests/integration/test_rescale_exactly_once.cpp and tests/integration/test_ha_failover.cpp.

Checkpoint metadata sidecar

kCheckpointMetaVersion = 1 (include/clink/state/checkpoint_integrity.hpp). A key=value text sidecar carrying the version, checkpoint id, payload byte count and CRC32C, written next to the payload and fsynced before the COMPLETED marker that makes a checkpoint eligible for restore.

• Compatible additions are new keys: CheckpointMeta::parse skips unknown keys. A sidecar declaring a NEWER version than this build is Unsupported, never guessed at (CheckpointIntegrityTest.NewerSidecarVersionIsUnsupportedNotGuessed), and recovery falls back to the newest checkpoint that verifies.
• Failure ordering: verification runs before restore and before any sink is told to commit; an unverifiable checkpoint cannot cause an external side effect.
• Tests: tests/test_checkpoint_integrity.cpp, fuzz/fuzz_checkpoint_meta.cpp, tests/fixtures/checkpoint-meta-v1.txt.

Checkpoint directory layout and COMPLETED markers

A checkpoint directory holds each job's markers under _jobs/<job id>/ and, for the file backend, each job's state under _state/<job id>/v<generation>/<subtask>/ (design record 013). A COMPLETED-<id> marker is key=value lines: job, checkpoint, generation, subtasks (the participant set, since v0.7.0) and state (where the checkpoint's state is: _state/<job id> relative to the checkpoint directory, or an absolute location).

• Compatible additions are new keys: parse_completed_marker skips any it does not know.
• Older layouts restore. A marker without state= was written when every job using the directory kept its state at the top of the base, <base>/v<generation>/, and its checkpoint is restored from there. A marker that records neither its participants nor its state was cut short, and is read at the job's own state root when that exists.
• Downgrade is not supported. A build from before the state= line reads state at the top of the base, where this build no longer writes it.
• Tests: tests/test_state_location.cpp, and the cluster and embedded restores of a directory converted to the older layout in tests/test_cluster.cpp and tests/test_embedded_engine.cpp.

State snapshots and savepoints

One Apache Arrow IPC stream per snapshot, clink.format_version = "1" in the schema metadata, absence reading as version 1. Checkpoints and savepoints are the same format; the full contract, including the changelog variant and the rule that readers must ignore unknown metadata keys, is docs/internals/state-snapshot-format.md.

• User-value evolution is separate from format evolution: the StateVersionMap stamps (clink.state_versions metadata) drive migrate-at-restore and the pre-deploy compatibility check, so a job can change its own state schema without the container format moving.
• Tests: the snapshot round-trip and schema-evolution suites; tests/fixtures/snapshot-v1.snap is a frozen stream that must stay loadable by state_processor::Savepoint.

Job plan (JobGraphSpec JSON)

Unversioned; the policy is additive optional keys with absent-tolerant readers, and it matters more than it looks: specs are persisted in the HA directory, so a NEW coordinator recovering leadership must read a spec an OLD build wrote. name, column_lineage, expected_state_versions and determinism_coverage are the precedents - each defaults sanely when absent. from_json validates structure and refuses malformed specs before admission. Pinned by tests/fixtures/job-spec-v1.json.

A row-schema code keeps its meaning for good: str means utf8 in every spec an old build persisted, and the typed timestamp layout uses new codes rather than changing an old one. The row_layout stamp a coordinator adds at deploy is never written into the spec: the submitted, retained and persisted spec is the one the client sent, and job_graph_fingerprint erases row_layout as a guard.

Connector offsets and committables

Source offsets and sink prepared-transaction handles are operator state inside the snapshot, so they version with the snapshot format; the bytes within are each connector's own encoding, owned and evolved by that connector. The commit-side restore contract (staged barrier-consistent rows, CONFIRMED markers, idempotent re-delivery of executed commits) is docs/internals/sink-committer-framework.md and docs/internals/checkpointing.md; the cross-process crash-window tests in tests/integration/ are what hold it.

Plugin ABI

Five exported constants (include/clink/plugin/abi_version.hpp and abi_surface.hpp, both generated): kAbiVersion (manual bump for semantic breaks), kAbiFingerprint (hash of the DECLARED extension surface - the headers in scripts/plugin-abi-surface.txt - plus the build options that surface uses, the pinned Arrow version and kAbiVersion), kToolchain (preprocessor-composed stdlib / dual-ABI / sanitizer identity), kAbiSurfaceManifest (per-header "path=sha256" lines, so a refusal names what differs), and kAbiHash (exact git commit, informational). The loader (src/cluster/plugin_loader.cpp) admits a plugin iff the fingerprints, the target triples and - when the plugin exports the symbol - the toolchain identities all match exactly; CLINK_STRICT_PLUGIN_ABI=1 restores exact commit-hash matching, and a legacy plugin exporting no fingerprint symbol is held to the strict rule rather than waved through. The submitter advertises each plugin's identity inside SubmitJob (PluginAbiAdvert, an eof-guarded tail) and the coordinator refuses an incompatible plugin before requesting bytes; SubmitJobAck carries the cluster's identity (and, on such a refusal, its manifest) in eof-guarded tail fields. There is no N-1 tolerance by design: a plugin links clink_core statically, so "compatible" means "built against an identical declared surface with a layout-compatible toolchain" (design record docs/design/010-stable-extension-model.md).

Embedded C ABI

CLINK_EMBED_ABI_VERSION = 2 (include/clink/embed/clink.h). The contract is caller-side: compare clink_abi_version() against the macro you compiled with before using anything else. Version 2 is the one deliberate break before 1.0 (design record docs/design/011-public-api-tiers.md): clink_engine_options gained a leading struct_size, so the surface can now grow without another bump, and within 1.x it never bumps again.

• Compatible additions are new functions (appended to scripts/libclink-abi-symbols.txt) and new fields appended to clink_engine_options. The library reads a field only when the caller's declared struct_size covers it, so a binary built against an older header gets the defaults for fields it never knew, and one built against a newer header is read only as far as the library knows. A zero struct_size is refused by name. CLINK_ENGINE_OPTIONS_INIT fills the size in.
• Removals happen only at a major release, after the function has carried CLINK_DEPRECATED for at least one minor.
• Enforcement: scripts/check-c-abi-symbols.py holds the manifest equal to the header's CLINK_EMBED_API declarations (CI and the pre-commit hook, no build) and, as the ClinkCAbiSymbolManifest test, to the dynamic symbol table of the built libclink. clink_version() names the library release for logging; the ABI question is clink_abi_version().
• Tests: clink_c_abi_tests (the shorter-struct and larger-struct reads, the zero-size refusal), clink_c_smoke (pure C99 consumer), pyclink's suite (the ctypes mirror of the struct).

Persisted SQL catalog

A catalog directory (--catalog-dir) holds one JSON document per table, model and function (<name>.json, models/<name>.json, functions/<name>.json), written by Catalog::to_json and read back by from_json, model_from_json and function_from_json (src/sql/catalog.cpp). A 1.0 catalog directory loads on every 1.x release.

• Compatible additions are new keys: every reader takes the keys it knows and ignores the rest, pinned by SqlCatalogFixtures.UnknownKeysAreIgnoredOnLoad. Column types are stored as their SQL spelling and parsed back through the SQL type grammar, so a type the grammar accepts is a type the catalog can hold.
• Tests: tests/test_sql_catalog_fixtures.cpp reads the frozen tests/fixtures/catalog-table-v1.json, catalog-function-v1.json and catalog-model-v1.json (regenerate with CLINK_REGEN_FORMAT_FIXTURES=1 only to add a fixture).

Incident capture (.cap)

Magic CCAP, kCaptureVersion = 2 (include/clink/runtime/record_capture.hpp); the header carries the version and a truncated flag, and replay surfaces the file's version in ReplayInfo::format_version. Determinism and cross-version A/B semantics are docs/internals/replay-determinism.md.

Capabilities manifest JSON

schema_version = 1, first key of clink --capabilities-json (kCapabilityManifestSchemaVersion, include/clink/connectors/capability.hpp). Bumped only when a key is renamed, removed or changes meaning; additions do not bump it. The build block carries git_sha and git_clean so a manifest maps back to code. Pinned by ConnectorCapabilityTest.ManifestJsonDeclaresItsSchemaVersionAndOrigin.