Runbook¶

What to do when one of clink's alerts fires.

The alert rules in deploy/prometheus/clink-alerts.yaml ship with the project; the response did not, so every operator worked it out independently. Each section below covers one alert: what it means, what to look at first, and what the likely causes are in order of how often they turn out to be the answer.

Two things apply throughout.

A red alert on a busy machine is often the machine. Several of clink's failure signatures - checkpoint duration, restart loops, subtask failures - move with load. Before treating an alert as a defect, check whether something else on the host is consuming the cores. This is not hypothetical: a full day of this project's own hardening work went into diagnosing thirteen "failures" that were an unrelated process holding seven of twelve cores.

The coordinator's log is the primary source. Every recovery decision it takes is logged with the job id and the reason: awaiting_restart (attempt N/M), worker lost, replanned job_id=..., refusing to rewrite active-leader.json. If the log is silent about a job that is misbehaving, that silence is itself the finding - it means the coordinator did not observe the event, which is a different class of problem from handling it badly.

The Grafana dashboard at deploy/grafana/clink-dashboard.json charts the same metrics these procedures reference, section for section, and is gated the same way as the alert rules: every metric a panel queries is checked against the metric constants by tests/test_dashboard.cpp, so a renamed metric fails the build rather than leaving a panel silently empty mid-incident. Import it and point its data-source variable at the Prometheus that scrapes the cluster.

ClinkCheckpointsStalled¶

time() - clink_ckpt_last_completed_unix_seconds > 300

No checkpoint has completed for five minutes. Until one does, the job's recovery point is frozen: a failure now replays from wherever the last completed checkpoint was, and every record since is reprocessed.

Look at: the coordinator log for checkpoint <id> of job <id> FAILED, which names the subtasks that could not snapshot.

Likely causes, in order:

One subtask cannot write. The failure line names it. Check the disk where that worker's state lives (see ClinkDiskFillingWhereStateLives) and its permissions. A checkpoint completes only when every subtask acks, so one stuck subtask stalls the whole job.
A subtask is wedged, not failing. No FAILED line at all, just silence. The barrier has not reached it. Check clink_op_input_depth for a backpressured operator upstream of the quiet one.

One cause of this used to be invisible: a source whose barrier could not be sent acked the checkpoint as successful anyway, so the component that knew the send had failed reported success and the checkpoint simply stopped. That now fails fast with source could not deliver the barrier to every downstream channel (F69). Silence with no such line means the barrier did leave the source. 3. The interval is longer than you think. --checkpoint-interval-ms is per job. A job configured at 600000 will trip a 300s alert permanently and correctly.

Do not restart the job to "unstick" it before reading the FAILED line. A restart rolls back to the last completed checkpoint, which is exactly the data you are trying not to lose.

ClinkCheckpointsFailing¶

Checkpoints are being attempted and none are completing. Same investigation as ClinkCheckpointsStalled, but the coordinator is definitely seeing attempts, so skip straight to the FAILED lines and the named subtasks.

If a restore fails on integrity, read the whole message. It now names the older checkpoint in the same directory that still verifies, or says plainly that none does. That older one is a recovery option, not an automatic one: the failing checkpoint was marked COMPLETED, so a sink may already have committed output for it, and restoring further back replays everything after it. Check the sinks before taking it.

Note that --checkpoint-num-retained defaults to 1, which means there is usually no older checkpoint to name. Raising it is what buys a fallback.

One specific cause worth knowing: a checkpoint whose payload and sidecar disagree is refused at restore, not at write. If the log shows checkpoint-N.snap is X bytes, sidecar declares Y, something wrote that file twice. That was a real defect in the rescale path (F59) where a restore staged state into a directory another subtask was reading; if it appears outside a rescale, treat it as a storage-layer problem and check whether two processes share a state directory.

ClinkCheckpointDurationP99High¶

p99 checkpoint duration above 20s. A warning, not an outage: checkpoints are completing, just slowly.

Look at: the size of each completed checkpoint alongside the duration: the checkpoint directory on disk (du -sh <checkpoint_dir>/<id> over the last few ids), or clink_disagg_checkpoint_object_bytes on a disaggregated backend. There is no per-snapshot bytes metric. If the size is growing steadily, this is state growth, not a storage problem.

Likely causes:

Unbounded keyed state. A GROUP BY or a window with no TTL accumulates indefinitely. The SQL planner refuses the clearest cases at submission, but a hand-built DAG can still do it. Set a state TTL.
Slow durable storage. Divide the checkpoint size by clink_ckpt_duration_ms for an effective write rate and check it against the device.
Checkpoint interval too short for the state size. If a snapshot takes 20s and the interval is 10s, checkpoints queue behind each other and the p99 climbs without bound.

ClinkWorkerLost¶

increase(clink_coordinator_workers_lost_total[10m]) > 0

The coordinator declared a worker lost. Its jobs roll back to their last completed checkpoint and redeploy onto the survivors.

Find the cause first. The coordinator logs it in its worker lost: <id> (<cause>) line, and puts the same text in the cause field of the coordinator.worker_lost event. It is one of three:

heartbeat timeout: nothing arrived from the worker for heartbeat_timeout. The process died, was paused, or the network between the two failed. Check whether the worker process is still running and what it logged last. A repeating count with live workers is a heartbeat timeout set too tight for the network or the hosts' load.
no outbound progress for <N> ms with <M> frames pending: the worker went on heartbeating but stopped reading what the coordinator sent it, for longer than outbound_stall_timeout. Check the worker's log for a frame whose handling never returned (a Deploy held in loading a fresh plugin is the usual one), and its host for CPU, memory or disk pressure.
outbound queue stopped (<reason>): <detail>: the worker's outbound queue on the coordinator stopped by itself. overflow means more was queued for the worker than the outbound bounds allow, so it had stopped reading for a while (the detail names the bound); allocation means the coordinator could not allocate memory to queue a frame, so look at the coordinator's own memory. A connection that broke under the coordinator's writer is not a cause of its own: the worker's heartbeats stop with it, and it reads heartbeat timeout.

Look at: whether the job recovered. The coordinator logs awaiting_restart (attempt N/M) and then a redeploy. If both appear and the job is running, the system did its job and this alert is informational.

If the job did NOT recover, check first whether the worker came straight back. A worker that dies and is restarted quickly - which is what a container orchestrator does by default - used to leave the job hung: the coordinator retired the previous session and never restarted the subtasks, because the watchdog does not declare a worker lost while it is alive, heartbeating and reading under the same id. That is fixed (F64), but the signature is worth recognising: the log shows worker=<id> re-registered; previous session retired and then nothing.

Capacity: if the lost worker's slots are needed and no survivor has room, the redeploy cannot be placed. See ClinkNoSlotCapacity.

ClinkJobRestartLoop¶

increase(clink_coordinator_job_restarts_total[30m]) > 5

A job is restarting repeatedly. Each restart rolls back to the last completed checkpoint, so a loop means the job is making no forward progress and reprocessing the same records.

Look at: the coordinator log for the restart reason. It differs per attempt and the pattern matters more than any single line.

Likely causes:

A deterministic subtask failure. The job restores, processes to the same record, and fails identically. The subtask error is in the log with the job id. A restart budget will not fix this; the job needs a code or data fix.
An unstable worker. The restarts follow worker losses rather than subtask errors. Correlate with clink_coordinator_workers_lost_total and treat it as an infrastructure problem.
The budget is masking it. --max-restarts-on-worker-loss bounds the attempts. A high budget turns a hard failure into a long loop; the job would be more diagnosable failing fast.

ClinkNoSlotCapacity¶

Every slot in the cluster is in use. New submissions are refused and a redeploy after a worker loss has nowhere to go, which turns a recoverable failure into a stuck job.

Look at: /api/v1/workers for slot capacity and usage per worker.

Note on sizing for recovery: a cluster sized so that the job exactly fits cannot survive a worker loss - the survivors have no room for the redeploy. Recovery needs one worker's worth of spare capacity, not just enough to run.

ClinkFencedFrames¶

increase(clink_worker_fenced_frames_total[10m]) > 0

Workers are rejecting frames from a coordinator whose epoch is below the one they are bound to. The fencing is working - a superseded coordinator is being ignored - but a healthy cluster does not produce these continuously.

A burst around a failover is expected. The old leader's in-flight frames arrive after the new one has taken over and are correctly refused.

A steady trickle is not. It means a superseded coordinator is still running and still trying to drive the cluster. Check for two live coordinators sharing an HA directory, and confirm the loser has actually exited.

The dangerous variant this cannot see: if the HA directory is on a filesystem that does not honour POSIX write locks, both coordinators believe they lead and each announces a higher epoch - so neither is fenced and this metric stays at zero. A coordinator now proves the lock works at start-up and refuses leadership if it does not (F57), logging refusing to stand for leadership. Bind mounts, 9p/virtiofs shares and NFS mounted nolock are the usual offenders. The check covers one host; it cannot detect a mount that honours locks locally but not between hosts, which is the NFS mode that matters most for multi-node. Use the etcd coordinator there.

ClinkProtocolMismatch¶

A peer was rejected for an incompatible protocol version. Almost always a partial upgrade: some binaries at the new version, some at the old.

Look at: the coordinator log, which names the refused peer and both versions. /api/v1/workers reports the protocol_version each admitted worker registered with (0 for a peer that predates the field), which shows how far a partial upgrade has got. No CLI command reports either.

ClinkWorkerControlDisconnected¶

clink_worker_control_connected == 0

This alert is not in the shipped rule set (deploy/prometheus/clink-alerts.yaml); define it from the expression above if you want it. The metrics it reads are exported by every worker.

The worker process is alive but has no admitted coordinator session. Its health endpoint returns 503 while it drains the old session, discovers a leader, or waits in reconnect backoff. It must not run or expose subtasks in this state.

Look at: the worker's /api/v1/worker recovery snapshot and worker.recovery log lines. state, last_error, coordinator_epoch, connection_attempts and successful_reconnections distinguish normal failover from a persistent refusal. Correlate with clink_worker_control_disconnects_total and clink_worker_control_connection_attempts_total.

A short dip during coordinator failover is expected. The worker has fenced and cancelled its old subtasks, then will register a fresh session without changing PID. A rising clink_worker_control_reconnects_total confirms recovery.

A sustained zero is actionable. Check leader discovery, reachability, TLS and the last registration refusal. Temporary coordinator connection-cap pressure is retried with jitter. Protocol or configuration refusals are fatal and the worker process exits with code 2 so the external process supervisor and operator can see that retrying the same binary/configuration is not progress.

ClinkRestoreDiscardedKeyedState¶

increase(clink_state_restore_keys_dropped_total[15m]) > 0

A restore loaded keyed entries whose key group falls outside the subtask's assigned range and discarded them.

During a rescale this is correct and expected. When an operator resizes, each new subtask reads a parent snapshot covering a wider key range than it owns and narrows it. The log says as much: during a rescale that is correct.

Outside a rescale it is not. It means a subtask restored a snapshot that does not belong to it, and the records for those keys are gone from its state. Check whether the job was rescaled recently and restarted from a PRE-rescale checkpoint - subtask directories are addressed by a job-global index, and that index is not stable across a topology change. That specific hole is fixed (F63), but the class is worth recognising: if state appears in the wrong subtask, suspect index reuse. A restore from a savepoint taken at a different subtask layout is the same class; a submit refuses it when the checkpoint's marker records its participants (see "refusing to restore checkpoint N" below), so seeing this alert after one means the marker was missing or ambiguous, or the two layouts happen to share an index set.

A job refuses to start with "refusing to restore subtask N with empty state"¶

Not an alert - a hard refusal at deploy. The coordinator named a completed checkpoint to restore from, and that subtask's snapshot file is not in the checkpoint directory.

This is the engine declining to lose your state quietly. It used to bring the subtask up holding nothing while its peers restored fully, which resumes a job that looks healthy with one operator's state gone.

Look at: the path in the message, and clink checkpoint-verify over the checkpoint directory. A completed checkpoint should have a snapshot for every participant, and the COMPLETED marker records which subtasks those were.

Likely causes, in order:

The directory was pruned by hand. Deleting snapshot files to reclaim disk is the usual way one goes missing. Drop whole checkpoints oldest first instead, and see ClinkDiskFillingWhereStateLives.
Storage lost it. A volume that was remounted, restored from a backup taken mid-checkpoint, or is silently dropping writes.
The operator is genuinely new. A stateful operator added to an existing job has no prior state. That case is legitimate and is what CLINK_ALLOW_MISSING_RESTORE_STATE=1 is for. If the new operator gave the job a different subtask layout, a submit-time restore is refused before this point (next section), and the override does not bypass that.
The state is not where its marker says. The COMPLETED marker records where its checkpoint's state is (state=): _state/<job id> under the checkpoint directory, or the state backend's location. A checkpoint directory copied without its _state/ directory, or the top-level v<N> directories an earlier release wrote deleted while a job still restores from one of that release's checkpoints, leaves the restore reading a location that holds nothing.

Do not set the override to make a restore start. It converts the refusal back into silent state loss for every subtask, not just the new one. Confirm which subtask is missing and why first; if the answer is not case 3, the checkpoint is not safe to restore from and an older complete one is the better recovery point.

A submit is refused with "refusing to restore checkpoint N ... this plan deploys M"¶

Not an alert - a refusal before anything deploys. The submitter gets it as a failed submit (an error response over HTTP); an HA recovery that hits it does not recover the job, and the coordinator logs recovery failed for job_id=... with the same text. The checkpoint's COMPLETED marker records which subtask indices took it, and the job as submitted plans a different set. State is restored by job-global subtask index, so restoring anyway would hand operators each other's state - which is what happened before this check, with a worker's restore discarded N keyed entries warning (ClinkRestoreDiscardedKeyedState) as the only trace.

Look at: the two counts and index ranges in the message, and the marker it read: the subtasks= line of <restore dir>/_jobs/<job>/COMPLETED-<id>, or of <restore dir>/COMPLETED-<id> when the restore named a job's marker directory.

Likely causes, in order:

An engine upgrade changed the plan. The message names the plan's single-instance operators. SQL scalar subqueries in SELECT and null-aware IN / NOT IN run as one instance from v0.9.0, so a v0.8.0 savepoint of such a job at a parallelism above 1 has more subtasks than the job now plans.
A different parallelism or graph. The job was submitted at another --parallelism than the savepoint was taken at, or its graph changed, including a stateful operator added with its own subtask.
An HA recovery after a rescale. Recovery plans the graph the job was submitted with, and a checkpoint taken after a rescale records the rescaled layout.

What to do: restore with the layout the checkpoint was taken with - the same graph and parallelism, on the engine version that took it - or submit without the restore to start from empty state. Do not edit the marker to get past the check: the operators would still read each other's files.

A submit is refused with "cannot restore checkpoint N from DIR: jobs A, B each completed a checkpoint N there"¶

Not an alert - a refusal before anything deploys. Several jobs checkpoint into the directory the restore named, each keeps its own state under _state/<job id>, and more than one of them completed a checkpoint with the id the restore asked for. Job ids number checkpoints independently, so this is the ordinary case for jobs that share a directory, such as the INSERT statements of one script.

What to do: name the job by its marker directory, --restore-from-dir DIR/_jobs/<job id>, with the id of the job whose state you want; the message lists each candidate. clink list, or the coordinator's job history, shows which id ran which pipeline. A restore point is one job's state, so a script that submits several jobs is restored one job at a time, or runs its statements as one job with EXECUTE STATEMENT SET.

ClinkSubtaskFailures¶

Subtasks are failing on a worker. The job restarts if it has budget; if not, it fails.

Look at: the worker log for the subtask's own error. The coordinator's line gives the job and subtask; the cause is on the worker.

Likely causes: an exception escaping user code in an operator, a connector losing its external system, or a state backend that cannot read or write. The three look identical from the coordinator and completely different on the worker.

ClinkMalformedFrames¶

The coordinator or a worker rejected a frame it could not parse. Frames are length-prefixed and size-capped, so a malformed one is refused rather than allocated for.

Likely causes: a non-clink client connecting to the control port, a partial upgrade (see ClinkProtocolMismatch), or a network middlebox altering the stream. Sustained malformed frames from one peer address with no upgrade in progress is worth treating as unexpected traffic on the port.

ClinkDiskFillingWhereStateLives¶

The filesystem holding checkpoints or state backends is filling.

This is the one to act on early. A checkpoint that cannot be written does not complete, the recovery point freezes, and the failure surfaces as ClinkCheckpointsStalled some minutes later - by which time the disk is fuller.

Look at: the checkpoint directory. Retention keeps a bounded number of checkpoints, but a job with large state and a short interval can still outrun a small volume between cleanups.

Do not delete checkpoint files by hand to buy room. A payload without its sidecar, or a partially deleted checkpoint, reads as incomplete and can cost you the recovery point you were trying to preserve. Use clink checkpoint-verify to see what is valid, and drop whole checkpoints oldest first.

Things that are worth checking whatever the alert¶

Is the coordinator the leader you think it is? /api/v1/config reports the epoch. Two coordinators serving the same cluster is a specific and serious failure - see ClinkFencedFrames.

Did the job actually recover, or did it merely restart? A restart that keeps failing looks like activity. clink list shows the terminal status; the coordinator log shows the attempt count against the budget.

A restarted coordinator shows no jobs. That is the contract, not a loss, when the coordinator runs without --ha-dir: job manifests are persisted (and recovered on leadership) only under an HA directory, so a plain restart abandons running jobs while their COMPLETED-N markers remain intact. Recover manually by resubmitting with --restore-from-dir=<checkpoint dir>/_jobs/<job id> and --restore-from-checkpoint-id=<latest N> (the checkpoint directory alone also works when no other job completed checkpoint N there), or run the coordinator with --ha-dir - one node is enough - so restarts recover jobs on their own.

What is this job actually configured to do? /api/v1/jobs/:id reports the checkpoint configuration the job is running with - directory, interval, state backend, restore source, restart budget and alignment. Check it before assuming a job that is not checkpointing is broken: an interval of 0, or an empty checkpoint_dir, means it was never asked to.

Is the recovery point moving? clink_ckpt_last_completed_unix_seconds advancing is the single best signal that a job is healthy. A job that is running, consuming and producing, but not completing checkpoints, is accumulating unbounded replay on its next failure.

What this runbook does not cover¶

There is no Grafana dashboard in the repository. The metric names above are stable and the alert rules encode the thresholds, but assembling a dashboard is left to the deployment.

None of the procedures here are exercised by an automated test, unlike the alert rules themselves (tests/test_alert_rules.cpp). They are written from the failure modes this engine has actually produced, but a runbook step that has never been walked through on a live incident should be read as informed guidance rather than a verified procedure.