Reference documentation for clink's source and sink connectors. Each connector
has its own page covering the dependency and pinned version, the CMake build
knob, the exact factory names, every configuration option, SQL usage where
available, an example, delivery semantics, and limitations.
Every connector is an optional module. It is gated by a CLINK_WITH_<NAME>
CMake option (default AUTO: built when its client library is found, skipped
otherwise; set ON to require it or OFF to exclude it). Most connectors link
a system client library obtained via apt (Debian) or brew (macOS); a few ride
the from-source toolchain (Apache Arrow/Parquet 24.0.0, iceberg-cpp v0.3.0,
aws-sdk-cpp 1.11.795, Pulsar client 4.2.0, DataStax cpp-driver 2.17.1,
clickhouse-cpp 2.5.1),
which is compiled at exact versions into CLINK_DEPS_PREFIX on both the host
and the Debian image. Versions are recorded per connector and in
scripts/versions.env.
The SQL connector= column lists the string to use in a SQL
CREATE TABLE ... WITH (connector='...') statement. A dash means the connector
is reachable through the programmatic API only.
Guarantees vary by connector and are stated on each page. In summary:
Exactly-once sinks require the on_barrier / on_commit two-phase-commit
contract. The Kafka transactional sink (kafka_2pc_sink_string) implements it.
The object-store and WebHDFS Parquet connectors offer both: the default
single-object sink is at-least-once, and a 2PC variant
(<connector>_2pc_*_sink, or delivery_guarantee='exactly_once' in SQL) stages
one file per checkpoint under <prefix>/staging and atomically promotes it to
<prefix>/committed only when the checkpoint completes globally.
Sources that record their position as operator state replay from the last
checkpoint on recovery; the exact mechanism (Kafka offsets, Postgres LSN,
object index, row index) and any caveats are documented per connector.
Messaging sources (RabbitMQ, NATS, Pulsar) acknowledge at the checkpoint
barrier rather than post-commit; unacknowledged messages are redelivered.