Skip to content

Apache Avro (serialization)

An Avro serialization codec for typed records, usable with any clink source or sink. It is an encoding, not a source or sink in its own right. For Avro values framed for a schema registry on a Kafka topic (the common enterprise shape), use format='avro' on the Kafka connector, documented under Schema Registry formats; that path is built on the same library.

Overview

The clink::avro impl ships only Codec<T> templates: there are no built-in Avro sources or sinks. The codecs encode and decode avrogencpp-generated record structs (any type for which avro::codec_traits<T> is specialised) to and from Avro's binary and JSON text wire formats. You compose them with an existing connector, for example registering an Avro codec for a record type and then reading or writing that type through a Kafka, file, or state-backend channel. Three codecs are provided: binary, JSON text, and a keyed-record wrapper that pairs a UTF-8 partition key with an Avro-binary payload.

Dependency and version

Component Provenance Version
Avro C++ (avrocpp / AvroCpp) From-source pin in the Debian image (scripts/install-connector-deps.sh, release tarball checksum-verified) / brew (macOS, avro-cpp, whose formula pins the same tarball) 1.12.1 (AVRO_CPP_VERSION in scripts/versions.env)

The codecs depend only on the Avro C++ library and clink::core. They do not use Arrow.

Enabling it

The impl is gated by the CMake cache variable CLINK_WITH_AVRO, which defaults to AUTO.

  • AUTO: probes for Avro C++ via find_package(AvroCpp CONFIG), falling back to a direct search for the avro/Specific.hh header and the avrocpp library under /opt/homebrew and /usr/local. If found, the target is built; if not, it is skipped quietly.
  • ON: requires Avro C++; configuration fails with CLINK_WITH_AVRO=ON but AvroCpp not found if it is absent.
  • OFF: the target is not defined.

The pinned toolchain image carries Avro C++ (built from source into /usr/local), so under AUTO the impl is built in CI and in every image-based build; on a host without it (brew install avro-cpp on macOS) the impl is skipped quietly. When built, the target defines CLINK_HAS_AVRO and links AvroCpp::AvroCpp, and impls/schema_registry gains its format='avro'.

cmake -S . -B build -DCLINK_WITH_AVRO=ON

Factories

None. The impl registers no sources or sinks. clink::avro::install() exists for API parity with the other impls but is a no-op, and clink::plugin::install_defaults does not call it. The codecs are header-only templates used directly from C++, not looked up by factory name.

Factory name Direction Record type
(none) n/a n/a

Configuration

There is no Options struct and no BuildContext parameter parsing. The codecs are constructed directly in C++:

Codec Argument Required Default Description
binary_codec<T>() none n/a n/a Avro binary encoder/decoder for T, raw bytes with no framing.
json_codec<T>(schema_path) schema_path (std::string) Yes none Avro JSON-text codec for T. The .avsc schema file is read once at construction via compileJsonSchemaFromFile.
keyed_record_codec<T>() none n/a n/a Codec for KeyedRecord<T>, a { std::string key; T payload; } pair. Wire shape: 4-byte little-endian key length, key UTF-8 bytes, 4-byte little-endian payload length, Avro-binary payload.

T must be an avrogencpp-generated struct, or any type with an avro::codec_traits<T> specialisation. There are no authentication options; this is a serialization layer only.

SQL usage

Not exposed through the SQL frontend; use the programmatic API. There is no connector='avro' mapping in src/sql/physical_plan.cpp.

Example

Based on impls/avro/tests/test_codec.cpp. For a record Greeting { long id; string message; } (an avrogencpp-generated struct, or a hand-written struct with an avro::codec_traits specialisation):

#include "clink/avro/binary_codec.hpp"
#include "clink/avro/json_codec.hpp"
#include "clink/avro/keyed_record_codec.hpp"

// Binary codec: raw Avro bytes, no framing.
clink::Codec<Greeting> bin = clink::avro::binary_codec<Greeting>();
Greeting g{42, "hello"};
auto bytes = bin.encode(g);
std::optional<Greeting> back = bin.decode(bytes);  // decode failure -> std::nullopt

// JSON text codec: schema file read once at construction.
clink::Codec<Greeting> json = clink::avro::json_codec<Greeting>("greeting.avsc");

// Keyed-record codec: partition key + Avro-binary payload in one buffer.
clink::Codec<clink::avro::KeyedRecord<Greeting>> keyed =
    clink::avro::keyed_record_codec<Greeting>();
clink::avro::KeyedRecord<Greeting> kr{.key = "customer-42", .payload = g};
auto kbytes = keyed.encode(kr);

Compose any of these with a typed source or sink by registering the codec for the record type and using that connector's typed fluent helpers (for example a Kafka message source carrying Greeting).

Delivery semantics

Not applicable. This impl provides no source or sink, so it carries no delivery guarantee. Any at-least-once, exactly-once, or replay behaviour comes from the connector the codec is composed with, not from the Avro layer. A failed decode (malformed bytes, schema mismatch, or a truncated KeyedRecord buffer) returns std::nullopt and increments the avro connector error metric rather than throwing.

Limitations

  • Codec-only: no built-in Avro source or sink. The codecs must be composed with another connector or a state backend.
  • T must be avrogencpp-generated or otherwise have an avro::codec_traits<T> specialisation; arbitrary structs are not supported automatically.
  • binary_codec emits raw Avro bytes with no length framing. To concatenate multiple records in one buffer, compose with a framing codec such as vector_codec or pair_codec from core.
  • json_codec reads its schema file once at construction; the file path must be present and readable at that point. A schema/record mismatch surfaces as a decode failure (std::nullopt) at runtime, not at construction.
  • keyed_record_codec keys are UTF-8 byte strings; both key and payload lengths are encoded as 4-byte little-endian, capping each at 2^32 - 1 bytes.

Testing

There is no env-gated live or integration test, and no Docker image is required: the codecs encode in memory and need no external broker or service. The in-process round-trip suite is impls/avro/tests/test_codec.cpp (binary, JSON, and keyed-record round trips, plus empty-field and truncated-buffer cases), built when CLINK_BUILD_TESTS is on and Avro C++ is present. Run it with:

ctest --test-dir build -L avro