This feature release completes online shard splitting with coordinator-driven automatic splits, introduces dynamic cluster administration (data-server and namespace CRUD through the admin API and CLI), and adds coordinator high availability with leader election and admin-request redirection. It also brings major performance, durability, and stability improvements across the WAL, replication, session, and read paths.

Compatibility

• Upgrade coordinators before data servers. v0.17.0 introduces a coordinator-to-data-server instance-ID handshake that gates all internal RPCs: an upgraded coordinator manages v0.16.x data servers through a deprecated GetInfo fallback, but a v0.16.x coordinator cannot initialize v0.17.0 data servers (#1048, #1199).
• Complete the data-server roll promptly. Once an upgraded data server is bound to the coordinator, it rejects internal RPCs that do not carry the instance ID, so v0.16.x leaders cannot replicate to already-upgraded followers; leadership converges onto upgraded nodes through the normal restart-driven elections. Avoid pausing the roll with a minority of data servers still on v0.16.x.
• Public RPC authority validation is enabled by default. Data servers reject client RPCs whose gRPC :authority does not match an advertised or configured address with PERMISSION_DENIED: oxia: unexpected authority, as v0.16.3 briefly did before v0.16.5 made the check opt-in. Clients connecting through proxies, load balancers, or alternate DNS names need those addresses listed in the cluster-config allowExtraAuthorities, or the check disabled with featureFlags.authorityValidation: false in the data server config (#1038, #1134).
• Client-facing errors now use standard gRPC status codes with Oxia error reasons attached as ErrorInfo details, and the unused LeaderHint message was removed. Mixed-version clients and servers interoperate, but non-Go client implementations matching the old custom status codes must adapt (#1118, #1119).
• The admin ListNamespaces response changed from a list of names to a list of NamespaceView objects carrying spec and live status, and the deprecated ListNodes RPC was removed together with the oxia admin list-nodes and oxia admin list-namespaces commands; use oxia admin dataserver get and oxia admin namespace get (#1087, #1203).
• Coordinator metadata written by v0.17.0 stays readable by v0.16.3+ readers for the Kubernetes ConfigMap and Raft providers, keeping rollback possible. Status-document compatibility for the file metadata provider is not preserved (#1099, #1105, #1207).

Requirements

• No data migration is required on data servers.
• Coordinator config: rename server.admin to server.public; the CLI bind flag changed from --admin-addr/-a to --public-addr/-p, keeping the default port 6651 (#1154).
• Coordinator cluster config is now sourced through the metadata provider: set metadata.kubernetes.configName or metadata.file.configName (with metadata.file.dir). The --conf/-f and --cconfig flags, including the configmap:<namespace>/<name> form, are deprecated but still accepted and mapped onto the new settings (#1059).
• Cluster config: namespace anti-affinity rules moved from policy.antiAffinities to antiAffinities directly on the namespace; the old key is not read (#1101).

Public API Changes

• Admin protocol: added CreateDataServer, PatchDataServer, and DeleteDataServer (completing the data-server registry introduced in v0.16.3) and namespace CRUD RPCs (CreateNamespace, GetNamespace, PatchNamespace, DeleteNamespace); ListDataServers and ListNamespaces responses now carry views with live status; removed the deprecated ListNodes RPC (#1087, #1091, #1092, #1093, #1094, #1095, #1096, #1097, #1129, #1203).
• Coordination protocol: added the Handshake RPC binding data servers to their coordinator and reporting supported features (GetInfo is deprecated, kept as a rolling-upgrade fallback), and the FreezeShard RPC used to quiesce a parent shard at split cutover; GetStatus now returns per-shard ShardStats with db size and operation counters (#1048, #1197, #1202).
• Replication protocol: leaders advertise cumulative_acks_supported on Append, letting followers acknowledge a whole sync round with one cumulative ack; older followers keep acking per entry (#1164).
• Client protocol: NotificationBatch entries changed from a map to a wire-compatible repeated field emitted in sorted key order, making the serialized batch deterministic across replicas; existing client implementations keep working unchanged (#1190).
• Cluster configuration schema: new coordinators (name and public address per coordinator) and autoSplit sections (#1201, #1202).

Go Client Library

Relative to the last published client module version, oxia/v0.16.1:

• SyncClient and AsyncClient are unchanged.
• Breaking: AdminClient was reworked. Every method takes a context.Context, data-server and namespace CRUD methods were added, ListNamespaces returns []*proto.NamespaceView, and ListNodes with its result types was removed (#1087, #1091, #1092, #1093, #1094, #1095, #1096, #1097, #1203).
• Added ErrUnauthorized, ErrUnauthenticated, and ErrUnknown sentinel errors, matchable with errors.Is (#1036).

Metrics Changes

• Added oxia_coordinator_autosplit_evaluations_total and oxia_coordinator_autosplit_splits_initiated_total counters on the coordinator (#1205).
• Shard-split observer metrics on the parent leader, introduced with the split groundwork in v0.16.2 but not documented then: oxia_server_observer_snapshots_started/_completed/_failed/_sent, oxia_server_observer_snapshots_transfer_time, and the per-observer oxia_server_observer_ack_offset gauge for monitoring split catch-up (#945).
• oxia_server_session_active changed from an asynchronous gauge to an up-down counter: the value and Prometheus exposition are unchanged, but OTLP pipelines filtering by instrument type now see a non-monotonic sum (#1182).

Operational Changes

• Automatic shard splitting is available and disabled by default: when autoSplit is enabled in the cluster config, the coordinator initiates splits from sustained per-shard size/throughput stats, with stabilization and cooldown windows and a max-shards-per-namespace guard (#1202, #1205).
• Multiple coordinators can run against shared metadata: each coordinator has a stable identity (metadata.name, defaulting to the hostname), the elected leader publishes its address in the metadata lease, and admin requests reaching a non-leader are redirected to the leader; coordinator public addresses are listed in the cluster config (#1155, #1156, #1192, #1201).
• The admin CLI was restructured into resource command groups: oxia admin namespace create|get|patch|delete, oxia admin dataserver create|get|patch|delete, and oxia admin shard split (replacing split-shard), with a new -o/--output json|yaml|table selector; admin and client CLIs accept --auth-token/--auth-token-file independently of TLS (#1057, #1104, #1109, #1114).
• Empty cluster configs are accepted, so a cluster can be bootstrapped and then populated entirely through the admin API (#1079).
• Node saturation no longer triggers leader elections: a server that misses health probes only because it is overloaded no longer causes re-election storms that amplify the overload (#1233).
• Sessions scale much further: per-session expiry goroutines were replaced by a shared scheduler, cutting per-session overhead from kilobytes plus a goroutine to about a hundred bytes (#1236).
• The coordinator exits on metadata leadership loss instead of risking a split brain (#1175).
• Log noise reduced during startup, elections, and client disconnects; session lifecycle logging moved to debug level (#1208, #1212, #1213, #1226, #1229).
• --profile now also enables mutex and block profiling (#1238).

Changes Since v0.16.3

Fixes that were backported to the v0.16.4-v0.16.7 patch releases are also included below, since v0.17.0 supersedes the release-0.16 branch (#1115, #1116, #1123, #1124, #1133, #1134, #1137, #1139, #1143, #1144, #1147, #1175).

• shard split: correct split cutover end to end — children fenced at the parent's term, parent frozen and drained before fencing, namespace term options carried to children, and commit offsets propagated to observers parked at the WAL head (#1173, #1197, #1224, #1232)
• coordinator: automatic shard splitting driven by per-shard stats (#1202, #1205)
• coordinator: admin CRUD APIs and CLI for data servers and namespaces, with status views and output formats; deprecated ListNodes removed (#1057, #1087, #1091, #1092, #1093, #1094, #1095, #1096, #1097, #1129, #1203)
• coordinator: leader election with stable identities, lease-published public addresses, and admin-request redirection to the leader (#1155, #1156, #1192, #1201)
• coordinator: instance-ID handshake binding data servers to the coordinator, with initial elections gated on the handshake (#1048, #1199)
• coordinator: config model cleanup — server.public rename, cluster config sourced through the metadata provider, empty cluster configs accepted, namespace anti-affinities directly on the namespace (#1154, #1059, #1079, #1101)
• coordinator: metadata schema normalized to protobuf while keeping YAML/JSON documents compatible with v0.16.3+ readers (#1099, #1105, #1207)
• coordinator: prevent node saturation from triggering leader elections (#1233)
• coordinator: raft metadata provider hardening — snapshot retention, resource lifecycle, FSM synchronization, and exit on leadership loss (#1170, #1172, #1175)
• coordinator: require synced status before balancing, avoid blocking best-effort election enqueue, synchronize runtime close with shard deletion (#1106, #1222, #1241)
• wal: recover segments with empty or stale index files instead of panicking, reject empty index files, close read-only files...

This is a patch release for the 0.16 line, with a single fix for coordinators that use Kubernetes ConfigMap-backed metadata.

Fixed

• The coordinator exits when it loses metadata leadership (#1157). The ConfigMap metadata leader elector can stop after losing its Kubernetes lease, but the coordinator process previously kept running — acting as a stale leader until a later metadata write happened to fail with a version conflict. The coordinator now terminates with a dedicated leadership-loss error as soon as the elector stops, letting Kubernetes restart it cleanly.

Upgrade notes

• No data migration, configuration, public API, or metrics changes.

Full Changelog: v0.16.6...v0.16.7

This is a patch release for the 0.16 line. It closes several shutdown races in the data server that could crash the process while a shard leader was closing, fixes notification handling after restarts and failed stream initialization, and makes the coordinator retry loading ConfigMap-backed cluster configuration at startup instead of exiting.

Fixed

Data server shutdown safety

• Follower cursors are drained on close (#1141). Leader controller shutdown could close the Pebble DB while follower-cursor goroutines were still sending a snapshot and holding snapshot/file references. Shutdown now waits for those goroutines to finish first.
• In-flight leader operations are drained before the Pebble DB closes (#1146). Reads, lists, range scans, notification dispatches, and proposals could still hold DB/WAL resources while the leader was closing, triggering Pebble element has outstanding references panics. The leader now drains this work — and unblocks pending commit waiters — before closing the WAL and DB.

Notifications

• The notification dispatcher stops after a failed initial send (#1145). If the initial (dummy) notification could not be sent, the dispatcher previously kept running on a stream that was never properly initialized; it now stops immediately.
• The notification tracker initializes from the last real notification (#1149). Control commands advance the commit offset without writing notification rows, so after a restart a tracker initialized from the commit offset could spin on empty reads at a control-only tail. It now initializes from the highest actual notification key.

Coordinator

• Startup retries loading the cluster config (#1148). A coordinator could exit during startup when the ConfigMap-backed cluster config was not readable yet; it now retries with backoff until the config can be loaded.

Upgrade notes

• No data migration, configuration, public API, or metrics changes.

Full Changelog: v0.16.5...v0.16.6

This is a patch release for the 0.16 line. It resolves the authority-validation compatibility issue introduced in v0.16.3 — validation is now opt-in and disabled by default — and fixes two WAL-replay bugs that could surface after a restart: followers failing with oxia: entry not found, and negotiated feature flags being silently lost.

Fixed

• Authority validation is now opt-in, restoring pre-v0.16.3 client compatibility (#1136). Since v0.16.3, data servers validated the public RPC :authority header against shard assignments, rejecting clients that connect through bootstrap addresses, proxies, load balancers, or alternate DNS hostnames with PERMISSION_DENIED: oxia: unexpected authority. Validation is now controlled by the new data server config flag featureFlags.authorityValidation and is disabled by default on the 0.16 line, so no configuration change is needed to restore the previous behavior.
• Control requests persist the database commit offset (#1135). Control requests were replicated through the WAL but did not advance the database commit offset on replay. After WAL trimming, a restarted follower could try to resume from a stale commit offset older than the first WAL entry and fail with oxia: entry not found.
• Enabled features are persisted in the database (#1138). Feature-enable control records kept the negotiated feature set only in memory, so a restart that skipped WAL replay would lose it. Enabled features are now stored under internal __oxia/features/ keys in the same batch as the commit offset and restored when the database opens.

Upgrade notes

• No data migration or configuration change is required.
• To opt in to authority validation, set featureFlags.authorityValidation: true in the data server config.
• No metrics changes.

Full Changelog: v0.16.4...v0.16.5

This is a patch release for the 0.16 line. It fixes an authority-matching bug in the validation introduced by v0.16.3, reduces Pebble memory usage in multi-shard deployments, and adds health checking so stale pooled gRPC connections are evicted instead of reused.

Compatibility

v0.16.4 keeps the public RPC authority validation introduced in v0.16.3 enabled. Clients that connect through bootstrap addresses, proxies, load balancers, or alternate DNS hostnames can be rejected with PERMISSION_DENIED: oxia: unexpected authority unless those hostnames are listed in the cluster-level allowExtraAuthorities setting.

This is resolved in v0.16.5, where authority validation is opt-in and disabled by default — if you are affected, upgrade directly to v0.16.5 or later.

Fixed

• Scheme-prefixed shard-assignment addresses no longer fail authority validation (#1125). Authority validation compares the client :authority header against plain host:port values, so allowed authorities generated from scheme-prefixed assignment addresses could reject valid clients. Schemes are now stripped when building the allowed-authority set.
• Pebble uses its default memtable size (#1126). Each shard owns its own Pebble instance, and the previous explicit 32 MiB memtable override multiplied across shards. Removing it reduces memory usage in multi-shard deployments.
• Stale gRPC connections are evicted from the connection pool (#1130). Pooled connections are now health-checked and evicted as soon as a probe fails, so clients going through proxies or load balancers stop reusing dead connections. gRPC health services are registered on the data server public endpoint and the coordinator admin endpoint to support the probes.

Other

• CI: the build workflow now also runs for pull requests targeting release-* branches (#1127).

Upgrade notes

• No data migration is required. No public API or metrics changes.
• Keep allowExtraAuthorities configured when using v0.16.4 with proxied or alternate-DNS access paths (see Compatibility above).

Full Changelog: v0.16.3...v0.16.4

This release adds public RPC authority validation, data server admin APIs, and checksum metric cleanup. It also fixes Docker image rebuild behavior so release images pick up current Alpine security patches.

Compatibility

• v0.16.3 introduced public RPC authority validation, which may reject clients connecting through bootstrap, proxy, load balancer, or alternate DNS hostnames with PERMISSION_DENIED: oxia: unexpected authority.
• For v0.16.3 and v0.16.4, add expected hostnames to cluster-level allowExtraAuthorities.
• This is resolved in v0.16.5, where authority validation is configurable and disabled by default on the 0.16 line.

Requirements

• Review data server access paths before upgrading to v0.16.3 or v0.16.4.
• Configure allowExtraAuthorities if clients connect through proxy, bootstrap, load balancer, or alternate DNS names.

Public API Changes

• Data servers validate public RPC :authority values against shard assignments.
• Added cluster config field allowExtraAuthorities.
• Added admin RPCs: ListDataServers, GetDataServer.
• Added CLI commands: oxia admin dataserver list, oxia admin dataserver get <name>.

Metrics Changes

• Removed the dynamic commit-offset label from oxia_dataserver_db_checksum and oxia_dataserver_wal_checksum.
• Checksum metrics are now labeled at shard level only to reduce high-cardinality series.

Operational Changes

• Docker release images now rebuild the runtime stage so Alpine package updates are included in published images.

Changes Since v0.16.2

• admin: add data server list/get APIs and CLI commands (#1036, #1043, #1044, #1045)
• dataserver: validate public RPC authority against shard assignments (#1038)
• cluster: allow extra authorities from cluster config (#1040)
• metrics: remove commit-offset label from checksum metrics (#1039)
• docker: rebuild runtime stage on every build (#1046)

Full Changelog: v0.16.2...v0.16.3

This release includes a security hardening pass, shard splitting support, and multiple stability improvements around Kubernetes config watching, namespace placement, notification trimming, and coordinator behavior.

Compatibility

• Security hardening may reject invalid namespace names, TLS versions below TLS 1.2, inconsistent TLS min/max settings, and malformed authority metadata that was previously accepted.
• TLS CA certificate loading now processes all certificates in multi-cert PEM bundles.

Requirements

• TLS configuration must use TLS 1.2 or newer.
• Namespace names must satisfy the stricter validation pattern.
• Review internal/admin authentication settings if those APIs should be protected; optional auth configuration is now supported.

Security Fixes

• Enforced OIDC audience validation and documented the multi-audience check path (#1024).
• Redacted bearer tokens from authentication failure logs (#1025).
• Fixed TLS CA chain parsing for multi-cert PEM bundles (#1026).
• Fixed session heartbeat race that could panic under concurrent session close and keepalive (#1027).
• Hardened namespace validation, standalone authority validation, enum handling, context cancellation, and TLS version validation (#1028, #1030, #1031, #1032).
• Updated vulnerable dependencies and Alpine/OpenSSL base packages (#950, #995, #1007, #1023).

Public API Changes

• Added shard splitting data server support, coordinator orchestration, and client re-routing to child shards after split (#945, #957, #970).
• Added custom gRPC resolver support through client dial options (#949).
• Added --random-payload flag to oxia perf (#968).
• Internal and admin gRPC servers now support optional authentication configuration (#1029).

Metrics Changes

• Metrics endpoints now support optional TLS configuration.
• No metric names or labels were removed in this release.

Operational Changes

• Coordinator now reacts to ConfigMap cluster config changes and reconnects Kubernetes watches after expiration.
• Namespace status is published during initial placement.
• Notification trimming handles offset gaps more safely.
• Leader selection, RF=1 load balancing, Kubernetes metadata updates, and status resource handling were hardened.
• Replaced a status-resource CAS loop with atomic ApplyChanges and used vtproto for notification batch unmarshalling.

Changes Since v0.16.1

• security: fix OIDC, token logging, TLS chain, heartbeat, namespace, authority, enum, and TLS validation issues (#1024-#1032)
• split: add shard split observer/snapshot, orchestration, and client re-routing support (#945, #957, #970)
• client: add custom gRPC resolver support (#949)
• perf: add random payload generation (#968)
• coordinator: fix ConfigMap watch/update and namespace placement behavior (#1013, #1014, #1021)
• dataserver: fix notification trimming and node/member handling (#1018, #1020, #1011)
• dependencies: update vulnerable packages and Alpine/OpenSSL base image (#950, #995, #1007, #1023)

Full Changelog: v0.16.1...v0.16.2

This patch release focuses on production stability, client resilience, and the foundation for shard splitting. It improves leadership-loss handling, follower recovery, Kubernetes health checks, and retry behavior for streaming client operations.

Compatibility

• No breaking compatibility changes are expected.
• Shard splitting APIs are introduced as groundwork for the split workflow.

Requirements

• No data migration is required.
• Source builds now use Go 1.26.

Public API Changes

• Added admin SplitShard RPC and oxia admin split-shard command.
• Added split lifecycle model with SplitPhase values: Init, Bootstrap, CatchUp, Cutover, Cleanup.
• Added dedicated CodeNodeIsNotMember / ErrNodeIsNotMember handling so leaders can recover clean-data followers.
• Client List and RangeScan operations now retry with leader hints and exponential backoff.

Metrics Changes

• None.

Operational Changes

• Coordinator leadership loss now returns typed errors and restarts cleanly instead of panicking on metadata version conflicts.
• Follower recovery now resets stale cursors and sends snapshots when a restarted node reports NOT_MEMBER.
• gRPC health servers start before leader election so Kubernetes probes can succeed on non-leader coordinator pods.
• Coordinator GetInfo calls now run on state transitions instead of every health check.
• Fixed data races and test flakes in follower cursor, shard balancing, sync client, and leader balancing paths.

Changes Since v0.16.0

• admin: add shard split RPC, CLI, and split phase model (#911)
• coordinator: handle metadata leadership loss without panic (#935)
• replication: recover followers that restart with clean data (#929)
• client: add retry support for List and RangeScan (#937, #938)
• health: start coordinator health server before leader election (#930)
• coordinator: call GetInfo only on state transitions (#933)
• build: upgrade to Go 1.26 (#926)

Full Changelog: v0.16.0...v0.16.1

This feature release adds database checksum fingerprints, negotiated features for safe rolling upgrades, WAL-replicated control requests, and leader hints for faster client retries. It also includes crash-safety and data-race fixes across the storage and replication layers.

Compatibility

• Mixed-version rolling upgrades are supported through feature negotiation. Nodes that do not support the new GetInfo RPC are treated as supporting no new negotiated features.
• New features activate only when supported by all quorum members.

Requirements

• No data migration is required.
• Roll through quorum members normally; feature negotiation gates new behavior until the quorum supports it.

Public API Changes

• Added replication GetInfo RPC for supported-feature discovery.
• Added LeaderHint details to non-leader client errors so clients can retry directly against the current leader.
• Added WAL control request replication for feature enablement and checksum control records.

Metrics Changes

• Added oxia_dataserver_db_checksum gauge for the current database checksum.
• Added oxia_dataserver_wal_checksum gauge for the current WAL checksum.
• Checksum samples include a commit-offset label in this release. That label is removed in v0.16.3 to reduce cardinality.

Operational Changes

• Control-plane commands are durably replicated through the WAL state machine.
• Followers seed WAL checksum state after snapshot installation to avoid checksum divergence.
• WAL cleanup now avoids dangling index files after crashes and handles empty index files without panics.
• Runtime log level changes now propagate to existing loggers.
• Several data races were fixed across replication, caching, and health-check paths.

Changes Since v0.15.3

• database: add chained CRC32 database checksum fingerprints (#877, #890, #891)
• replication: negotiate features for safe rolling upgrades (#878)
• statemachine: replicate control requests through the WAL (#882)
• client: use leader hints for faster retries (#883)
• wal: fix crash-safety and checksum recovery paths (#901, #912, #920)
• controller: fix balancer loop, follower retry behavior, and lifecycle races (#895, #907, #918)
• client: retry EOF on write streams during leadership transitions (#917)
• logging: propagate dynamic log level updates (#914)

Full Changelog: v0.15.3...v0.16.0

What's Changed

• feat: introduce the auto release by @mattisonchao in #866
• Fix release workflow by using Makefile target for consistent builds by @Copilot in #868
• Fix: Binary releases missing execute permissions by @Copilot in #870
• feat: add static key file support for OIDC authentication with per-issuer configuration by @Copilot in #874
• Revert "feat: add static key file support for OIDC authentication with per-issuer configuration (#874)" by @mattisonchao in #875
• Feat: add per-issuer OIDC configuration with static key file support by @mattisonchao in #876
• Allow to override version id and modification count by @merlimat in #872

Full Changelog: v0.15.2...v0.15.3