Skip to content

doc: mark the materialize-sink-id header and HEADERS option as GA - #39829

Merged
val-materialize merged 1 commit into
mainfrom
valeriechiang/ss-504-sink-id-header-ga
Oct 28, 2026
Merged

val-materialize merged 1 commit into
mainfrom
valeriechiang/ss-504-sink-id-header-ga

Conversation

@val-materialize

@val-materialize val-materialize commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

⚠️ Do not merge until v26.48 reaches Cloud on Thursday 2026-10-29 (Self-Managed: Friday 2026-10-30). Staged now so it's ready.

Removes the private preview callouts for the materialize-sink-id header and the HEADERS option, which both become generally available in v26.48:

  • Header table: materialize-sink-id is now added "Always", and the HEADERS row drops "(private preview)".
  • materialize-sink-id subsection: removes the callout, "When enabled for your environment," and the "will become generally available" note.
  • HEADERS option subsection: removes the callout and the "will become generally available" note.

Rebased on main after #40122. The flags turn on by default in #39908 (SS-517), which ships in v26.48.0.

Part of SS-504. Follow-up to #39590, #39824, #39830, and #40122.

🤖 Generated with Claude Code

@linear-code

linear-code Bot commented Oct 14, 2026

Copy link
Copy Markdown

SS-504

@val-materialize val-materialize changed the title doc: document the materialize-sink-id Kafka sink header as GA doc: mark the materialize-sink-id header and HEADERS option as GA Oct 14, 2026
@val-materialize

Copy link
Copy Markdown
Contributor Author

Hold: merge next week, on Thursday 2026-10-22 or Friday 2026-10-23, after v26.47 ships with materialize-sink-id and HEADERS on by default. Merge #39830 first.

@val-materialize
val-materialize force-pushed the valeriechiang/ss-504-sink-id-header-ga branch from 91f0879 to c3b6a39 Compare October 14, 2026 17:58
val-materialize added a commit that referenced this pull request Oct 14, 2026
#39830)

The Headers section of the Kafka sink page was marked private preview as
a whole, including the always-on `materialize-timestamp` header. This
scopes the private preview callouts to the two features that are
actually in private preview.

- Replaces the section-level private preview callout with a table of the
headers a Kafka sink emits (`materialize-timestamp`,
`materialize-sink-id`, and user-defined headers from the `HEADERS`
option) and what each contains.
- Moves `materialize-sink-id` into its own subsection, still marked
private preview, with a note that it becomes GA in v26.47.
- Moves the `HEADERS` option into its own subsection, still marked
private preview, with a note that it becomes GA in v26.47.
- Expands the `HEADERS` known limitations: duplicate keys can't be
expressed, `materialize-` keys are silently dropped, the headers column
also appears in the message body, and the option works only with the
upsert envelope.

Part of [SS-504](https://linear.app/materializeinc/issue/SS-504). #39829
(staged, merge next week) removes both private preview callouts when
they go GA in v26.47.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <[email protected]>
@bosconi bosconi mentioned this pull request Oct 16, 2026
@val-materialize
val-materialize force-pushed the valeriechiang/ss-504-sink-id-header-ga branch from c3b6a39 to 15f5222 Compare October 23, 2026 14:45
@val-materialize

Copy link
Copy Markdown
Contributor Author

Update: GA moved to v26.48. Hold until v26.48 ships (provisionally Thu 2026-10-29 / Fri 2026-10-30). Rebuilt on current main; merge #40122 first.

Part of SS-504. Removes the private preview callouts for the
`materialize-sink-id` header and the `HEADERS` option, which become
generally available in v26.48.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@val-materialize
val-materialize force-pushed the valeriechiang/ss-504-sink-id-header-ga branch from 15f5222 to c263eac Compare October 26, 2026 15:02
@val-materialize
val-materialize marked this pull request as ready for review October 26, 2026 15:04
@val-materialize
val-materialize requested a review from a team as a code owner October 26, 2026 15:04
@val-materialize
val-materialize merged commit dfae7b1 into main Oct 28, 2026
12 checks passed
@val-materialize
val-materialize deleted the valeriechiang/ss-504-sink-id-header-ga branch October 28, 2026 18:33
val-materialize added a commit that referenced this pull request Oct 29, 2026
…ader (#40120)

Adds a new **Export data > Patterns** section with one page,
**Transactional downstream updates using Kafka**. The page shows how a
consumer loading a Kafka sink into another store (Snowflake example) can
expose only timestamps the sink has fully committed, by gating on the
sink's progress frontier and joining data and progress records via the
`materialize-sink-id` header.

### Where each fact comes from

| Fact | Source |
|------|--------|
| Every message carries `materialize-timestamp` (value is the timestamp
as a decimal string) | `src/storage/src/sink/kafka.rs`
(`KafkaTxProducer::send`, `time.to_string()`) |
| `materialize-sink-id` header value is the sink's `GlobalId` (for
example `u123`), emitted when `kafka_sink_emit_sink_id_header` is on |
`src/storage/src/sink/kafka.rs` (`sink_id_header: ...
sink_id.to_string()`); `src/storage-types/src/dyncfgs.rs` |
| Progress record key is `mz-sink-{sink_id}` |
`src/storage-client/src/sink.rs` (`ProgressKey::new`) |
| Progress record value is JSON `{"frontier":[<ts>],"version":<n>}`;
`frontier` is the exclusive upper of committed times |
`src/storage/src/sink/kafka.rs` (`ProgressRecord`, `serialize_frontier`,
`commit_transaction`, `progress_record_migration` test) |
| Progress record is written in the same Kafka transaction as the data |
`src/storage/src/sink/kafka.rs` (`commit_transaction`) |
| A transaction and progress record are committed on every frontier
advance, even with no data | `src/storage/src/sink/kafka.rs` (sink loop,
`Event::Progress` branch) |
| Default progress topic name
`_materialize-progress-{env}-{connection}`, shared by all sinks on a
connection; name exposed in `mz_kafka_connections.sink_progress_topic` |
`src/storage-types/src/connections.rs` (`progress_topic`);
`doc/user/content/sql/create-sink/kafka.md`;
`doc/user/content/sql/system-catalog/mz_catalog.md` |
| Consumers need `isolation.level=read_committed`; exactly-once details
| `doc/user/content/sql/create-sink/kafka.md` (Exactly-once processing)
|
| `materialize-` header prefix is reserved |
`src/storage/src/sink/kafka.rs`;
`doc/user/content/sql/create-sink/kafka.md` |
| Snowflake connector stores payload in `RECORD_CONTENT`, headers in
`RECORD_METADATA:headers`, and the key in `RECORD_METADATA:key` only
with `StringConverter` | [Snowflake Kafka connector
overview](https://docs.snowflake.com/en/user-guide/kafka-connector-overview)
|

### Notes for reviewers

- **GA status:** the `materialize-sink-id` header and the `HEADERS`
option are GA in v26.48 (#39908 turns both flags on by default), so the
page has no private preview caveat. The Kafka sink docs drop their
preview callouts in #39829, and the v26.48.0 release notes (#40113) link
to this page.
- The Snowflake SQL is marked illustrative. The page names no connector
settings except `key.converter`, which the Snowflake docs confirm.

Part of [SS-504](https://linear.app/materializeinc/issue/SS-504).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants