Repository navigation
[RFC] Snapshot-and-follow output connectors #5980
Description
This RFC proposes a feature that allows an output connector to dump the entire state of the view before streaming incremental updates. This is helpful in two situations
- Adding a new connector to a pipeline.
We currently have two ways to add a connector: HTTP connector can be added at runtime. Other types of connectors can be added by stopping and modifying the pipeline.
In both cases, the current behavior is to only stream new changes to the view to the connector. In some (most?) cases, the user may want to send the current state of the view to the data sink before streaming incremental changes.
1.1. Subscribing to view changes
This is an important special case. Currently we have to go through ugly hoops to subscribe to all view changes since the start of the pipeline in the Python API and the webconsole. This wouldn't be necessary if there was a way to stream the view snapshot first. Unfortunately, this RFC will not fully solve that problem as it will only work for materialized views.
- Resetting an existing connector.
It may also be useful to have an existing connector send the complete state snapshot. This is helpful, e.g., if a connector error or an issue with the data sink has caused data corruption. While this can be achieved by deleting and recreating the connector, a single reset API is more convenient.
This feature can only be supported for materialized views, since the pipeline maintains the snapshot of the view's state.
API Design
Non-HTTP output connectors
Add a new Boolean property called snapshot to connector config (on the same level as transport and format). The value is false by default for backward compatibility.
Alternatively we could add a mode property with possible values follow (current default) or snapshot_and_follow, and maybe snapshot if we're interested in snapshot-only connectors.
HTTP egress
Add a mode argument with the same possible values to the /egress endpoint. When set to snapshot_and_follow, the connector will stream the current content of the table before streaming changes.
Resetting a connector
Add a new API:
POST /pipeline/<pipeline_name>/views/<view_name>/connectors/<connector_name>/reset
If the connector is configured with snapshot or snapshot_and_follow mode, this will cause it to send the full snapshot again, as well as perform any other actions it does during normal initialization, such as truncate the output delta table.
Implementation plan
TBD
Gotchas
Sending a full snapshot can take a long time and stall the pipeline.
Activity
lalithsuresh commentedon Apr 2, 2026
The API design makes sense. I think output
modeshould havefollowandsnapshot_and_follow. Not sure what asnapshotoutput mode does -- only writes out results once after the first step where it is used? That might have limited utility or be difficult to control for the user.Let's make sure the setting is also called
modeon the input side and has similar types for symmetry:snapshot,snapshot_and_followandfollow. I guess there's alsotransaction_modefor some input connectors.ryzhyk commentedon Apr 2, 2026
We can postpone
snapshotuntil it it needed.Yes, the input config is also called
modein the Delta connector, but in that case it's a transport property. This new config will be a global property supported by all connectors.lalithsuresh commentedon Apr 2, 2026
We can then retire the transport property in the delta connector right?
lalithsuresh commentedon Apr 2, 2026
Overall, I think this is a huge improvement. Allows users to build level-triggered systems as opposed to strictly edge-triggered.
ryzhyk commentedon Apr 2, 2026
I don't think we can. It's in the input Delta connector and is connector-specific (i.e., we don't have a design to support it for all input connectors yet, even though it logically makes sense)
ouatu-ro commentedon Apr 5, 2026
I just noticed in the docs that Delta and HTTP sinks don't support fault tolerance.
There are a couple things I'd want to clarify:
If someone asks for snapshot-and-subscribe, I assume they need continuity guarantees. So if at-least-once semantics are not provided, the natural client behavior is to reset and resubscribe whenever continuity is unclear.
If that's the intended model, I think there are two separate things that would help a lot:
Otherwise recovery itself can become the failure mode, especially if the initial snapshot is large (which I see that is already a concern).
gz commentedon Apr 5, 2026
hi @ouatu-ro, delta and http should support fault tolerance e.g., see https://docs.feldera.com/pipelines/fault-tolerance#fault-tolerant-connectors
(it is an enterprise only feature that we do not ship in the OSS version though, feel free to reach out to @lalithsuresh to learn more)
ouatu-ro commentedon Apr 5, 2026
Hello, sorry for the confusion. This feature is about output connectors: https://docs.feldera.com/connectors/sinks/http/
The page you linked mentions that pipelines can include non-fault-tolerant output connectors.
From the HTTP sink docs ("does not yet support fault tolerance") and from the code (
crates/adapters/src/transport/http/output.rs, whereis_fault_tolerantreturnsfalse), I inferred that HTTP egress itself is not fault-tolerant.If the docs or the code are outdated (or feature gated) and HTTP output does support fault tolerance, could you please confirm?
PS: I reread that section. I am interested in contributing to
snapshot_and_followaround HTTP. It sounds like you are saying that Enterprise pipelines can still get at-least-once behavior overall even though HTTP output itself is not fault-tolerant. Ifsnapshot_and_followis only meant to work well in that Enterprise-only setting, then it would be harder for me to contribute to directly, but my point about subscriber-visible continuity contracts would still stand either way.mythical-fred commentedon Apr 6, 2026
The RFC mentions "sending a full snapshot can take a long time and stall the pipeline" as a gotcha, but does not address the consistency model during that window.
Specifically: when a connector is in snapshot mode, what happens to incremental updates that arrive while the snapshot is being delivered?
Option 3 is the only correct one, but it needs to be explicit: the snapshot must correspond to a specific pipeline step/checkpoint boundary, and incremental delivery must resume from exactly that point. If that is the intended design, the API should expose the snapshot timestamp or step number so consumers can verify continuity.
This is also relevant to ouatu-ro's question about sequence number resumability — a sequence number only enables recovery if the snapshot is taken at a well-defined step boundary.
ouatu-ro commentedon Apr 6, 2026
Not sure if redundant from my part, but: buffering would be required either way because of transport. If not handled explicitly, it will occur implicitly in the OS/network stack (in addition to backpressure).
If you apply updates to the snapshot while it is already being transmitted, you get a "fuzzy snapshot" that corresponds to no well-defined cutover boundary (neither the start nor the end state). So that option is only sound before transport commitment. At that point, post-cutover updates must live somewhere until snapshot delivery completes.
gz commentedon Apr 6, 2026
cc @blp who was working on this feature
ryzhyk commentedon Apr 6, 2026
@ouatu-ro , FYI, @mythical-fred is an AI and is still learning :) He clearly doesn't know that buffered outputs, especially large ones, are mostly kept in storage, not in-memory. In addition, there is a backpressure mechanism that will stall the pipeline when there's >1M outputs buffered by an output connector.