Skip to content

feat: add SSE streaming endpoint for deduplication#22

Merged
Siddhant-K-code merged 1 commit into
mainfrom
feat/sse-streaming
Feb 15, 2026
Merged

feat: add SSE streaming endpoint for deduplication#22
Siddhant-K-code merged 1 commit into
mainfrom
feat/sse-streaming

Conversation

@Siddhant-K-code

Copy link
Copy Markdown
Owner

Adds POST /v1/dedupe/stream - a streaming variant of /v1/dedupe that sends Server-Sent Events as each pipeline stage completes.

Motivation

Large dedup requests can take several seconds (especially with embedding generation). Streaming provides progress visibility and faster time-to-first-byte.

SSE event protocol

event: progress
data: {"stage":"embedding","progress":0}

event: progress
data: {"stage":"embedding","progress":1}

event: progress
data: {"stage":"clustering","progress":1,"stats":{"clusters_formed":5,"input_count":12}}

event: progress
data: {"stage":"selection","progress":1,"stats":{"selected":5}}

event: complete
data: {"chunks":[...],"stats":{"input_count":12,"output_count":5,...}}

Three event types:

  • progress - stage name, completion ratio (0-1), optional per-stage stats
  • complete - final chunks + stats (same schema as /v1/dedupe response)
  • error - stage + error message

Usage

curl -N -X POST http://localhost:8080/v1/dedupe/stream \
  -H "Content-Type: application/json" \
  -d '{"chunks": [{"text": "..."}], "threshold": 0.15}'
const es = new EventSource('/v1/dedupe/stream', { method: 'POST', body: ... });
es.addEventListener('progress', (e) => console.log(JSON.parse(e.data)));
es.addEventListener('complete', (e) => console.log(JSON.parse(e.data)));

Changes

  • pkg/sse/sse.go - SSE Writer with typed event methods (progress, complete, error), stage timer
  • pkg/sse/sse_test.go - 9 unit tests
  • cmd/api.go - New /v1/dedupe/stream route, streaming handler with per-stage events, tracing + metrics integration

Closes #6

Add POST /v1/dedupe/stream that streams progress events via
Server-Sent Events during each pipeline stage (embedding,
clustering, selection, MMR).

Event types:
- progress: stage name + completion ratio + optional stats
- complete: final chunks + stats (same schema as /v1/dedupe)
- error: stage + error message

New package pkg/sse provides the Writer abstraction with typed
event methods. 9 unit tests.

Closes #6

Co-authored-by: Ona <[email protected]>
@Siddhant-K-code Siddhant-K-code added the enhancement New feature or request label Feb 15, 2026
@Siddhant-K-code Siddhant-K-code merged commit d60b57f into main Feb 15, 2026
2 checks passed
@Siddhant-K-code Siddhant-K-code deleted the feat/sse-streaming branch February 15, 2026 15:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Streaming Response Support

1 participant