feat: Added SSE Last Event Id Header #1104

MathurAditya724 · 2025-11-05T21:24:32Z

vercel · 2025-11-05T21:24:41Z

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Preview	Comments	Updated (UTC)
spotlightjs	Skipped			Nov 10, 2025 5:07pm

Copilot

Pull Request Overview

This PR implements SSE (Server-Sent Events) reconnection support using the Last-Event-ID mechanism to allow clients to resume receiving events from where they left off after a disconnection.

Key Changes:

Added Last-Event-ID header support in the SSE stream endpoint
Modified the message buffer subscribe method to support resuming from a specific event ID
Refactored telemetry component to eliminate redundant useMemo usage

Reviewed Changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.

File	Description
packages/sidecar/src/routes/stream/index.ts	Extracts Last-Event-ID header and passes it to subscribe; adds event IDs to SSE messages
packages/sidecar/src/messageBuffer.ts	Adds lastEventId parameter to subscribe method with logic to find and resume from the last received event
packages/overlay/src/telemetry/index.tsx	Refactors contentTypeListeners from useMemo to inline object literal in useEffect

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-11-06T10:55:07Z

packages/sidecar/src/messageBuffer.ts

+
+        if (item[1] instanceof EventContainer) {
+          const envelope = item[1].getParsedEnvelope();
+          if (envelope?.envelope && envelope.envelope[0].__spotlight_envelope_id === lastEventId) {


The comparison envelope.envelope[0].__spotlight_envelope_id === lastEventId may have a type mismatch issue. The __spotlight_envelope_id is generated using uuidv7() which returns a string, but lastEventId comes from an HTTP header and is already a string. However, for consistency and safety, consider using strict equality with explicit string conversion: String(envelope.envelope[0].__spotlight_envelope_id) === lastEventId as done elsewhere in the codebase (lines 26, 51, 243).

Suggested change

if (envelope?.envelope && envelope.envelope[0].__spotlight_envelope_id === lastEventId) {

if (envelope?.envelope && String(envelope.envelope[0].__spotlight_envelope_id) === lastEventId) {

@MathurAditya724 can you make sure __spotlight_envelope_id is always a string or undefined?

Copilot · 2025-11-06T10:55:08Z

packages/sidecar/src/messageBuffer.ts

+      for (let i = start; i < end; i++) {
+        const item = this.items[i % this.size];
+        if (item == null) continue;
+
+        if (item[1] instanceof EventContainer) {
+          const envelope = item[1].getParsedEnvelope();
+          if (envelope?.envelope && envelope.envelope[0].__spotlight_envelope_id === lastEventId) {
+            // Start from the position after the found event
+            startPos = i + 1;
+            break;
+          }
+        }
+      }


When searching for the lastEventId, this loop iterates through the entire buffer from head to writePos, potentially scanning hundreds of items (default buffer size is 500). This happens on every subscription with a Last-Event-ID header. Consider optimizing this by: 1) Searching backwards from the end since recent events are more likely, or 2) Maintaining an index mapping event IDs to positions, similar to how filenameCache works. The backwards search would typically find the event faster since SSE reconnections usually happen shortly after disconnection.

For this only I was telling that we can use the timestamp, that way we can do a binary search.

We are using uuidv7 which should be alpha/time-sortable: https://uuid7.com/ (see the first item in benefits)

So we should be able to do binary search here which would probably the most efficient way. Starting from the end also makes sense. Not so sure about a revers-hash-map as it's more book-keeping. I'd say binary search should be the optimal solution.

packages/sidecar/src/routes/stream/index.ts

packages/overlay/src/telemetry/index.tsx

packages/sidecar/src/messageBuffer.ts

BYK

Good one!

BYK · 2025-11-10T15:17:27Z

packages/sidecar/src/messageBuffer.ts

+      for (let i = start; i < end; i++) {
+        const item = this.items[i % this.size];
+        if (item == null) continue;
+
+        if (item[1] instanceof EventContainer) {
+          const envelope = item[1].getParsedEnvelope();
+          if (envelope?.envelope && envelope.envelope[0].__spotlight_envelope_id === lastEventId) {
+            // Start from the position after the found event
+            startPos = i + 1;
+            break;
+          }
+        }
+      }


We are using uuidv7 which should be alpha/time-sortable: https://uuid7.com/ (see the first item in benefits)

So we should be able to do binary search here which would probably the most efficient way. Starting from the end also makes sense. Not so sure about a revers-hash-map as it's more book-keeping. I'd say binary search should be the optimal solution.

betegon

We should add some tests to this. We're not adding them to UI or other parts of spotlight but we have to make sure the message buffer works.

So pls add some tests for this, like you did when we added the other week 🙏

Some ideas cursor gave me:

Subscriber with lastEventId that exists in buffer
Subscriber with lastEventId that was evicted (buffer overflow)
Subscriber with invalid/non-existent lastEventId
Subscriber with lastEventId at buffer boundary positions
Integration test simulating reconnection scenario

packages/sidecar/src/messageBuffer.ts

Copilot

Pull Request Overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated no new comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

BYK · 2025-11-10T17:29:29Z

packages/sidecar/src/parser/processEnvelope.ts

  }

-  envelopeHeader.__spotlight_envelope_id = uuidv7();
+  envelopeHeader.__spotlight_envelope_id = uuidv7obj();


uuidv7obj() and uuidv4obj() return an object that represents a UUID as a 16-byte byte array:

Okay, why do we even need this?

BYK · 2025-11-10T17:31:17Z

packages/sidecar/src/messageBuffer.ts

+    }
+
+    while (left <= right) {
+      const mid = Math.floor((left + right) / 2);


Shouldn't this be Math.floor((right - left) / 2)?

BYK · 2025-11-10T17:32:24Z

packages/sidecar/src/messageBuffer.ts

+      // Skip null items or non-EventContainer items
+      if (item == null || !(item[1] instanceof EventContainer)) {
+        // Linear scan nearby to handle gaps
+        let boundsAdjusted = false;
+
+        // Check left side
+        for (let i = mid - 1; i >= left; i--) {
+          const leftItem = this.items[i % this.size];
+          if (leftItem && leftItem[1] instanceof EventContainer) {
+            const envelope = leftItem[1].getParsedEnvelope();
+            if (envelope?.envelope) {
+              const envelopeId = envelope.envelope[0].__spotlight_envelope_id;
+              const isEqual = envelopeId.compareTo(targetUuid);
+              if (isEqual === 0) {
+                return i;
+              }
+              // Adjust search bounds based on comparison
+              if (isEqual < 0) {
+                left = mid + 1;
+              } else {
+                right = mid - 1;
+              }
+              boundsAdjusted = true;
+              break;
+            }
+          }
+        }
+
+        // Check right side if left didn't help
+        if (!boundsAdjusted) {
+          for (let i = mid + 1; i <= right; i++) {
+            const rightItem = this.items[i % this.size];
+            if (rightItem && rightItem[1] instanceof EventContainer) {
+              const envelope = rightItem[1].getParsedEnvelope();
+              if (envelope?.envelope) {
+                const envelopeId = envelope.envelope[0].__spotlight_envelope_id;
+                const isEqual = envelopeId.compareTo(targetUuid);
+                if (isEqual === 0) {
+                  return i;
+                }
+                // Adjust search bounds
+                if (isEqual < 0) {
+                  left = mid + 1;
+                } else {
+                  right = mid - 1;
+                }
+                boundsAdjusted = true;
+                break;
+              }
+            }
+          }
+        }


Okay, this entire block is very hard to follow. Vibe code?

BYK · 2025-11-10T17:33:17Z

packages/sidecar/src/messageBuffer.ts

+   * Returns the position of the event, or -1 if not found.
+   * Takes advantage of UUIDv7's time-ordered property.
+   */
+  private binarySearchEventId(targetId: string): number {


Let's just name this helper findEventIdIdx?

BYK

You know what, all the tests pass and they make sense. Let's address my comments in a follow-up

Updated afterwards

This PR was opened by the [Changesets release](https://github.com/changesets/action) GitHub action. When you're ready to do a release, you can merge this and publish to npm yourself or [setup this action to publish automatically](https://github.com/changesets/action#with-publishing). If you're not ready to do a release yet, that's fine, whenever you add more changesets to main, this PR will be updated. # Releases ## @spotlightjs/[email protected] ### Minor Changes - Docker compose support for Spotlight run command ([#1108](#1108)) - Added support for Last-Event-Id in SSE stream ([#1104](#1104)) - Change tool names: change `.` for `_` and remove `spotlight` preffix ([#1114](#1114)) - Improved human formatter readability ([#1112](#1112)) ### Patch Changes - Fix breaking error when event type is not supported ([#1111](#1111)) - Fix SDK categorization for Next.js by using User-Agent headers to distinguish browser from server events ([#1110](#1110)) ## @spotlightjs/[email protected] ### Patch Changes - Updated dependencies \[[`0942c8a`](0942c8a), [`bce012e`](bce012e), [`7624030`](7624030), [`15a7f41`](15a7f41), [`13da542`](13da542), [`b3a654a`](b3a654a)]: - @spotlightjs/[email protected] - @spotlightjs/[email protected] ## @spotlightjs/[email protected] ### Patch Changes - Fix SDK categorization for Next.js by using User-Agent headers to distinguish browser from server events ([#1110](#1110)) - Updated dependencies \[[`0942c8a`](0942c8a), [`bce012e`](bce012e), [`7624030`](7624030), [`15a7f41`](15a7f41), [`13da542`](13da542), [`b3a654a`](b3a654a)]: - @spotlightjs/[email protected] - @spotlightjs/[email protected] Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

feat: initial draft

40d57dd

fix: minor change

d694a40

vercel bot temporarily deployed to Preview November 5, 2025 21:30 Inactive

chore: removed changes

f4aa600

vercel bot temporarily deployed to Preview November 6, 2025 10:51 Inactive

MathurAditya724 requested a review from Copilot November 6, 2025 10:51

chore: added changeset

5f38018

MathurAditya724 marked this pull request as ready for review November 6, 2025 10:54

vercel bot deployed to Preview November 6, 2025 10:54 View deployment

Copilot AI reviewed Nov 6, 2025

View reviewed changes

chore: reverted the changes

c036f1c

vercel bot temporarily deployed to Preview November 6, 2025 11:22 Inactive

cursor bot reviewed Nov 6, 2025

View reviewed changes

packages/sidecar/src/messageBuffer.ts Outdated Show resolved Hide resolved

MathurAditya724 requested review from BYK and betegon November 10, 2025 04:47

BYK approved these changes Nov 10, 2025

View reviewed changes

betegon previously requested changes Nov 10, 2025

View reviewed changes

packages/sidecar/src/messageBuffer.ts Outdated Show resolved Hide resolved

packages/sidecar/src/messageBuffer.ts Show resolved Hide resolved

fix: added types for __spotlight_envelope_id

faf5208

MathurAditya724 deployed to Preview November 10, 2025 15:34 — with GitHub Actions Active

feat: added binary search for uuid

bbf066d

MathurAditya724 deployed to Preview November 10, 2025 16:22 — with GitHub Actions Active

cursor bot reviewed Nov 10, 2025

View reviewed changes

packages/sidecar/src/messageBuffer.ts Show resolved Hide resolved

MathurAditya724 added 2 commits November 10, 2025 21:55

chore: minor fixes

c545519

chore: minor changes

6c234af

MathurAditya724 deployed to Preview November 10, 2025 16:44 — with GitHub Actions Active

chore: added tests

90da4fa

MathurAditya724 deployed to Preview November 10, 2025 17:07 — with GitHub Actions Active

MathurAditya724 requested review from betegon and Copilot November 10, 2025 17:08

Copilot AI reviewed Nov 10, 2025

View reviewed changes

BYK requested changes Nov 10, 2025

View reviewed changes

BYK approved these changes Nov 10, 2025

View reviewed changes

BYK merged commit 7624030 into main Nov 10, 2025
26 checks passed

BYK deleted the adi/feat/sse-last-event-id branch November 10, 2025 21:45

sentry-release-bot bot mentioned this pull request Nov 10, 2025

meta(changelog): Update package versions #1118

Merged

	if (envelope?.envelope && envelope.envelope[0].__spotlight_envelope_id === lastEventId) {
	if (envelope?.envelope && String(envelope.envelope[0].__spotlight_envelope_id) === lastEventId) {

Uh oh!

feat: Added SSE Last Event Id Header #1104

feat: Added SSE Last Event Id Header #1104

Uh oh!

Conversation

MathurAditya724 commented Nov 5, 2025

Uh oh!

vercel bot commented Nov 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Nov 6, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 6, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

BYK left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

betegon left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

BYK left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

vercel bot commented Nov 5, 2025 •

edited

Loading