Skip to content

Commit b416cf3

Browse files
docs(relay): inbox run details live on the store spans
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
1 parent 2a55e71 commit b416cf3

1 file changed

Lines changed: 18 additions & 10 deletions

File tree

‎docs/operations/relay-observability.md‎

Lines changed: 18 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -70,12 +70,14 @@ Cloudflare rate limiter was unavailable and the request went through unlimited.
7070

7171
Held requests are handled in the endpoint's `HookInboxObject`, and each call into it is its own
7272
trace, not a child of the forward span: `relay.inbox.hold`, `relay.inbox.wake`, and
73-
`relay.inbox.deliver` for each alarm run. Join them to forwards on the inbox, which is named by
74-
endpoint key. A hold the inbox refused carries `relay.inbox.refused` (`max_per_hook`,
75-
`max_requests`, `max_bytes`, or `already_held`). A delivery run carries `relay.inbox.run_result`
76-
(`drained`, `more_pending`, `busy`, `unreachable`, or `failed`), how many requests it sent and
77-
delivered, the consecutive failures behind the retry delay, the longest time a delivered request
78-
waited, and the backlog left.
73+
`relay.inbox.deliver` for each alarm run. Each carries `relay.hook.endpoint_key`, so they join to
74+
forwards. The details are on the child spans: `HookInboxStore.hold` carries
75+
`relay.inbox.refused` when the inbox refused a request (`max_per_hook`, `max_requests`,
76+
`max_bytes`, or `already_held`). `HookInboxStore.deliverDue` carries `relay.inbox.run_result`
77+
(`drained`, `more_pending`, `busy`, or `unreachable`), how many requests it sent and delivered,
78+
the consecutive failures behind the retry delay, the longest time a delivered request waited, and
79+
the backlog left. A run that errored outright has `relay.inbox.run_result = failed` on the root
80+
`relay.inbox.deliver` span instead.
7981

8082
Questions these answer:
8183

@@ -85,11 +87,17 @@ Questions these answer:
8587
| where name == 'relay.hooks.forward'
8688
| summarize count() by bin(_time, 1h), outcome = tostring(['attributes.custom']['relay.hook.outcome'])
8789
88-
// Inboxes stuck behind an unreachable environment
90+
// Inboxes stuck behind an unreachable environment, by endpoint
8991
['t3-code-relay-traces-prod']
90-
| where name == 'relay.inbox.deliver'
92+
| where name == 'HookInboxStore.deliverDue'
9193
| extend c = ['attributes.custom']
9294
| where tostring(c['relay.inbox.run_result']) == 'unreachable'
93-
| summarize runs = count(), failures = max(toint(c['relay.inbox.consecutive_failures'])),
94-
backlog = max(toint(c['relay.inbox.held_count'])) by endpoint = tostring(c['relay.hook.endpoint_key'])
95+
| project trace_id, failures = toint(c['relay.inbox.consecutive_failures']),
96+
backlog = toint(c['relay.inbox.held_count'])
97+
| join kind=inner (
98+
['t3-code-relay-traces-prod']
99+
| where name == 'relay.inbox.deliver'
100+
| project trace_id, endpoint = tostring(['attributes.custom']['relay.hook.endpoint_key'])
101+
) on trace_id
102+
| summarize runs = count(), failures = max(failures), backlog = max(backlog) by endpoint
95103
```

0 commit comments

Comments
 (0)