@@ -70,12 +70,14 @@ Cloudflare rate limiter was unavailable and the request went through unlimited.
7070
7171Held requests are handled in the endpoint's ` HookInboxObject ` , and each call into it is its own
7272trace, not a child of the forward span: ` relay.inbox.hold ` , ` relay.inbox.wake ` , and
73- ` relay.inbox.deliver ` for each alarm run. Join them to forwards on the inbox, which is named by
74- endpoint key. A hold the inbox refused carries ` relay.inbox.refused ` (` max_per_hook ` ,
75- ` max_requests ` , ` max_bytes ` , or ` already_held ` ). A delivery run carries ` relay.inbox.run_result `
76- (` drained ` , ` more_pending ` , ` busy ` , ` unreachable ` , or ` failed ` ), how many requests it sent and
77- delivered, the consecutive failures behind the retry delay, the longest time a delivered request
78- waited, and the backlog left.
73+ ` relay.inbox.deliver ` for each alarm run. Each carries ` relay.hook.endpoint_key ` , so they join to
74+ forwards. The details are on the child spans: ` HookInboxStore.hold ` carries
75+ ` relay.inbox.refused ` when the inbox refused a request (` max_per_hook ` , ` max_requests ` ,
76+ ` max_bytes ` , or ` already_held ` ). ` HookInboxStore.deliverDue ` carries ` relay.inbox.run_result `
77+ (` drained ` , ` more_pending ` , ` busy ` , or ` unreachable ` ), how many requests it sent and delivered,
78+ the consecutive failures behind the retry delay, the longest time a delivered request waited, and
79+ the backlog left. A run that errored outright has ` relay.inbox.run_result = failed ` on the root
80+ ` relay.inbox.deliver ` span instead.
7981
8082Questions these answer:
8183
@@ -85,11 +87,17 @@ Questions these answer:
8587| where name == 'relay.hooks.forward'
8688| summarize count() by bin(_time, 1h), outcome = tostring(['attributes.custom']['relay.hook.outcome'])
8789
88- // Inboxes stuck behind an unreachable environment
90+ // Inboxes stuck behind an unreachable environment, by endpoint
8991['t3-code-relay-traces-prod']
90- | where name == 'relay.inbox.deliver '
92+ | where name == 'HookInboxStore.deliverDue '
9193| extend c = ['attributes.custom']
9294| where tostring(c['relay.inbox.run_result']) == 'unreachable'
93- | summarize runs = count(), failures = max(toint(c['relay.inbox.consecutive_failures'])),
94- backlog = max(toint(c['relay.inbox.held_count'])) by endpoint = tostring(c['relay.hook.endpoint_key'])
95+ | project trace_id, failures = toint(c['relay.inbox.consecutive_failures']),
96+ backlog = toint(c['relay.inbox.held_count'])
97+ | join kind=inner (
98+ ['t3-code-relay-traces-prod']
99+ | where name == 'relay.inbox.deliver'
100+ | project trace_id, endpoint = tostring(['attributes.custom']['relay.hook.endpoint_key'])
101+ ) on trace_id
102+ | summarize runs = count(), failures = max(failures), backlog = max(backlog) by endpoint
95103```
0 commit comments