feat: add core.response.get_response_source() API for response origin classification by nic-6443 · Pull Request #13224 · apache/apisix

nic-6443 · 2026-04-14T07:42:45Z

Summary

Add core.response.get_response_source(ctx) API to classify HTTP responses as:

"apisix" — generated by APISIX Lua code (route not found, plugin rejection, core.response.exit())
"nginx" — error from NGINX proxy module (connection refused, upstream timeout)
"upstream" — real HTTP response from the upstream service

Also add core.response.set_response_source(ctx, source) for plugins that bypass NGINX proxy (e.g. ai-proxy).

Detection mechanism

Uses ngx.var.upstream_header_time (not ctx.var) to detect whether upstream sent response headers:

"-" = no headers received (connection refused, timeout) → "nginx"
numeric value = headers received → "upstream"

Why ngx.var instead of ctx.var: lua-var-nginx-module's FFI path clamps header_time from -1 to 0 via ngx_max(ms, 0), losing the "-" sentinel that NGINX uses to distinguish "no response" from "fast response". ngx.var preserves the raw string.

Integration with observability plugins

OTel: Added apisix.response_source span attribute; use ngx.status instead of upstream_status for span status code
Prometheus: Added response_source label to http_status metric
Zipkin: Added apisix.response_source span tag; use ngx.status for status tag

Tests

19 test cases covering:

Parser helper unit tests (get_last_upstream_token) for single values, retries, spaces, colons, empty/nil
resp_exit marking, set_response_source priority
Integration: upstream 200, upstream 502, connection refused, plugin rejection

Add core.response.get_response_source(ctx) API that returns one of three values to identify where a response originated: - "apisix": response generated by APISIX Lua code (route not found, plugin rejection, upstream not configured, etc.) - "nginx": error generated by NGINX proxy module (connection refused, read timeout, DNS resolution failure, etc.) - "upstream": real HTTP response returned by the upstream service Implementation: - core.response.exit() now marks ctx._resp_source = "apisix" for 4xx/5xx - init.lua sets ctx._apisix_proxied = true after before_proxy phase, before proxy_pass dispatch - get_response_source() uses the last token of $upstream_header_time to distinguish NGINX errors (header_time = "-") from upstream responses (header_time has numeric value) Plugin updates: - opentelemetry: adds apisix.response_source span attribute, uses ngx.status for http.status_code (fixes nil status for gateway errors) - prometheus: adds response_source label to http_status counter - zipkin: adds apisix.response_source span tag, uses ngx.status

Copilot

Pull request overview

Adds a new core.response.get_response_source(ctx) API and propagates its classification into tracing (OpenTelemetry/Zipkin) and Prometheus metrics, enabling users to distinguish gateway-generated errors from upstream responses.

Changes:

Introduces response-source classification (apisix / nginx / upstream) in core.response, with _resp_source and _apisix_proxied markers.
Updates OpenTelemetry and Zipkin spans to tag apisix.response_source and to use ngx.status for status reporting.
Extends Prometheus http_status counter with a response_source label; adds a new core test file for the API.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
`apisix/core/response.lua`	Implements `get_response_source()` and marks `_resp_source` on `core.response.exit()` for error codes.
`apisix/init.lua`	Marks requests as proxied (`_apisix_proxied`) after `before_proxy` and before NGINX proxy dispatch.
`apisix/plugins/opentelemetry.lua`	Adds `apisix.response_source` span attribute and switches status reporting to `ngx.status`.
`apisix/plugins/zipkin.lua`	Adds `apisix.response_source` span tag and switches status reporting to `ngx.status`.
`apisix/plugins/prometheus/exporter.lua`	Adds `response_source` label to the `http_status` counter and emits it in log phase.
`t/core/response-source.t`	Adds tests for `get_response_source()` and `_resp_source` behavior.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

- Use gmatch(%S+) loop instead of str_match pattern to correctly handle spaces in comma-separated upstream_header_time (e.g. '- , -') - Guard ctx.var access with nil check for defensive safety - Improve integration tests to verify response_source via serverless-pre-function log phase plugin - Add tests for spaced separators and nil ctx.var edge cases

ctx.var may return upstream_header_time as a number (not string) for single numeric values. Apply tostring() before gmatch to avoid 'attempt to index local (a number value)' error.

…EST 17 - opentelemetry4-bugfix-pb-state.t TEST 3: add expected apisix.response_source attribute - response-source.t TEST 17: add diagnostic logging to debug connection refused case

The new response_source label in apisix_http_status metric needs to be accounted for in existing test assertions.

…var issues

Add core.response.set_response_source(ctx, source) API for plugins that bypass NGINX proxy (e.g. ai-proxy) to explicitly mark whether the response came from upstream. The ai-proxy base now sets source to 'upstream' when the LLM provider responds, ensuring correct classification even for 429/5xx responses from upstream.

Switch from ngx.var (which broke unit tests using fake ctx) to pcall-wrapped ctx.var access. This handles both unit tests with mock objects and integration tests where resty.ngxvar may error in NGINX error paths.

…e value

Also adds diagnostic logging in TEST 17 to debug connection refused classification.

$upstream_header_time returns 0 (not "-") for connection refused, making it impossible to distinguish from fast upstream responses. $upstream_bytes_received is 0 when no data was received from upstream (connection refused, timeout) and > 0 when upstream sent a response, providing a reliable signal for response source classification.

…es_received Switch back to $upstream_header_time but read via ngx.var directly instead of ctx.var. The lua-var-nginx-module FFI path clamps header_time -1 to 0 via ngx_max(ms, 0), making connection refused indistinguishable from fast upstream responses. ngx.var preserves the "-" sentinel. Changes: - Extract get_last_upstream_token() helper for parsing comma-separated NGINX upstream variable strings (unit-testable) - Use ngx_var.upstream_header_time in get_response_source() - Cache computed result in ctx._resp_source for subsequent calls - Replace mock-based unit tests with: - Parser helper unit tests (TEST 4-10) - Integration tests covering upstream 502, plugin rejection (TEST 18-19) - Add TEST 15 verifying set_response_source is not overridden by resp_exit

… classification (apache#13224)

Copilot AI review requested due to automatic review settings April 14, 2026 07:42

dosubot Bot added size:L This PR changes 100-499 lines, ignoring generated files. enhancement New feature or request labels Apr 14, 2026

Copilot started reviewing on behalf of nic-6443 April 14, 2026 07:43 View session

Copilot AI reviewed Apr 14, 2026

View reviewed changes

Comment thread apisix/core/response.lua Outdated

Comment thread apisix/core/response.lua Outdated

Comment thread t/core/response-source.t Outdated

Comment thread t/core/response-source.t Outdated

nic-6443 added 14 commits April 14, 2026 15:48

fix: handle numeric upstream_header_time from ctx.var

144525f

ctx.var may return upstream_header_time as a number (not string) for single numeric values. Apply tostring() before gmatch to avoid 'attempt to index local (a number value)' error.

fix: localize tostring to satisfy lj-releng lint check

35a4097

fix: update OTel test expected output and add diagnostic logging to T…

73d056d

…EST 17 - opentelemetry4-bugfix-pb-state.t TEST 3: add expected apisix.response_source attribute - response-source.t TEST 17: add diagnostic logging to debug connection refused case

fix: update prometheus test patterns to include response_source label

c1c2495

The new response_source label in apisix_http_status metric needs to be accounted for in existing test assertions.

fix: finalize TEST 17 with proper nginx assertion for connection refused

fa15b77

fix: add no_shuffle() to response-source test suite

2f2005c

fix: use ngx.var directly for upstream_header_time to avoid resty.ngx…

592294d

…var issues

fix: use pcall for ctx.var.upstream_header_time to handle error paths

aa090f1

Switch from ngx.var (which broke unit tests using fake ctx) to pcall-wrapped ctx.var access. This handles both unit tests with mock objects and integration tests where resty.ngxvar may error in NGINX error paths.

debug: use grep_error_log in TEST 17 to capture actual response_sourc…

6eaa9a9

…e value

fix: set _resp_source for all status codes, not just errors

000291c

Also adds diagnostic logging in TEST 17 to debug connection refused classification.

debug: log raw upstream variable values for connection refused diagnosis

b190dd4

nic-6443 changed the title ~~feat: add response source differentiation for gateway vs upstream errors~~ feat: add core.response.get_response_source() API for response origin classification Apr 14, 2026

nic-6443 requested review from AlinsRan, Baoyuantop, membphis, moonming and shreemaan-abhishek April 15, 2026 00:54

membphis reviewed Apr 15, 2026

View reviewed changes

Comment thread apisix/core/response.lua Outdated

membphis requested changes Apr 15, 2026

View reviewed changes

dosubot Bot removed the size:L This PR changes 100-499 lines, ignoring generated files. label Apr 15, 2026

dosubot Bot added the size:XL This PR changes 500-999 lines, ignoring generated files. label Apr 15, 2026

nic-6443 requested a review from membphis April 16, 2026 01:15

membphis approved these changes Apr 16, 2026

View reviewed changes

shreemaan-abhishek approved these changes Apr 16, 2026

View reviewed changes

AlinsRan approved these changes Apr 16, 2026

View reviewed changes

nic-6443 merged commit ddddeaf into apache:master Apr 16, 2026
27 of 29 checks passed

nic-6443 deleted the feat/upstream-error-differentiation branch April 16, 2026 02:14

wistefan pushed a commit to wistefan/apisix that referenced this pull request Jun 16, 2026

feat: add core.response.get_response_source() API for response origin…

47e535e

… classification (apache#13224)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: add core.response.get_response_source() API for response origin classification#13224

feat: add core.response.get_response_source() API for response origin classification#13224
nic-6443 merged 16 commits into
apache:masterfrom
nic-6443:feat/upstream-error-differentiation

nic-6443 commented Apr 14, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

nic-6443 commented Apr 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Detection mechanism

Integration with observability plugins

Tests

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

nic-6443 commented Apr 14, 2026 •

edited

Loading