Skip to content

feat: extend secret references to all plugins with central resolution#13312

Merged
nic-6443 merged 29 commits into
apache:masterfrom
nic-6443:feat/central-secret-refs
Apr 30, 2026
Merged

feat: extend secret references to all plugins with central resolution#13312
nic-6443 merged 29 commits into
apache:masterfrom
nic-6443:feat/central-secret-refs

Conversation

@nic-6443

@nic-6443 nic-6443 commented Apr 28, 2026

Copy link
Copy Markdown
Member

Description

Extend secret references ($secret://, $env://) to work with all plugins, SSL certificates, and consumer auth configurations automatically, removing the need for each plugin to call fetch_secrets() individually.

Changes

  1. Central secret resolution in plugin.filter(): Added resolve_plugin_conf() with a weak-keyed cache that resolves all secret references in any plugin configuration before plugin execution. Removed explicit fetch_secrets() calls from jwt-auth, authz-keycloak, openid-connect, csrf, and limit-count plugins.

  2. jsonschema skip_validation hook: Leverages the new skip_validation(value, schema) hook in jsonschema v0.9.13 to automatically bypass schema validation constraints (enum, pattern, minLength, maxLength, format) for $secret:// / $env:// values. This allows secret references in any string field without schema validation failures.

  3. Removed legacy SSL oneOf workaround: The SSL schema previously used oneOf: [{certificate_scheme}, {secret_uri_schema}] to accept both real certificates and secret URIs. This workaround is no longer needed — the jsonschema hook handles it automatically. SSL cert/key/certs/keys fields now use plain certificate_scheme / private_key_schema directly.

  4. check_schema guards: Added is_secret_ref bypass in plugins with Lua-based validation that would reject $secret:// values: response-rewrite, proxy-rewrite, redirect, ai-proxy, ai-proxy-multi, openid-connect, request-validation, jwt-auth.

  5. Documentation: Updated English and Chinese secret manager docs to describe the full supported scope (all plugin string fields, SSL cert/key, consumer auth fields).

Supported scope

  • Plugin configurations: Any string field in any plugin
  • SSL certificates: cert, key, certs, keys fields
  • Consumer auth configurations: Any string field in auth plugin configs

Fixes #13493

Copilot AI review requested due to automatic review settings April 28, 2026 09:39
@dosubot dosubot Bot added size:XL This PR changes 500-999 lines, ignoring generated files. enhancement New feature or request labels Apr 28, 2026

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR centralizes $secret:// / $env:// reference resolution so all plugins automatically receive resolved values, and updates docs/tests to reflect the new behavior.

Changes:

  • Resolve secret references for all HTTP/stream plugin confs in plugin.filter() / plugin.stream_filter() using a weak-key cache for stable resolved table identities.
  • Add secret-ref detection/collection helpers in apisix.secret, and remove plugin-local fetch_secrets() calls from several plugins.
  • Bypass schema constraints for secret-ref fields during config load; add new tests and update secret terminology docs.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 7 comments.

Show a summary per file
File Description
apisix/plugin.lua Central secret resolution + cache; adds secret-aware schema validation bypass logic.
apisix/secret.lua Adds is_secret_ref, has_secret_ref, and collect_secret_values helpers used by central resolver/validator.
apisix/plugins/openid-connect.lua Removes per-plugin secret resolution in rewrite.
apisix/plugins/limit-count.lua Removes per-plugin secret resolution in access.
apisix/plugins/clickhouse-logger.lua Removes per-call secret resolution in log sending path.
apisix/plugins/authz-keycloak.lua Removes per-plugin secret resolution in access.
apisix/plugins/ai-aws-content-moderation.lua Removes per-plugin secret resolution in rewrite.
t/secret/central-secret-refs.t Adds test coverage for centrally resolved secret refs and schema-bypass behavior.
docs/en/latest/terminology/secret.md Documents secret refs working for any plugin + schema-bypass behavior.
docs/zh/latest/terminology/secret.md Same as EN docs, for ZH.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread apisix/plugin.lua Outdated
Comment thread apisix/plugin.lua Outdated
Comment thread t/secret/central-secret-refs.t
Comment thread t/secret/central-secret-refs.t Outdated
Comment thread docs/en/latest/terminology/secret.md Outdated
Comment thread docs/zh/latest/terminology/secret.md Outdated
Comment thread apisix/plugin.lua Outdated
@nic-6443 nic-6443 changed the title feat: centralize secret reference resolution for all plugins feat: extend secret references to all plugins with central resolution Apr 29, 2026
@nic-6443
nic-6443 force-pushed the feat/central-secret-refs branch from c6d6f54 to 39ae26b Compare April 29, 2026 07:40
nic-6443 added 22 commits April 30, 2026 16:19
Previously, only a handful of plugins (authz-keycloak, openid-connect,
limit-count, clickhouse-logger, ai-aws-content-moderation) explicitly
called fetch_secrets() to resolve $secret:// and $env:// references.
Plugins that didn't call fetch_secrets() would silently ignore secret
references, causing confusing failures.

This commit makes secret references work automatically for ALL plugins
by injecting resolution at the end of plugin.filter() and
stream_filter(). Key design decisions:

1. Single injection point in plugin.filter() covers all code paths
   (HTTP requests, consumer merge, global rules, stream plugins).

2. Stable table references via weak-keyed cache: on most requests
   (no secret rotation), resolved values haven't changed, so the
   same resolved table reference is returned. This preserves
   plugin-internal caching that uses conf as cache key.

3. Schema validation is made secret-aware: when a plugin conf
   contains secret refs, the validator strips those fields from
   both conf and schema copies before checking, so constraints
   like enum/pattern/minLength/maxLength don't reject the
   $secret://... placeholder strings.

4. Remove explicit fetch_secrets() calls from 5 plugins that
   previously had them.

Non-plugin call sites (consumer.lua, radixtree_sni.lua) are
unchanged as they operate on different code paths.
- Extract strip_secret_refs as module-level function for reuse
- Add secret-aware validation to stream_check_schema
- Handle scalar array items (string arrays with secret refs)
- Fix test: add no_shuffle(), remove unused require, improve assertions
Instead of bypassing plugin_obj.check_schema() entirely when secret
refs are detected, move the strip logic into core.schema.check().
This preserves all custom code validation in each plugin's
check_schema (regex_uri compilation, headers checks, group field
validation, claim_schema, etc.) while only bypassing jsonschema
constraints for the specific fields containing secret references.
- Handle anyOf/oneOf/allOf at property level for string type detection
- Recurse into object-typed branches of anyOf/oneOf/allOf
- Adjust minItems when stripping secret ref elements from arrays
- Add warning log when secret resolution fails
- Add code comment explaining stable reference cache rationale
- Remove empty more_headers directive in test
- Adjust minProperties when stripping object properties (fixes
  proxy-rewrite and nested object validations)
- Extend strip_required to handle if/then/else conditional schemas
  (fixes SSL cert/key secret ref validation)
openid-connect check_schema requires either bearer_only=true or
session.secret to be set.
- Rename TEST 8/9 to match actual assertions
- Add TEST 11: limit-count redis_host as env ref exercises if/then
  conditional required stripping
- Renumber old TEST 11 to TEST 12
When core.schema.check() validates a copy of the json (to strip
secret ref fields before jsonschema validation), the validator sets
default values on the copy but not the original. This caused:
- upstream.scheme defaulting to nil instead of 'http' (balancer crash)
- plugin defaults like redis_database not being set
- SSL and other resource defaults being lost

Add merge_defaults() to copy newly-set fields from the validated copy
back to the original json after successful validation.
…handle dependencies keyword

- has_secret_ref and collect_secret_values now use a visited set to
  prevent stack overflow on tables with circular references
- strip_required now recurses into 'dependencies' keyword to handle
  schemas like jwt-auth that use dependencies.field.oneOf[].required
Several plugins perform Lua-level validation in check_schema that
would reject $secret:// or $env:// reference strings:
- jwt-auth: base64 decode of secret
- response-rewrite: base64 decode of body
- proxy-rewrite: regex compilation of regex_uri elements
- redirect: regex compilation of regex_uri elements
- ai-proxy: JSON decode of gcp service_account_json
- ai-proxy-multi: JSON decode of gcp service_account_json
- openid-connect: jsonschema compilation of claim_schema
- request-validation: schema validation of body_schema/header_schema

Add is_secret_ref() guard before each check so secret reference
strings pass through check_schema and get resolved at runtime.
…lidation hook

Replace the complex schema stripping approach (deepcopy + strip_secret_refs +
strip_required + merge_defaults, ~180 lines) with a single skip_validation
hook in jsonschema. The hook tells the validator to accept secret reference
strings as-is, bypassing type/enum/format/pattern constraints since they
will be resolved at runtime.

Also bump jsonschema dependency from 0.9.9 to 0.9.13 which adds the
skip_validation feature.
The jsonschema skip_validation hook now passes (value, schema) instead of
just (value). Update is_secret_ref to accept the schema parameter.
Also fix stale test descriptions referencing old 'stripping' approach.
The skip_validation hook should only bypass validation when the schema
expects a string type. For non-string fields (integer, boolean, etc.),
secret ref strings are not allowed at schema validation time.
For string-typed schemas, only skip validation if the value would fail a
constraint (enum, pattern, minLength, format). This prevents breaking
oneOf validators (like SSL key/cert) where one branch already accepts
secret ref patterns natively.

For non-string schemas (integer, boolean, etc.), always skip since a
string secret ref would fail the type check.
The jsonschema skip_validation hook now handles secret references
automatically, making the oneOf workaround in SSL cert/key fields
unnecessary. Simplify is_secret_ref to a straightforward prefix check.

Also update secret docs to describe the full supported scope:
- All plugin config string fields
- SSL cert/key/certs/keys fields
- Consumer auth config string fields
ed25519 private keys in PEM format are ~119 bytes, shorter than
the previous minLength of 128. Lower to 64 for future compatibility.
Only skip validation when the schema declares type='string' or has no
explicit type constraint. Fields with type='object', 'integer', etc.
will no longer be bypassed even if the value looks like a secret ref.
In redirect.lua and proxy-rewrite.lua, still validate the regex pattern
when only the replacement is a secret ref (using empty string as test
replacement). Only skip validation when the pattern itself is a secret ref.
Workflow plugin calls action handlers directly, bypassing plugin.filter()
where central secret resolution happens. Add fetch_secrets() in workflow
handlers for limit-conn and limit-count to ensure secret refs are resolved
when these plugins are used as workflow actions.
fetch_secrets() always returns a table when given a table input, so the
nil check was unreachable dead code.
Add _no_secret_ref weak-keyed cache so configs without secret refs are
only scanned once. Subsequent requests skip has_secret_ref() entirely.
@nic-6443
nic-6443 merged commit c22058c into apache:master Apr 30, 2026
22 checks passed
@nic-6443
nic-6443 deleted the feat/central-secret-refs branch April 30, 2026 10:33
hiades-devops pushed a commit to hiades-devops/apisix that referenced this pull request Jun 15, 2026
- Remove no-op ssl_trusted_certificate option from resty.http request
  (the api7 fork only honors ssl_verify/ssl_server_name/ssl_send_status_req);
  document that apisix.ssl.ssl_trusted_certificate must be set in config.yaml
  to the cluster CA for in-cluster TLS verification to work
- Add optional 'endpoint' field to schema to allow a full base URL override
  (e.g. http://127.0.0.1:1984 for test mocks), replacing the hardcoded https://
- Fix tests 8-12: use endpoint instead of kubernetes_host/kubernetes_port so
  plain HTTP mock servers are reachable without a TLS handshake
- Fix TEST 13: add service_account_file to manager registration so TEST 14
  does not fall back to the non-existent in-cluster token path in CI
- Revert authz-keycloak maxLength bump (100 -> 4096): the issue is already
  addressed on master since apache#13312 via skip_validation for secret refs
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request size:XL This PR changes 500-999 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: authz-keycloak encrypt_fields causes service load failure when used with Kubernetes Secret Manager ($secret://kubernetes)

5 participants