Skip to content

feat: add data-mask plugin#13347

Merged
AlinsRan merged 10 commits into
apache:masterfrom
AlinsRan:feat/data-mask-plugin
May 15, 2026
Merged

feat: add data-mask plugin#13347
AlinsRan merged 10 commits into
apache:masterfrom
AlinsRan:feat/data-mask-plugin

Conversation

@AlinsRan

@AlinsRan AlinsRan commented May 9, 2026

Copy link
Copy Markdown
Contributor

Summary

The data-mask plugin masks or redacts sensitive fields in HTTP request data — query parameters, headers, and body — before they appear in access logs or logger plugin outputs (e.g. file-logger, http-logger, kafka-logger).

Problem

API gateways sit in front of every service and naturally observe all request data, including credentials, tokens, PII, and payment card numbers. When access logging is enabled, this sensitive data is routinely written to log files or shipped to log aggregation systems, creating compliance risk under GDPR, PCI-DSS, HIPAA, and similar regulations.

Currently, APISIX has no built-in mechanism to sanitize sensitive fields before logging. Operators must either:

  • Disable logging of request data entirely, losing visibility
  • Post-process logs after the fact, which is operationally complex and leaves a window of exposure
  • Implement masking in each upstream service, which is inconsistent

Scenarios

Scenario 1 — Credential sanitization in query parameters

A legacy API accepts credentials as query parameters (e.g. /api?password=secret&token=abc). With access logging enabled, these values appear in the request URI logged by every logger plugin. data-mask can remove the password field and replace the token with ***** before any logger sees the request.

Scenario 2 — Authorization header redaction

A service passes Authorization: Bearer <token> on every request. Log pipelines shipped to third-party SIEM systems must not contain bearer tokens. data-mask removes or redacts the header in the log phase; upstream communication is unaffected.

Scenario 3 — Partial masking of payment card numbers

An e-commerce API receives card numbers in JSON bodies or query strings. PCI-DSS requires only the first 6 and last 4 digits to be visible. data-mask's regex action can transform 1234-5678-9012-34561234-****-****-3456 using a single rule.

Scenario 4 — Nested JSON body field masking via JSONPath

A batch API receives a JSON body with user records, each containing a token and nested credit.card. JSONPath expressions ($.users[*].token, $.users[*].credit.card) let a single rule mask all matching nested fields without enumerating each one.

Design

The plugin runs in the log phase and modifies the request context (ctx.var) used by logger plugins. Upstream communication is unaffected — the plugin never alters the actual request sent to the upstream.

Three masking actions are supported:

  • remove — deletes the field entirely from the logged data
  • replace — substitutes the field value with a fixed string
  • regex — applies an ngx.re.sub regex substitution; capture groups are available as $1, $2, etc.

Three field types are supported:

  • query — URL query parameters
  • header — HTTP request headers
  • body — request body (json format supports JSONPath; urlencoded for form data)

Example

Mask sensitive query parameters and log with file-logger:

admin_key=$(yq '.deployment.admin.admin_key[0].key' conf/config.yaml | sed 's/"//g')

curl http://127.0.0.1:9180/apisix/admin/routes/1 -H "X-API-KEY: $admin_key" -X PUT -d '
{
  "uri": "/anything",
  "plugins": {
    "data-mask": {
      "request": [
        { "type": "query", "name": "password", "action": "remove" },
        { "type": "query", "name": "token",    "action": "replace", "value": "*****" },
        { "type": "query", "name": "card",     "action": "regex",
          "regex": "(\\d+)\\-\\d+\\-\\d+\\-(\\d+)", "value": "$1-****-****-$2" }
      ]
    },
    "file-logger": { "path": "logs/access.log" }
  },
  "upstream": { "type": "roundrobin", "nodes": { "httpbin.org:80": 1 } }
}'

Send a request:

curl "http://127.0.0.1:9080/anything?password=secret&token=abc123&card=1234-5678-9012-3456"

The logged URI in logs/access.log will be:

/anything?token=*****&card=1234-****-****-3456

The password field is absent; token is replaced; card digits are partially masked.

Changes

  • apisix/plugins/data-mask.lua: Plugin implementation
  • conf/config.yaml.example: Register plugin at priority 1500
  • apisix/cli/config.lua: Add to default plugin list
  • docs/en/latest/plugins/data-mask.md: English documentation with full examples
  • docs/zh/latest/plugins/data-mask.md: Chinese documentation

Checklist

  • I have explained the need for this PR and the problem it solves
  • I have explained the changes or the new features added to this PR
  • I have added tests corresponding to this change
  • I have updated the documentation to reflect this change
  • I have verified that this change is backward compatible

The data-mask plugin masks sensitive fields in request data (query parameters,
headers, and body) before they appear in access logs or logger plugins.

It supports three masking actions:
- remove: completely remove the field
- replace: replace the field value with a fixed string
- regex: apply a regex substitution to the field value

For body masking, it supports both JSON (via JSONPath) and URL-encoded formats.

Co-authored-by: Copilot <[email protected]>
@dosubot dosubot Bot added size:XXL This PR changes 1000+ lines, ignoring generated files. enhancement New feature or request labels May 9, 2026
nic-6443
nic-6443 previously approved these changes May 12, 2026
…king

The data-mask plugin writes masked query params to ctx.var.request_line
so that access logs reflect the redacted request line. Three supporting
changes are required:

- apisix/cli/ngx_tpl.lua: declare $request_line nginx variable in both
  http server blocks so nginx knows the variable exists
- apisix/core/ctx.lua: register request_line as a writable ctx.var field
- apisix/init.lua: copy the read-only $request nginx variable to the
  writable $request_line at the start of the access phase

Also remove the EE-specific uri_before_strip fallback in data-mask.lua
(strip_path_prefix is an EE-only feature; upstream always uses ctx.var.uri).

Co-authored-by: Copilot <[email protected]>
Foo Bar and others added 3 commits May 12, 2026 18:20
Add two new tests:
- TEST 15: create a route with data-mask removing 'password' and
  replacing 'token' query parameters
- TEST 16: verify that the access log records the masked request line
  (password absent, token replaced with *****) via $request_line

Also update the main log_format in t/APISIX.pm to use $request_line
instead of $request, matching the EE configuration. $request_line is a
writable copy of the read-only $request variable initialized in the
access phase so that data-mask (and other plugins) can update it.

Co-authored-by: Copilot <[email protected]>
- Add `set $request_line ''` declarations to server blocks in t/APISIX.pm
  to fix nginx startup failure (unknown variable error)
- Add warn log when body masking is skipped due to max_body_size limit
- Use `uri_before_strip or uri` for request_uri to handle proxy-rewrite
  URI stripping correctly

Co-authored-by: Copilot <[email protected]>
get_http_version() returns a number (e.g. 1.1), not the string
"HTTP/1.1", so the prefix must be prepended explicitly.

Co-authored-by: Copilot <[email protected]>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a new data-mask plugin to redact/mask sensitive request data (query params, headers, body) before it is emitted by APISIX logging surfaces, and wires it into the default plugin list, docs, and test suite.

Changes:

  • Introduces apisix/plugins/data-mask.lua implementing masking actions (remove/replace/regex) for query/header/body (JSONPath + urlencoded).
  • Adds request_line ctx var support so access log formats can log a maskable request line, and updates test harness log format accordingly.
  • Adds documentation (EN/ZH), config registration, and a comprehensive test file covering query/header/body + access-log masking.

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 10 comments.

Show a summary per file
File Description
apisix/plugins/data-mask.lua New plugin implementation to redact request query/header/body for log outputs.
apisix/init.lua Initializes ctx.var.request_line from $request so it can be overridden later for masked access-log output.
apisix/core/ctx.lua Allows request_line to be stored in ctx.var.
apisix/cli/ngx_tpl.lua Declares $request_line Nginx var in generated config so it can be set from Lua.
apisix/cli/config.lua Adds data-mask to the default enabled plugin list.
conf/config.yaml.example Registers data-mask in the example plugin priority list.
docs/en/latest/plugins/data-mask.md English plugin documentation and usage examples.
docs/zh/latest/plugins/data-mask.md Chinese plugin documentation and usage examples.
docs/en/latest/config.json Adds the new plugin doc page to the English docs sidebar/config.
docs/zh/latest/config.json Adds the new plugin doc page to the Chinese docs sidebar/config.
t/plugin/data-mask.t New test coverage for query/header/body masking + access-log masking behavior.
t/APISIX.pm Test harness log format switched to $request_line to validate access-log masking.
t/admin/plugins.t Ensures data-mask is present in the admin plugins list test.
Comments suppressed due to low confidence (1)

t/plugin/data-mask.t:664

  • In the failure branch, this test prints match which is never defined (likely meant to print match10/match11 or the request body). This makes debugging failures harder if the assertion fails.
            local match10, err = ngx.re.match(log.request.body, "arg10=10")
            local match11, err = ngx.re.match(log.request.body, "arg11=11")
            os.remove("mask-urlencoded-body.log")
            if match10 and not match11 then
                ngx.say("success")
                return
            end
            ngx.say(match)
            ngx.say(err)


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread apisix/plugins/data-mask.lua Outdated
Comment thread apisix/plugins/data-mask.lua
Comment thread apisix/plugins/data-mask.lua
Comment thread apisix/plugins/data-mask.lua Outdated
Comment thread apisix/plugins/data-mask.lua Outdated
Comment thread docs/en/latest/plugins/data-mask.md
Comment thread docs/zh/latest/plugins/data-mask.md
Comment thread t/plugin/data-mask.t Outdated
Comment thread t/plugin/data-mask.t Outdated
Comment thread apisix/plugins/data-mask.lua
- Do not log sensitive values in regex_replace error messages
- Handle multi-value query/form params in mask_table (regex action)
- Fix mask_json to skip regex on non-string values with a warning
- Fix mask_json to not assign nil when regex_replace fails
- Fix header regex action to not clear header when regex fails
- Defer get_post_args until a urlencoded rule is actually matched
- Add warn log when request body is stored in a temporary file
- Add docs note about access_log_format $request_line requirement
- Fix test: remove extra backtick in TEST 12 title
- Fix test: replace undefined 'match' variable with meaningful debug output

Co-authored-by: Copilot <[email protected]>
- regex_replace() now returns substitution count n; callers only set
  masked=true when n > 0 to avoid false-positive masking when the
  regex does not match
- mask_table(): add type guards before calling regex_replace() to handle
  valueless boolean query args (e.g. ?token) without crashing ngx.re.sub
- get_post_args(): capture and log error return; warn on truncation
- Header regex: only update header when substitution actually matched

Co-authored-by: Copilot <[email protected]>
@AlinsRan
AlinsRan merged commit ece5cca into apache:master May 15, 2026
25 of 30 checks passed
@AlinsRan
AlinsRan deleted the feat/data-mask-plugin branch May 15, 2026 00:51
wistefan pushed a commit to wistefan/apisix that referenced this pull request Jun 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request size:XXL This PR changes 1000+ lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants