Skip to content

TT-17297: support file based kv references#8267

Merged
vladzabolotnyi merged 24 commits into
masterfrom
TT-17297
Jun 15, 2026
Merged

TT-17297: support file based kv references#8267
vladzabolotnyi merged 24 commits into
masterfrom
TT-17297

Conversation

@vladzabolotnyi

@vladzabolotnyi vladzabolotnyi commented Jun 2, 2026

Copy link
Copy Markdown
Contributor

Description

Ticket: https://tyktech.atlassian.net/browse/TT-17297

Adds a file-based KV provider to Tyk Gateway so that KV references in gateway config and API definitions can resolve to the contents of local files. This is the primary mechanism for consuming Kubernetes secrets mounted as volume files on data-plane pods - enabling certificates and tokens to be referenced by name without ever storing private key material in the management plane.

Related Issue

Part of a two-ticket solution:
TT-17297 (this PR) — file-based KV provider: file:// URI scheme for load-time resolution, $secret_file. prefix for runtime resolution
TT-17295 (dependency) — PEM string support in API definitions: allows a certificate field to contain the raw PEM content rather than a file path

The mTLS client certificate use case (K8s secret → API definition → mTLS handshake) requires both PRs to be merged. This PR resolves the KV reference to the PEM content; TT-17295 makes the gateway accept that PEM content string as a certificate.

Motivation and Context

Currently the management plane stores private key material for every TLS certificate used in mTLS. This creates a security risk by centralising sensitive keys. The proposed solution: the management plane stores only a reference (file://client.crt), and the data-plane gateway resolves it from a K8s secret mounted as a volume at runtime - the private key never leaves the pod.

How This Has Been Tested

  1. Automated tests
  2. Manual testing

NOTE:

  1. I've done some manual testing with local setup and asked LLM to generate some guide file; attached the file to the ticket.
  2. To test pem cert value on api def e2e we have to combine changes from TT-17297 and TT-17295. Because raw cert pem value can be added to api def and code from TT-17295 make it work. I've created parent branch where both ticket's change are included: https://github.com/TykTechnologies/tyk/tree/TT-15079

DEMO

Given the api def with client cert added with using file:// syntax to allowed list. Then client send request with using valid cert, then cert is updated with new one and temporaly old cert is still valid until gateway is hot-reloaded. Then we trigger the hot-reload with api def update and old cert stopped working but new cert value was used without gateway process shutdown

Screen.Recording.2026-06-03.at.08.36.06.mov

Given the config file with file:// reference by using relative path

Screen.Recording.2026-06-03.at.13.27.27.mov

Given the config file with file:// reference by using absolute path

Screen.Recording.2026-06-03.at.14.00.41.mov

Given api def with rewrite URL pattern to add query param with file referencing

Screen.Recording.2026-06-03.at.13.55.00.mov

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Refactoring or add test (improvements in base code or adds test coverage to functionality)

Checklist

TODO: VALIDATE DOCS WITH ANDY

  • I ensured that the documentation is up to date
  • I explained why this PR updates go.mod in detail with reasoning why it's required
  • I would like a code coverage CI quality gate exception and have explained why

Ticket Details

TT-17297
Status In Code Review
Summary Support File Based KV Referencing.

Generated at: 2026-06-03 12:17:15

@github-actions

github-actions Bot commented Jun 2, 2026

Copy link
Copy Markdown
Contributor

🎯 Recommended Merge Targets

Based on JIRA ticket TT-17297: Support File Based KV Referencing.

Fix Version: Tyk 5.14.0

⚠️ Warning: Expected release branches not found in repository

Required:

  • master - No matching release branches found. Fix will be included in future releases.

📋 Workflow

  1. Merge this PR to master first

@probelabs

probelabs Bot commented Jun 2, 2026

Copy link
Copy Markdown
Contributor

This PR introduces a file-based Key-Value (KV) provider to the Tyk Gateway. This feature enables the gateway to resolve configuration values and secrets directly from local files, which is particularly useful for securely consuming Kubernetes secrets mounted as volumes. It enhances security by allowing private key material to remain on data-plane pods instead of being stored in the management plane.

Two distinct mechanisms for file-based referencing are implemented:

  1. Load-Time Resolution (file://...): Used in the main gateway configuration (tyk.conf) and API definitions. These references are resolved at gateway startup or during a hot reload.
  2. Request-Time Resolution ($secret_file.<key>): Used in middleware that transforms requests, such as URL rewrites or header injection, allowing for dynamic injection of file contents into the request/response flow.

A new configuration section, kv.file, is introduced with a base_path option to securely confine relative file lookups to a specific directory, preventing path traversal vulnerabilities.

Files Changed Analysis

The changes introduce a new, self-contained file KV provider and integrate it into key parts of the gateway:

  • New Core Logic: The new file gateway/kv_file.go and its comprehensive test suite gateway/kv_file_test.go form the core of this feature. They contain the ResolveFileKV function, which handles securely reading files, validating paths, preventing path traversal, and resolving Kubernetes-style AtomicWriter symlinks. These two files account for a significant portion of the new code (345 lines).
  • Configuration: config/config.go is updated with the FileConfig struct, and cli/linter/schema.json is modified to validate this new configuration section.
  • Integration Points:
    • gateway/server.go: The central kvStore function is updated to recognize and handle the file:// prefix, making this feature available to any part of the gateway configuration that uses it (e.g., mTLS certificates).
    • gateway/api_definition.go: The API loader now processes file:// URIs within API definitions, including JSON-escaping multi-line content like PEM certificates to ensure validity.
    • gateway/mw_url_rewrite.go: The ReplaceTykVariables function, used by various middleware, is extended to support the $secret_file. syntax for request-time injection.
  • Testing: Comprehensive test coverage is added across gateway/api_definition_test.go, gateway/kv_file_test.go, gateway/mw_url_rewrite_test.go, and gateway/server_test.go to validate the new functionality, including security checks for path traversal and symlink escapes.

Architecture & Impact Assessment

  • What this PR accomplishes: It adds a native file-based secret provider, reducing reliance on external KV stores like Consul or Vault and enabling a more secure secret management pattern for cloud-native deployments.
  • Key technical changes introduced:
    • A new ResolveFileKV function serves as the single, secure entry point for reading secrets from files.
    • Integration of file-based secret resolution at two distinct points in the lifecycle: gateway bootstrap/reload (load-time) and per-request middleware processing (request-time).
    • Introduction of a kv.file.base_path setting to establish a security boundary for file access.
  • Affected system components: The changes impact three core areas: Gateway Configuration Loading, the API Definition Loader, and the URL Rewrite/Header Injection Middleware.

Visualization

This diagram illustrates the two distinct flows for resolving file-based secrets:

graph TD
    subgraph "Load-Time Resolution (Gateway/API Config)"
        A[tyk.conf / api_definition.json] --|contains "file:///path/to/secret"|--> B{Config/API Loader};
        B -- calls --> C[gateway.ResolveFileKV];
        C -- reads --> D[Filesystem];
        B -- injects value --> E[Live Gateway Config / API Spec];
    end

    subgraph "Request-Time Resolution (Middleware)"
        F[HTTP Request] --> G{URL Rewrite Middleware};
        G --|template has "$secret_file.key"|--> H[gateway.ReplaceTykVariables];
        H -- calls --> C[gateway.ResolveFileKV];
        H -- injects value --> I[Rewritten URL / Header];
    end
Loading

Scope Discovery & Context Expansion

  • Broad Impact: By modifying the central kvStore and ReplaceTykVariables functions, this PR makes file-based secret resolution available across a wide range of gateway functionalities that rely on these mechanisms, not just the primary mTLS use case.
  • Performance Considerations: The introduction of $secret_file. adds filesystem I/O to the request hot path. While the OS page cache will mitigate the performance impact of repeated reads, this is a notable architectural change that introduces syscalls into request processing.
  • Security Context: The feature's security model is centered on the ResolveFileKV function in gateway/kv_file.go. It correctly uses filepath.Rel and filepath.IsLocal to prevent path traversal. It also handles filepath.EvalSymlinks to support Kubernetes secret volume mounts while re-validating the path after symlink resolution to prevent escape vulnerabilities. The base_path configuration is a critical security boundary, especially for the request-time $secret_file. feature, which is accessible to API designers.
Metadata
  • Review Effort: 4 / 5
  • Primary Label: feature

Powered by Visor from Probelabs

Last updated: 2026-06-15T06:12:16.540Z | Triggered by: pr_updated | Commit: 691325d

💡 TIP: You can chat with Visor using /visor ask <your question>

@probelabs

probelabs Bot commented Jun 2, 2026

Copy link
Copy Markdown
Contributor

Security Issues (3)

Severity Location Issue
🟠 Error gateway/kv_file.go:61
The function `ResolveFileKV` reads an entire file into memory using `os.ReadFile` without any size validation. This can be triggered at request-time by middleware using `$secret_file.` variables. A reference to a very large file (e.g., `/dev/zero`) could cause the gateway to exhaust its memory, leading to a Denial of Service (DoS) vulnerability.
💡 SuggestionImplement a size check before reading the file. Use `os.Stat` to get the file's size and return an error if it exceeds a reasonable, configurable limit (e.g., 1MB).
🟠 Error gateway/api_definition.go:615-619
When resolving a `file://` secret in an API definition, a file read error is logged but the process continues. The `file://` placeholder is left unresolved, which can lead to an insecure configuration being loaded. For example, if a JWT signing key is not loaded, the API might be deployed with signature validation effectively disabled. Failures to load critical secrets should be treated as fatal errors for that API definition.
💡 SuggestionModify `replaceFileSecrets` to return an error that propagates up to the API loader. The API loader should then fail to load the specific API definition, preventing it from being deployed in an insecure state. The loop should not `continue` on error; it should return immediately.
🟡 Warning gateway/api_definition.go:608-632
The `replaceFileSecrets` function, which handles `file://` URIs in API definitions, does not prevent the use of absolute file paths when `kv.file.base_path` is configured. This is inconsistent with the safer implementation for `$secret_file.` variables, which explicitly rejects absolute paths. This allows a user with permission to edit API definitions to read any file on the gateway's filesystem, potentially leading to information disclosure or privilege escalation.
💡 SuggestionAdd a check within the `replaceFileSecrets` loop to reject absolute paths when `base_path` is set. This will enforce the security boundary for all file-based secret resolution mechanisms.
// Inside the for loop in replaceFileSecrets
key := m[1]
if basePath != &#34;&#34; &amp;&amp; filepath.IsAbs(key) {
    err := fmt.Errorf(&#34;file KV: absolute path rejected for file:// key %q - base_path is configured&#34;, key)
    log.WithError(err).Warn(&#34;Failed to resolve file secret&#34;)
    if firstErr == nil {
        firstErr = err
    }
    continue
}
val, err := ResolveFileKV(basePath, key)
// ...

Architecture Issues (2)

Severity Location Issue
🟠 Error gateway/api_definition.go:571-573
The API definition loading process continues even if a `file://` secret cannot be resolved. The error is only logged, and the `file://` placeholder remains in the API definition data. This leads to a misconfigured and likely non-functional API being loaded, masking deployment or configuration errors. The system should adopt a "fail-safe" approach, where an API definition is rejected if its secret dependencies cannot be met.
💡 SuggestionPropagate the error from `replaceFileSecrets` up to the API loading functions. The loader should then treat this error as fatal for the specific API definition and prevent it from being loaded into the gateway, providing clear feedback that the configuration is invalid.
🟡 Warning gateway/mw_url_rewrite.go:247-252
The implementation of `$secret_file.` variables introduces blocking file system I/O into the request processing path (the "hot path"). While the OS page cache can mitigate the performance impact for frequently accessed files, this design adds syscall overhead to every request that uses this feature. This could become a performance bottleneck under very high throughput or when secrets are stored on slower (e.g., network-attached) storage.
💡 SuggestionThe current design is pragmatic for ensuring secret freshness. However, for performance-critical use cases, consider introducing an optional, short-lived in-memory cache for file contents. A configurable TTL (e.g., 1-5 seconds) could offer a balance between performance and timely secret rotation, significantly reducing syscalls on the hot path.

Performance Issues (1)

Severity Location Issue
🟡 Warning gateway/mw_url_rewrite.go:335-351
The request-time resolution of `$secret_file.` variables performs blocking file system I/O on every request. The `ResolveFileKV` function is called for each variable, which executes multiple syscalls for path and symlink resolution (`filepath.EvalSymlinks`) and file reading (`os.ReadFile`). While the OS page cache mitigates disk latency, the overhead of repeated syscalls on the request hot path can introduce latency and degrade gateway performance under high load.
💡 SuggestionIntroduce an in-process cache with a short TTL (e.g., 1-5 seconds) for file contents resolved via `$secret_file.`. This would significantly reduce the number of syscalls by serving subsequent requests from the cache, while still allowing the gateway to pick up rotated secrets in a timely manner. The cache would store the resolved file content, keyed by the secret path.

Security Issues (3)

Severity Location Issue
🟠 Error gateway/kv_file.go:61
The function `ResolveFileKV` reads an entire file into memory using `os.ReadFile` without any size validation. This can be triggered at request-time by middleware using `$secret_file.` variables. A reference to a very large file (e.g., `/dev/zero`) could cause the gateway to exhaust its memory, leading to a Denial of Service (DoS) vulnerability.
💡 SuggestionImplement a size check before reading the file. Use `os.Stat` to get the file's size and return an error if it exceeds a reasonable, configurable limit (e.g., 1MB).
🟠 Error gateway/api_definition.go:615-619
When resolving a `file://` secret in an API definition, a file read error is logged but the process continues. The `file://` placeholder is left unresolved, which can lead to an insecure configuration being loaded. For example, if a JWT signing key is not loaded, the API might be deployed with signature validation effectively disabled. Failures to load critical secrets should be treated as fatal errors for that API definition.
💡 SuggestionModify `replaceFileSecrets` to return an error that propagates up to the API loader. The API loader should then fail to load the specific API definition, preventing it from being deployed in an insecure state. The loop should not `continue` on error; it should return immediately.
🟡 Warning gateway/api_definition.go:608-632
The `replaceFileSecrets` function, which handles `file://` URIs in API definitions, does not prevent the use of absolute file paths when `kv.file.base_path` is configured. This is inconsistent with the safer implementation for `$secret_file.` variables, which explicitly rejects absolute paths. This allows a user with permission to edit API definitions to read any file on the gateway's filesystem, potentially leading to information disclosure or privilege escalation.
💡 SuggestionAdd a check within the `replaceFileSecrets` loop to reject absolute paths when `base_path` is set. This will enforce the security boundary for all file-based secret resolution mechanisms.
// Inside the for loop in replaceFileSecrets
key := m[1]
if basePath != &#34;&#34; &amp;&amp; filepath.IsAbs(key) {
    err := fmt.Errorf(&#34;file KV: absolute path rejected for file:// key %q - base_path is configured&#34;, key)
    log.WithError(err).Warn(&#34;Failed to resolve file secret&#34;)
    if firstErr == nil {
        firstErr = err
    }
    continue
}
val, err := ResolveFileKV(basePath, key)
// ...
\n\n ### Architecture Issues (2)
Severity Location Issue
🟠 Error gateway/api_definition.go:571-573
The API definition loading process continues even if a `file://` secret cannot be resolved. The error is only logged, and the `file://` placeholder remains in the API definition data. This leads to a misconfigured and likely non-functional API being loaded, masking deployment or configuration errors. The system should adopt a "fail-safe" approach, where an API definition is rejected if its secret dependencies cannot be met.
💡 SuggestionPropagate the error from `replaceFileSecrets` up to the API loading functions. The loader should then treat this error as fatal for the specific API definition and prevent it from being loaded into the gateway, providing clear feedback that the configuration is invalid.
🟡 Warning gateway/mw_url_rewrite.go:247-252
The implementation of `$secret_file.` variables introduces blocking file system I/O into the request processing path (the "hot path"). While the OS page cache can mitigate the performance impact for frequently accessed files, this design adds syscall overhead to every request that uses this feature. This could become a performance bottleneck under very high throughput or when secrets are stored on slower (e.g., network-attached) storage.
💡 SuggestionThe current design is pragmatic for ensuring secret freshness. However, for performance-critical use cases, consider introducing an optional, short-lived in-memory cache for file contents. A configurable TTL (e.g., 1-5 seconds) could offer a balance between performance and timely secret rotation, significantly reducing syscalls on the hot path.
\n\n \n\n

✅ Quality Check Passed

No quality issues found – changes LGTM.


Powered by Visor from Probelabs

Last updated: 2026-06-15T06:12:14.028Z | Triggered by: pr_updated | Commit: 691325d

💡 TIP: You can chat with Visor using /visor ask <your question>

Comment thread gateway/server.go Fixed
@vladzabolotnyi

Copy link
Copy Markdown
Contributor Author

/visor

The function ResolveFileKV is called on every request that uses a '$secret_file.' variable in URL rewrites or header transformations. This function performs blocking file system I/O operations (e.g., os.ReadFile, filepath.EvalSymlinks), which introduces significant latency and will severely degrade gateway performance under load. This can lead to request queuing and high response times.

Once a file is read, subsequent reads hit the kernel's buffer cache — not the disk. For a small secret file (bytes to a few KB), that's microseconds. File reads are not the same as disk I/O in practice.

@probelabs

probelabs Bot commented Jun 2, 2026

Copy link
Copy Markdown
Contributor

Powered by Visor from Probelabs

Last updated: 2026-06-02T09:33:37.130Z | Triggered by: vladzabolotnyi | Commit: 94cc57e

💡 TIP: You can chat with Visor using /visor ask <your question>

@vladzabolotnyi

Copy link
Copy Markdown
Contributor Author

/visor

Using strings.Replace inside a loop to perform multiple replacements is inefficient. Each call re-scans the entire input string and allocates a new string for the result, which can be slow if there are many replacements in a large API definition.

using Replacer is a super overkill in our case

@probelabs

probelabs Bot commented Jun 2, 2026

Copy link
Copy Markdown
Contributor

Powered by Visor from Probelabs

Last updated: 2026-06-02T09:41:29.682Z | Triggered by: vladzabolotnyi | Commit: 94cc57e

💡 TIP: You can chat with Visor using /visor ask <your question>

Comment thread gateway/server.go Dismissed
@vladzabolotnyi

Copy link
Copy Markdown
Contributor Author

/visor

When resolving a 'file://' secret during API definition loading, an error in reading the file is logged, but the loading process continues with an empty string in place of the secret. This "fail-open" behavior can lead to severe security vulnerabilities, such as a JWT-validating API being loaded with an empty secret, effectively disabling signature checks. The API definition loading process should fail completely if any required secret cannot be resolved.

This is the way any other kv ref is resolved, no hard fail but skip and log

@probelabs

probelabs Bot commented Jun 2, 2026

Copy link
Copy Markdown
Contributor

Powered by Visor from Probelabs

Last updated: 2026-06-02T10:55:25.525Z | Triggered by: vladzabolotnyi | Commit: 94cc57e

💡 TIP: You can chat with Visor using /visor ask <your question>

@vladzabolotnyi

Copy link
Copy Markdown
Contributor Author

/visor

The function ResolveFileKV uses a non-existent function filepath.IsLocal for path traversal protection, both before and after resolving symlinks. Since this function is not part of Go's standard library, the checks are ineffective, creating a path traversal vulnerability. An attacker could craft a key like ../.../etc/passwd or use a symlink to escape the base_path directory and read arbitrary files on the gateway's filesystem.

False positive, the IsLocal was added on 1.20+

@probelabs

probelabs Bot commented Jun 2, 2026

Copy link
Copy Markdown
Contributor

Powered by Visor from Probelabs

Last updated: 2026-06-02T11:35:00.072Z | Triggered by: vladzabolotnyi | Commit: 94cc57e

💡 TIP: You can chat with Visor using /visor ask <your question>

@sentinelone-cnapp-eu1

Copy link
Copy Markdown

SentinelOne CNS Hardcoded Secret Detector
✅ Congratulations, your code is safe

SentinelOne CNS is a cloud-agnostic, agentless CSPM & CWPP solution that continuously detects and prevents vulnerabilities that have the highest probability of being exploited in Azure, AWS, Google Cloud, and Kubernetes.

@github-actions

Copy link
Copy Markdown
Contributor

🚨 Jira Linter Failed

Commit: 691325d
Failed at: 2026-06-15 06:10:29 UTC

The Jira linter failed to validate your PR. Please check the error details below:

🔍 Click to view error details
failed to get Jira issue: failed to fetch Jira issue TT-17297: Issue does not exist or you do not have permission to see it.: request failed. Please analyze the request body for more details. Status code: 404

Next Steps

  • Ensure your branch name contains a valid Jira ticket ID (e.g., ABC-123)
  • Verify your PR title matches the branch's Jira ticket ID
  • Check that the Jira ticket exists and is accessible

This comment will be automatically deleted once the linter passes.

@sonarqubecloud

Copy link
Copy Markdown

Quality Gate Passed Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
90.1% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

@vladzabolotnyi
vladzabolotnyi enabled auto-merge (squash) June 15, 2026 07:03
@vladzabolotnyi
vladzabolotnyi merged commit c90ec7b into master Jun 15, 2026
50 of 56 checks passed
@vladzabolotnyi
vladzabolotnyi deleted the TT-17297 branch June 15, 2026 07:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants