Skip to content

Commit 40194f9

Browse files
authored
fix(network): fail closed when credential placeholders cannot be rewritten (#2162)
* fix(network): fail closed when credential placeholders cannot be rewritten When the credential rewriter degrades internally, the proxy forwarded the literal `openshell:resolve:env:<NAME>` placeholder (or its provider alias marker) to the upstream instead of the resolved secret, leaking the reserved token on the wire and causing upstream auth failures (#2161). Two fail-open paths are closed: - secrets: `rewrite_http_header_block` returned the header block verbatim when no `SecretResolver` was available, so the fail-closed marker scan (which ran only on the resolved path) never saw the placeholder. It now scans the header region for reserved markers even with no resolver and returns `UnresolvedPlaceholderError` when one is present. Marker-free traffic still passes through unchanged. - proxy: when TLS was detected on a CONNECT but `tls_state` was `None` (ephemeral CA generation or CA file write failed at startup), the handler fell back to a raw `copy_bidirectional` tunnel, bypassing credential rewrite. Inside the proxy handler `tls_state` is `None` only on CA-init failure (`mode != Proxy` never starts the handler, and `tls: skip` is handled earlier), so it now refuses the connection with a 503 and a High-severity denial event instead of tunneling. The two startup CA-failure logs are raised from Medium to High. Tests: resolver=None with a placeholder in the request line, a header value, and the provider-alias form now fail closed; marker-free passthrough is unchanged; the relay integration test asserts the request is rejected before any byte reaches upstream; and the 503 fail-closed response contract is locked. Signed-off-by: Tony Luo <[email protected]> * fix(network): refuse CONNECT before 200 when TLS termination is unavailable The fail-closed refusal for a terminating CONNECT with no TLS termination state (ephemeral CA init failed) was written after the 200 Connection Established response. Because a CONNECT client only sends its TLS ClientHello after reading the 200, the peek-based TLS detection is inherently post-200, so the 503 landed inside the established tunnel and surfaced to the client as a TLS protocol error rather than a readable status. An 'allowed CONNECT' event was also logged first. Move the decision to a pre-200 gate: query_tls_mode resolves purely from the policy decision + host/port (no peeked bytes), so the route's TLS treatment is known before the tunnel is acknowledged. When TLS state is absent and the route is not tls: skip, write the 503 as the first bytes on the socket, emit the High-severity Denied event, and close. tls: skip routes tunnel raw exactly as before, and no allowed-CONNECT event is emitted on the refusal path. The now-unreachable post-200 branch is kept as defense in depth but no longer writes an in-tunnel 503; it fails closed by dropping the connection instead. Add connection-level regression tests over a real loopback socket: the gate refuses with HTTP/1.1 503 as the first bytes for a terminating route, and writes nothing when TLS termination is present or the route is tls: skip. Signed-off-by: Tony Luo <[email protected]> * fix(network): order the CONNECT TLS-unavailable refusal after SSRF Addresses the gator re-check on #2162. Ordering: the pre-200 fail-closed refusal ran before SSRF/allowed_ips validation, so during CA-init failure an internal-address CONNECT got a 503 tls_termination_unavailable instead of the normal 403 ssrf_denied, weakening operator visibility in degraded state. The SSRF branches now return validated addresses; the refusal runs after that validation (an internal address has already been denied with 403) but still before the upstream connect and before 200 Connection Established. effective_tls_skip is still resolved up front since the refusal consumes it. Tests: add connection-level regressions through the real handle_tcp_connection, driving a CONNECT from a child /bin/bash copy so the /proc process-identity binding resolves it against a permissive policy (the hot-swap test's identity pattern). They assert the first bytes are HTTP/1.1 503 for a terminating route with no TLS state, a 403 (not 503) for an internal address, and no refusal for a tls: skip route. These are gated to Linux at runtime (evaluate_opa_tcp needs /proc); a companion test verifies the OPA policy shape (glob allow, tls mode) on every platform so the precondition is locked where /proc is unavailable. rest.rs: tighten the fail-closed relay test to assert the forwarded buffer is_empty() rather than merely lacking the placeholder/secret. Signed-off-by: Tony Luo <[email protected]> * test(network): keep the CONNECT handler test client fork-free The handler regression tests forked cat to read the proxy reply, so the client socket fd was inherited by a second process with a different binary. The identity resolver correctly denies that as ambiguous shared-socket ownership (the same invariant resolve_process_identity_denies_fork_exec_shared_socket_ambiguity pins), so the tests exercised the deny path instead of the allow path — and on busy CI runners the deny-path /proc fallback scan exceeded the test budget and looked like a hang. The client script now uses only bash builtins (exec, printf, read -d '') so exactly one process owns the socket, and the child is left to exit on EOF instead of being killed mid-read. Signed-off-by: Tony Luo <[email protected]> * test(network): drive the CONNECT handler tests with an in-process client The child-process client (even fork-free) made the handler tests environment-sensitive: on CI runners with a busy or restricted /proc, resolving the child's socket ownership degraded into the whole-/proc fallback scan and a deny, which surfaced as a hang. The client is now an in-process TcpStream and the test policy allows current_exe(), so identity resolution binds the socket to the test process itself in the descendant scan — the same in-process pattern the passing resolve_process_identity tests rely on. The tls: skip test additionally asserts that the handler emitted no DenialEvent at any stage, so it can no longer pass vacuously on a policy or identity deny. Refusal budgets widened to 30s as a belt for slow runners; the refusals themselves return in milliseconds. Signed-off-by: Tony Luo <[email protected]> * test(core): pin the percent-encoded marker no-resolver fail-closed path The no-resolver scan already catches the percent-encoded canonical marker through its decoded pass; this regression pins it: a request line carrying openshell%3Aresolve%3Aenv%3AKEY with no resolver must fail closed with UnresolvedPlaceholderError { location: header }. Signed-off-by: Tony Luo <[email protected]> --------- Signed-off-by: Tony Luo <[email protected]>
1 parent 614c8c1 commit 40194f9

4 files changed

Lines changed: 597 additions & 78 deletions

File tree

‎crates/openshell-core/src/secrets.rs‎

Lines changed: 51 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -929,6 +929,20 @@ pub fn rewrite_http_header_block(
929929
resolver: Option<&SecretResolver>,
930930
) -> Result<RewriteResult, UnresolvedPlaceholderError> {
931931
let Some(resolver) = resolver else {
932+
// Fail-closed: with no resolver there is nothing that can rewrite a
933+
// credential placeholder, so forwarding the block verbatim would leak
934+
// the reserved token to the upstream. Scan the header region (mirroring
935+
// the resolved-path scan below) and reject if any reserved marker is
936+
// present. Marker-free traffic still passes through unchanged, which is
937+
// the intended behavior when the endpoint injects no credentials.
938+
let scan_end = raw
939+
.windows(4)
940+
.position(|w| w == b"\r\n\r\n")
941+
.map_or(raw.len(), |p| raw.len().min(p + 4 + 256));
942+
let header_region = String::from_utf8_lossy(&raw[..scan_end]);
943+
if contains_reserved_credential_marker(&header_region) {
944+
return Err(UnresolvedPlaceholderError { location: "header" });
945+
}
932946
return Ok(RewriteResult {
933947
rewritten: raw.to_vec(),
934948
redacted_target: None,
@@ -1889,11 +1903,44 @@ mod tests {
18891903
}
18901904

18911905
#[test]
1892-
fn no_resolver_passes_through_without_scanning() {
1893-
// Even if placeholders are present, None resolver means no scanning
1906+
fn no_resolver_with_placeholder_in_request_line_fails_closed() {
1907+
// Fail-closed: a placeholder in the request line with no resolver must
1908+
// never be forwarded verbatim — it would leak the reserved token.
18941909
let raw = b"GET /api/openshell:resolve:env:KEY HTTP/1.1\r\nHost: x\r\n\r\n";
1895-
let result = rewrite_http_header_block(raw, None).expect("should succeed");
1896-
assert_eq!(raw.as_slice(), result.rewritten.as_slice());
1910+
let err = rewrite_http_header_block(raw, None)
1911+
.expect_err("placeholder without resolver must fail closed");
1912+
assert_eq!(err.location, "header");
1913+
}
1914+
1915+
#[test]
1916+
fn no_resolver_with_placeholder_in_header_value_fails_closed() {
1917+
let raw =
1918+
b"GET / HTTP/1.1\r\nAuthorization: Bearer openshell:resolve:env:KEY\r\nHost: x\r\n\r\n";
1919+
let err = rewrite_http_header_block(raw, None)
1920+
.expect_err("placeholder without resolver must fail closed");
1921+
assert_eq!(err.location, "header");
1922+
}
1923+
1924+
#[test]
1925+
fn no_resolver_with_percent_encoded_marker_fails_closed() {
1926+
// Percent-encoded form of the canonical marker: a client (or an evasive
1927+
// sender) can URL-encode the reserved token, and the no-resolver scan
1928+
// must still catch it via the percent-decoded pass.
1929+
let raw = b"GET /api/openshell%3Aresolve%3Aenv%3AKEY HTTP/1.1\r\nHost: x\r\n\r\n";
1930+
let err = rewrite_http_header_block(raw, None)
1931+
.expect_err("percent-encoded placeholder without resolver must fail closed");
1932+
assert_eq!(err.location, "header");
1933+
}
1934+
1935+
#[test]
1936+
fn no_resolver_with_provider_alias_marker_fails_closed() {
1937+
// The provider-shaped alias marker is also a reserved credential token
1938+
// and must fail closed when no resolver can rewrite it.
1939+
let raw =
1940+
b"GET / HTTP/1.1\r\nX-Api-Key: OPENSHELL-RESOLVE-ENV-CUSTOM_TOKEN\r\nHost: x\r\n\r\n";
1941+
let err = rewrite_http_header_block(raw, None)
1942+
.expect_err("alias marker without resolver must fail closed");
1943+
assert_eq!(err.location, "header");
18971944
}
18981945

18991946
#[test]

‎crates/openshell-supervisor-network/src/l7/rest.rs‎

Lines changed: 27 additions & 40 deletions
Original file line numberDiff line numberDiff line change
@@ -4857,12 +4857,15 @@ mod tests {
48574857
assert!(forwarded.contains("Host: integrate.api.nvidia.com\r\n"));
48584858
}
48594859

4860-
/// Verifies that without a `SecretResolver` (i.e. the L4-only raw tunnel
4861-
/// path, or no TLS termination), credential placeholders pass through
4862-
/// unmodified. This documents the behavior that causes 401 errors when
4863-
/// `tls: terminate` is missing from the endpoint config.
4860+
/// Verifies that when a request carries a credential placeholder but no
4861+
/// `SecretResolver` is available to rewrite it, the relay fails closed
4862+
/// instead of forwarding the reserved token verbatim. A `None` resolver
4863+
/// here stands in for a degraded internal state (e.g. secrets could not be
4864+
/// wired up); forwarding the placeholder would leak it to the upstream and
4865+
/// cause 401s. The fail-closed scan in `rewrite_http_header_block` rejects
4866+
/// the request before any byte reaches upstream.
48644867
#[tokio::test]
4865-
async fn relay_request_without_resolver_leaks_placeholders() {
4868+
async fn relay_request_without_resolver_fails_closed_on_placeholder() {
48664869
let (child_env, _resolver) = SecretResolver::from_provider_env(
48674870
[("NVIDIA_API_KEY".to_string(), "nvapi-secret".to_string())]
48684871
.into_iter()
@@ -4887,55 +4890,39 @@ mod tests {
48874890
body_length: BodyLength::ContentLength(2),
48884891
};
48894892

4890-
let upstream_task = tokio::spawn(async move {
4891-
let mut buf = vec![0u8; 4096];
4892-
let mut total = 0;
4893-
loop {
4894-
let n = upstream_side.read(&mut buf[total..]).await.unwrap();
4895-
if n == 0 {
4896-
break;
4897-
}
4898-
total += n;
4899-
if let Some(hdr_end) = buf[..total].windows(4).position(|w| w == b"\r\n\r\n") {
4900-
if total >= hdr_end + 4 + 2 {
4901-
break;
4902-
}
4903-
}
4904-
}
4905-
upstream_side
4906-
.write_all(b"HTTP/1.1 200 OK\r\nContent-Length: 2\r\n\r\nok")
4907-
.await
4908-
.unwrap();
4909-
upstream_side.flush().await.unwrap();
4910-
String::from_utf8_lossy(&buf[..total]).to_string()
4911-
});
4912-
4913-
// Pass `None` for the resolver — simulates the L4 path where no
4914-
// rewriting occurs.
4893+
// Pass `None` for the resolver — simulates a degraded state where no
4894+
// rewriting can occur. The relay must reject the request.
49154895
let relay = tokio::time::timeout(
49164896
std::time::Duration::from_secs(5),
49174897
relay_http_request_with_resolver(
49184898
&req,
49194899
&mut proxy_to_client,
49204900
&mut proxy_to_upstream,
4921-
None, // <-- No resolver, as in the L4 raw tunnel path
4901+
None,
49224902
),
49234903
)
49244904
.await
49254905
.expect("relay must not deadlock");
4926-
relay.expect("relay should succeed");
4927-
4928-
let forwarded = upstream_task.await.expect("upstream task should complete");
49294906

4930-
// Without a resolver, the placeholder LEAKS to upstream — this is the
4931-
// documented behavior that causes 401s when `tls: terminate` is missing.
49324907
assert!(
4933-
forwarded.contains("openshell:resolve:env:NVIDIA_API_KEY"),
4934-
"Expected placeholder to leak without resolver, got: {forwarded}"
4908+
relay.is_err(),
4909+
"relay must fail closed when a placeholder cannot be resolved"
49354910
);
4911+
4912+
// The fail-closed scan rejects the request before any byte reaches
4913+
// upstream, so nothing at all must have been forwarded — a strictly
4914+
// stronger invariant than "the placeholder/secret did not appear". Drop
4915+
// the proxy write half so the read observes EOF instead of blocking.
4916+
drop(proxy_to_upstream);
4917+
let mut forwarded = Vec::new();
4918+
upstream_side
4919+
.read_to_end(&mut forwarded)
4920+
.await
4921+
.expect("upstream read should complete");
49364922
assert!(
4937-
!forwarded.contains("nvapi-secret"),
4938-
"Real secret should NOT appear without resolver, got: {forwarded}"
4923+
forwarded.is_empty(),
4924+
"no bytes may reach upstream when the placeholder cannot be resolved; got: {}",
4925+
String::from_utf8_lossy(&forwarded)
49394926
);
49404927
}
49414928

0 commit comments

Comments
 (0)