Skip to content

Commit e985c5f

Browse files
TRO-Wolfclaude
andcommitted
fix(rest): Critic remediation — stop leaking raw response bodies via error context (F1); correct trust-model provenance and residue scoping (F2-F7)
Remediates the NOT-CONVERGED review of 38802b7 (1 S1 + 2 S2 + 4 S3/S4). F1 (S1) — the raw-body leak defeated the whole unit. `deserialize_catalog_response` attached the RAW 200-OK body via `.with_context("json", …)` on any parse failure, and `iceberg::Error` renders context VERBATIM in both `Display` and `Debug`. Every type redacted in the parent commit routes through that function — `CatalogConfig` (catalog.rs:592), `LoadTableResult` incl. `storage_credentials` (:988/:1059/:1208), `LoadViewResult` (:1439/:1475), `NamespaceResponse` (:810/:838) — so a single parse failure printed exactly what the `Debug` impls had just masked. The trigger is realistic: a server returning a payload this parser rejects (unknown V3 field, version skew) hands back the same body carrying the vended `s3.session-token`. Fixed with the pattern already used on the token endpoint (client.rs:295-307): attach `status`, `url` and `response_body_len` — never the body. Two mockito guard tests drive the Critic's probe shape end-to-end (`/v1/config` with a sentinel in `defaults` and `overrides` missing; and a real `load_table` body carrying vended credentials, corrupted so it cannot parse) and assert the sentinel is absent from BOTH renderings while the safe diagnostics are present. Both go RED when the raw-body attach is restored. `with_source` is retained (AGENTS.md requires the chain); `serde_json`'s message can echo the offending token at the failure POSITION, never the surrounding payload. The load_table test pins that bound. Documented as residue. SCOPED, not changed: the NON-2xx path (`deserialize_unexpected_catalog_error`) still attaches its body. A non-2xx response means no table was loaded, so no `config` overlay and no vended credentials were issued; the body is an `ErrorResponse` whose `message` is the diagnostic value, and Java surfaces exactly this (`ErrorHandlers`). The one non-2xx body that could echo credentials — the token endpoint's — already withholds it. Rationale recorded on the function. F2 (S2) — the trust-model doc was factually wrong about provenance. `merge_with_config` lets the SERVER's /v1/config `overrides.uri` replace `self.uri`, and server defaults/overrides merge into `props`, so `oauth2-server-uri` can be server-supplied: a compromised server can redirect the token POST — carrying the operator's `client_secret` — to a third-party host. The doc now names BOTH provenances, adjudicates the server-override variant as parity-shared (Java builds both the client and `AuthConfig.fromProperties` from `mergedProps`, RESTSessionCatalog.java:211-230), and reconciles with the `disable-header-redaction` strip: that key is a client-only logging control with no wire meaning (pure downside if server-set, so stripped), whereas `uri`/`oauth2-server-uri` server override is a documented functional part of the config protocol. No behavior change. F3 (S2) — `CommitTableResponse` carries `metadata: TableMetadata` (with `.properties`); it was wrongly in the "identifier/pagination-only, no value map" all-clear bucket. Moved into the named core-crate residue paragraph. F4 (S3) — corrected the `OAuth2Manager` citation to :275-293 and the guard wording: the WARN fires only when the key is absent AND (`initToken != null` OR non-empty `credential`). F5 (S3) — added `ViewVersion.summary` (derived `Debug`, `String -> String`, reachable via `CreateViewRequest.view_version` and both view-metadata paths) to the named residue. F6 (S3) — reframed `ErrorModel`/`ErrorResponse`/`OAuthError` from all-clear to RESIDUE: their fields are server-controlled free text, and `message`/`error_description` are now the one channel by which response-body content still reaches logs after F1. F7 (S4) — added the transport-layer nuance: Iceberg configures no redirect policy on either side, so Java inherits HC5 defaults (max 50, no cross-origin header strip) while this fork inherits reqwest's (max 10, strips Authorization/Cookie/Proxy-Authorization cross-origin). We are STRICTER, not looser — "no restriction of any kind" was imprecise. Tests: rest lib 93 -> 95 (+2); 15/15 mutations RED (13 prior + F1a/F1b). `cargo test -p iceberg --lib` 2855 green. Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01MRhAQnro5233xoVDLfib1X
1 parent 38802b7 commit e985c5f

3 files changed

Lines changed: 256 additions & 31 deletions

File tree

‎crates/catalog/rest/src/catalog.rs‎

Lines changed: 184 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -52,14 +52,40 @@ use crate::types::{
5252
///
5353
/// # Trust model (SEC-011 adjudication, Java parity)
5454
///
55-
/// This value, and the `oauth2-server-uri` property that overrides the token endpoint (see
56-
/// [`RestCatalogConfig::get_token_endpoint`]), are **operator-supplied catalog configuration and
57-
/// are treated as trusted**. The client will connect to whatever host they name, including
58-
/// loopback, RFC 1918 / link-local addresses, and cluster-internal DNS names — which is the normal
59-
/// deployment, since catalog services usually live on a private network.
55+
/// This value and the `oauth2-server-uri` property that overrides the token endpoint (see
56+
/// [`RestCatalogConfig::get_token_endpoint`]) have **two provenances**, and both are treated as
57+
/// trusted:
58+
///
59+
/// 1. **Operator-supplied** — the properties passed to `load`/the builder.
60+
/// 2. **Server-overridable** — [`RestCatalogConfig::merge_with_config`] lets the `/v1/config`
61+
/// response replace `uri` outright (`overrides.uri`) and merges the server's `defaults` /
62+
/// `overrides` into `props`, from which `oauth2-server-uri` is read. So a compromised catalog
63+
/// server can redirect subsequent requests, and can point the token POST — which carries the
64+
/// operator's `client_secret` — at a third-party host.
65+
///
66+
/// Provenance (2) is **parity-shared, not a fork defect**: Java does exactly the same. Its
67+
/// `RESTSessionCatalog.initialize` builds `mergedProps = config.merge(props)` from the `/v1/config`
68+
/// response and then constructs BOTH the real client (`clientBuilder.apply(mergedProps)`, whose
69+
/// `uri` is `mergedProps.get(CatalogProperties.URI)`) and the auth config
70+
/// (`AuthConfig.fromProperties(mergedProps)`, which reads `oauth2-server-uri`) from that merged map
71+
/// (`RESTSessionCatalog.java:211-230`). A client that trusts a catalog server for its endpoint set
72+
/// necessarily trusts it for the endpoints themselves.
73+
///
74+
/// **This does not contradict the `disable-header-redaction` strip** in `merge_with_config`, and the
75+
/// distinction is the point: that key is a *client-only logging control* with no wire meaning, so a
76+
/// server has no legitimate reason to set it and a malicious one would use it to unmask the
77+
/// `Authorization` header this client already holds — pure downside, so it is stripped. `uri` and
78+
/// `oauth2-server-uri` are the opposite: server override is a *documented, functional* part of the
79+
/// REST config protocol (endpoint discovery / migration), and honoring it is required for
80+
/// correctness. Strip the former, honor the latter — and where the trust in the server is
81+
/// misplaced, the credential exposure via a redirected token POST is a risk Java carries identically.
82+
///
83+
/// Against either provenance the client applies no address filtering: it will connect to whatever
84+
/// host is named, including loopback, RFC 1918 / link-local addresses, and cluster-internal DNS
85+
/// names — the normal deployment, since catalog services usually live on a private network.
6086
///
6187
/// This deliberately mirrors Java `iceberg-core` 1.10.0, which applies **no** host, IP-range, or
62-
/// scheme restriction of any kind:
88+
/// scheme restriction:
6389
///
6490
/// * `HTTPClient.Builder.uri(String)` only null-checks and parses:
6591
/// `Preconditions.checkNotNull(baseUri, …)` then `URI.create(RESTUtil.stripTrailingSlash(baseUri))`,
@@ -71,19 +97,31 @@ use crate::types::{
7197
/// (`core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java:175`).
7298
/// * `oauth2-server-uri` is taken verbatim from properties, defaulting to the catalog-relative
7399
/// `ResourcePaths.tokens()`
74-
/// (`core/src/main/java/org/apache/iceberg/rest/auth/AuthConfig.java:85`); `OAuth2Manager`
75-
/// only emits a deprecation WARN when it is unset — never a restriction
76-
/// (`core/src/main/java/org/apache/iceberg/rest/auth/OAuth2Manager.java:274-291`).
100+
/// (`core/src/main/java/org/apache/iceberg/rest/auth/AuthConfig.java:85`); `OAuth2Manager`'s
101+
/// `warnIfOAuthServerUriNotSet` emits a deprecation WARN only when the key is ABSENT **and** the
102+
/// client is authenticating — `initToken != null` OR a non-empty `credential`
103+
/// (`core/src/main/java/org/apache/iceberg/rest/auth/OAuth2Manager.java:275-293`). A log line
104+
/// under a narrow condition, never a restriction.
77105
/// * A grep of the entire `core/src/main/java/org/apache/iceberg/rest/` tree finds no loopback /
78106
/// `InetAddress` / site-local / allowlist logic. TLS is a pluggable operator *extension* point
79107
/// (`rest.client.tls.configurer-impl`, `HTTPClient.java:98`), not a restriction.
80108
///
109+
/// One transport-layer nuance, for precision: "no restriction" is true of Iceberg's *own* code on
110+
/// both sides, but the underlying HTTP clients differ, and this fork is the STRICTER of the two.
111+
/// Iceberg configures no redirect policy anywhere in `core/.../rest/`, so Java inherits Apache
112+
/// HttpClient5's defaults (follow redirects, max 50, no cross-origin header stripping), whereas this
113+
/// fork inherits `reqwest`'s (follow redirects, max 10, and `Authorization` / `Cookie` /
114+
/// `Proxy-Authorization` stripped on a cross-origin redirect). Redirect-following is therefore a
115+
/// smaller credential-forwarding surface here than in Java — a divergence in the safe direction, and
116+
/// not one to "fix".
117+
///
81118
/// SSRF is a concern when a URI arrives from an *untrusted* source (a request parameter, a
82-
/// user-submitted document). That is not this surface: whoever can set catalog properties can
83-
/// already point the client anywhere and read whatever it fetches. Adding a private-IP blocklist
84-
/// here would break legitimate private-endpoint deployments — the majority case — and diverge from
85-
/// Java. Callers that DO accept a catalog URI from untrusted input must validate it before it
86-
/// reaches this configuration; that check belongs at their trust boundary, not in this client.
119+
/// user-submitted document). That is not this surface: whoever can set catalog properties — or
120+
/// operate the catalog server the client was pointed at — can already point the client anywhere and
121+
/// read whatever it fetches. Adding a private-IP blocklist here would break legitimate
122+
/// private-endpoint deployments, the majority case, and diverge from Java. Callers that DO accept a
123+
/// catalog URI from untrusted input must validate it before it reaches this configuration; that
124+
/// check belongs at their trust boundary, not in this client.
87125
pub const REST_CATALOG_PROP_URI: &str = "uri";
88126
/// REST catalog warehouse location
89127
pub const REST_CATALOG_PROP_WAREHOUSE: &str = "warehouse";
@@ -2373,6 +2411,138 @@ mod tests {
23732411
}
23742412
}
23752413

2414+
/// SECURITY (SEC-010 / F1): redacting the wire types' `Debug` is defeated if the RAW response
2415+
/// body reaches the error context — `iceberg::Error` renders context VERBATIM in both `Display`
2416+
/// and `Debug`, so one parse failure printed everything the redaction had just masked.
2417+
///
2418+
/// This is the `/v1/config` path: `defaults` carries a vended credential and `overrides` is
2419+
/// MISSING, so `CatalogConfig` fails to deserialize while the secret is in the body. The
2420+
/// realistic trigger is version skew — a server returning a payload this parser rejects hands
2421+
/// back the same body that carries the credentials.
2422+
///
2423+
/// RED-able: restore `.with_context("json", String::from_utf8_lossy(&bytes))` in
2424+
/// `deserialize_catalog_response`.
2425+
#[tokio::test]
2426+
async fn test_config_parse_failure_does_not_leak_body() {
2427+
const SENTINEL: &str = "SENTINEL_MUST_NOT_APPEAR_IN_ERROR";
2428+
2429+
let mut server = Server::new_async().await;
2430+
// 200 OK, secret present in `defaults`, but `overrides` is absent => parse failure.
2431+
let config_mock = server
2432+
.mock("GET", "/v1/config")
2433+
.with_status(200)
2434+
.with_body(format!(
2435+
r#"{{"defaults": {{"s3.secret-access-key": "{SENTINEL}"}}}}"#
2436+
))
2437+
.create_async()
2438+
.await;
2439+
2440+
let catalog = RestCatalog::new(
2441+
RestCatalogConfig::builder().uri(server.url()).build(),
2442+
Some(Arc::new(LocalFsStorageFactory)),
2443+
);
2444+
2445+
let err = catalog
2446+
.list_namespaces(None)
2447+
.await
2448+
.expect_err("an unparsable /v1/config body must surface an error");
2449+
2450+
config_mock.assert_async().await;
2451+
2452+
let rendered = format!("{err}");
2453+
let debug = format!("{err:?}");
2454+
assert!(
2455+
!rendered.contains(SENTINEL),
2456+
"config body leaked into Display: {rendered}"
2457+
);
2458+
assert!(
2459+
!debug.contains(SENTINEL),
2460+
"config body leaked into Debug: {debug}"
2461+
);
2462+
// Anti-over-redaction: the safe diagnostics must still be attached, or the error is
2463+
// useless and this test would pass vacuously against a context-free error.
2464+
assert!(
2465+
rendered.contains("response_body_len"),
2466+
"expected the safe body-length diagnostic: {rendered}"
2467+
);
2468+
assert!(
2469+
rendered.contains("status"),
2470+
"expected the safe status diagnostic: {rendered}"
2471+
);
2472+
}
2473+
2474+
/// SECURITY (SEC-010 / F1), the higher-value payload: a `load_table` 200 body carries BOTH the
2475+
/// `config` overlay and the vended `storage-credentials`. Here the body is rejected because
2476+
/// `metadata` is the wrong JSON type (standing in for an unknown V3 field / version skew) while
2477+
/// the vended `s3.session-token` sits in the same body.
2478+
///
2479+
/// This shape also pins the `with_source` residue bound: `serde_json` echoes the offending
2480+
/// token at the failure POSITION (`invalid type: integer ...`), never the surrounding payload,
2481+
/// so the credentials elsewhere in the body stay unreachable.
2482+
///
2483+
/// RED-able: restore `.with_context("json", …)` in `deserialize_catalog_response`.
2484+
#[tokio::test]
2485+
async fn test_load_table_parse_failure_does_not_leak_vended_credentials() {
2486+
const SENTINEL: &str = "SENTINEL_MUST_NOT_APPEAR_IN_ERROR";
2487+
2488+
let mut server = Server::new_async().await;
2489+
let config_mock = create_config_mock(&mut server).await;
2490+
2491+
// A real load_table body, carrying vended credentials, corrupted so it cannot parse.
2492+
let mut body: serde_json::Value = serde_json::from_str(&load_table_body(
2493+
json!({ "s3.secret-access-key": SENTINEL }),
2494+
Some(json!([{
2495+
"prefix": "s3://warehouse",
2496+
"config": { "s3.session-token": SENTINEL }
2497+
}])),
2498+
))
2499+
.expect("patched load_table body must be valid JSON");
2500+
body["metadata"] = json!(12345);
2501+
let broken = serde_json::to_string(&body).expect("serialize the corrupted body");
2502+
assert!(
2503+
broken.contains(SENTINEL),
2504+
"precondition: the body under test must actually carry the secret"
2505+
);
2506+
2507+
let table_mock = server
2508+
.mock("GET", "/v1/namespaces/ns1/tables/table1")
2509+
.with_status(200)
2510+
.with_body(broken)
2511+
.create_async()
2512+
.await;
2513+
2514+
let catalog = RestCatalog::new(
2515+
RestCatalogConfig::builder().uri(server.url()).build(),
2516+
Some(Arc::new(LocalFsStorageFactory)),
2517+
);
2518+
2519+
let err = catalog
2520+
.load_table(&TableIdent::new(
2521+
NamespaceIdent::new("ns1".to_string()),
2522+
"table1".to_string(),
2523+
))
2524+
.await
2525+
.expect_err("an unparsable load_table body must surface an error");
2526+
2527+
config_mock.assert_async().await;
2528+
table_mock.assert_async().await;
2529+
2530+
let rendered = format!("{err}");
2531+
let debug = format!("{err:?}");
2532+
assert!(
2533+
!rendered.contains(SENTINEL),
2534+
"vended credentials leaked into Display: {rendered}"
2535+
);
2536+
assert!(
2537+
!debug.contains(SENTINEL),
2538+
"vended credentials leaked into Debug: {debug}"
2539+
);
2540+
assert!(
2541+
rendered.contains("response_body_len"),
2542+
"expected the safe body-length diagnostic: {rendered}"
2543+
);
2544+
}
2545+
23762546
/// SEC-011 adjudication guard (Java parity — see the trust-model note on
23772547
/// [`REST_CATALOG_PROP_URI`]).
23782548
///

‎crates/catalog/rest/src/client.rs‎

Lines changed: 40 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -555,17 +555,47 @@ pub(crate) fn commit_transport_failure_may_have_reached_service(error: &reqwest:
555555
/// Deserializes a catalog response into the given [`DeserializedOwned`] type.
556556
///
557557
/// Returns an error if unable to parse the response bytes.
558+
///
559+
/// SECURITY (SEC-010): this is the SUCCESS (2xx) body path, so the bytes in hand are a
560+
/// credential-bearing payload — `CatalogConfig` (`defaults`/`overrides`, which the catalog merges
561+
/// into its runtime props and hands to `FileIO`), `LoadTableResult` (`config` plus the vended
562+
/// `storage-credentials`), `LoadViewResult` (`config`), and `NamespaceResponse` (`properties`) all
563+
/// deserialize through here. Attaching the raw body to the error context — as this function used to
564+
/// — put those secrets into `Error`'s context, which `iceberg::Error` renders VERBATIM in both
565+
/// `Display` and `Debug` (`crates/iceberg/src/error.rs`), so a single parse failure leaked live
566+
/// credentials into every `{e}` / `tracing::error!(?e)` site. The trigger is entirely realistic: a
567+
/// server returning a payload this parser rejects (an unknown V3 field, a version skew) hands back
568+
/// the SAME body that carries the vended `s3.session-token`.
569+
///
570+
/// So: never attach the raw body. Only non-secret diagnostics are attached — the HTTP status, the
571+
/// request URL, and the body's byte LENGTH — mirroring the token-endpoint paths in
572+
/// [`HttpClient::exchange_credential_for_token`], which have withheld the body since the SEC-fix
573+
/// series. This costs parse-failure debuggability; operators who need the payload can capture it at
574+
/// the HTTP layer (a proxy or a `reqwest` middleware), where the exposure is a deliberate choice
575+
/// rather than an unconditional log write.
576+
///
577+
/// RESIDUE: `with_source(e)` is retained because AGENTS.md requires the error chain to survive, and
578+
/// `serde_json`'s own message can echo the offending token (e.g. `invalid type: integer `12345``).
579+
/// That echo is bounded to the single value at the parse-failure position — it is never the whole
580+
/// body, and for the realistic failures (missing/unknown field) it echoes only a FIELD NAME. A
581+
/// secret is only reachable through it if the server puts the secret itself in a type-mismatched
582+
/// structural position, which no real server response does. Both shapes are pinned by
583+
/// `test_config_parse_failure_does_not_leak_body` / `test_load_table_parse_failure_does_not_leak_vended_credentials`.
558584
pub(crate) async fn deserialize_catalog_response<R: DeserializeOwned>(
559585
response: Response,
560586
) -> Result<R> {
587+
let status = response.status();
588+
let url = response.url().clone();
561589
let bytes = response.bytes().await?;
562590

563591
serde_json::from_slice::<R>(&bytes).map_err(|e| {
564592
Error::new(
565593
ErrorKind::Unexpected,
566594
"Failed to parse response from rest catalog server",
567595
)
568-
.with_context("json", String::from_utf8_lossy(&bytes))
596+
.with_context("status", status.to_string())
597+
.with_context("url", url.to_string())
598+
.with_context("response_body_len", bytes.len().to_string())
569599
.with_source(e)
570600
})
571601
}
@@ -615,6 +645,15 @@ fn format_headers_redacted(headers: &HeaderMap, disable_redaction: bool) -> Stri
615645
}
616646

617647
/// Deserializes a unexpected catalog response into an error.
648+
///
649+
/// SEC-010 scope note: unlike [`deserialize_catalog_response`], this is the NON-2xx path, and it
650+
/// deliberately still attaches the body. A non-2xx response means the request FAILED — no table was
651+
/// loaded, so no `config` overlay and no vended `storage-credentials` were issued — and the body is
652+
/// an `ErrorResponse` whose `message` is the whole diagnostic value. Java surfaces exactly this to
653+
/// the caller (`ErrorHandlers` parse the payload and rethrow `ErrorResponse.message`), so
654+
/// withholding it would cost real debuggability and diverge from Java for no security gain. The one
655+
/// non-2xx body that COULD echo submitted credentials — the token endpoint's — is handled
656+
/// separately in [`HttpClient::exchange_credential_for_token`], which withholds it.
618657
pub(crate) async fn deserialize_unexpected_catalog_error(
619658
response: Response,
620659
disable_header_redaction: bool,

0 commit comments

Comments
 (0)