Skip to content

security: findings from a read-only scan of main at 388a0441b #1680

Closed

Description

@geoHeil

A read-only scan of main at 388a0441b against the secure-by-design rules: inbound identity, scopes, every transport, SQL binding per dialect, brokers and delegation, startup refusals, secrets in logs, bounds on untrusted input, the multi-node worker path, the operator and the chart, TLS, CI and deny.toml.

Limits. Every finding comes from reading source; nothing was built, run, fuzzed or reproduced. Two claims were re-read in the pinned sources before filing: datafusion-proto 55.1.0 panics on an Avro scan node without its avro feature, and axum 0.8.9 serves each connection with no timer, so hyper drops its 30-second header-read default. The release profile is panic = "abort". Not covered: the internals of the vendored and Go/C drivers beyond the one logging path cited, the image build, anything dynamic. No SQL injection was found: on every adapter a caller's value reaches the data system only as a bound parameter, and identifiers come from the catalog through a fixed character set.

Each finding names the smallest fix and the test or gate that would hold it, red today. Each is worked as its own PR that says Part of this issue.

1. Refuse a peer node's malformed plan or reply without stopping the process

  • Severity: medium. In scope per SECURITY.md: yes - the shipped binary, a shipped dependency, and the input stops the process.
  • Where: crates/sutura-cli/src/serve/nodes.rs:44, crates/sutura-cli/src/serve/worker.rs:60, crates/sutura-http/src/router/worker.rs:39, crates/sutura-exec-datafusion/src/combine/across.rs:219,505; upstream datafusion-distributed 4.0.0 impl_coordinator_channel.rs:77, datafusion-proto 55.1.0 physical_plan/mod.rs:1106, arrow-ipc 59.2.0 reader.rs:66 and compression.rs:313.
  • What: with nodes declared, a serving node mounts its worker on the listener that serves /v1 and /mcp, gated only on a certificate the node authority signed. The worker decodes a peer's plan before reads_no_source inspects it. The decoder panics on an Avro scan node, arrow-ipc trusts the buffer offsets and uncompressed lengths a sender declares, and both ends set max_decoding_message_size(usize::MAX). Under panic = "abort" each of these stops the whole process, so one holder of a node certificate can stop every node, including the serving ones. The same IPC decode sits on the remote-leg parameter batch (crates/sutura-flight/src/receive.rs:87) and a peer's reply (crates/sutura-exec-node/src/call.rs:114); no release ships flight or node today.
  • Fix: return an error on the Avro arm - propose upstream, vendor datafusion-proto meanwhile under VENDOR.md; propose an offset and declared-length bound to arrow-ipc; cap the decoding message size on the server sutura builds and on its client.
  • Held by: a worker-route cell sending a plan with an Avro scan node that expects an error status; a fuzz target under just fuzz over the plan decode and a FlightData stream decode.
  • Related: the task grant (feat(exec): elastic multi-node DataFusion on demand, so a federated question too wide for one replica is answered rather than refused #828, Decision 5) does not cover this unless it is checked before the plan is decoded.

2. Bound how long the public listener waits for a request head

  • Severity: medium (low behind a proxy that times out heads). In scope per SECURITY.md: yes - sutura's own listener, reachable with no credential.
  • Where: crates/sutura-http/src/server.rs:205, crates/sutura-http/src/tls/listener.rs:44,51.
  • What: axum::serve sets no timer, so hyper drops its header-read timeout. Nothing else bounds an idle or slow connection, or the number of open ones, once a TLS handshake completes or a plaintext socket is accepted. The request timeout, the rate limiter and the body bound are middleware and run only after a full request head. A client with no credential can hold connections until file descriptors or memory run out. This applies wherever tlsTermination is in-process or the listener is exposed with no proxy in front.
  • Fix: serve through hyper-util's auto::Builder with a TokioTimer, an HTTP/1 header-read timeout, HTTP/2 keep-alive with a timeout, and a semaphore on open connections shaped like the handshake permits; keep the drain.
  • Held by: a cell that sends half a request head and expects the server to close the connection within the bound.

3. Refuse to start while the BigQuery driver's debug log is on

  • Severity: low - the trigger is an operator setting. In scope per SECURITY.md: yes - a token and a credential reach a log.
  • Where: crates/sutura-cli/src/main.rs:301, crates/sutura-oracle-connect/src/oracle_trace.rs:7, crates/sutura-exec-bigquery/src/adbc/subject.rs:71,313-320.
  • What: the linked BigQuery driver's Go auth library logs each HTTP request and response, headers and body, unredacted, whenever GOOGLE_SDK_GO_LOGGING_LEVEL is set. That would write the loopback subject-token request with its secret header and the caller's own identity assertion to stderr, outside tracing, breaking the bound subject.rs:71 states. sutura refuses the Oracle driver's equivalent variable but not this one. Read from the upstream source at the pinned versions; not reproduced against the linked driver.
  • Fix: a refusal beside oracle_trace for that variable at any value, called where the Oracle one is.
  • Held by: the refusal type and a cell that sets the variable and expects the refusal.

4. Name the Deployments the operator may scale, and give it no node client identity

  • Severity: low. In scope per SECURITY.md: no - deployment and RBAC hardening.
  • Where: charts/sutura/templates/worker-pool.yaml:122-129,186-200, charts/sutura/templates/_helpers.tpl:249, crates/sutura-operator/src/reconcile.rs:92,103,117, docs/adr/20261030193029-operator-sizes-the-worker-pool.md:85.
  • What: the operator's Role may scale every Deployment in its namespace, and it scales whatever a SuturaWorkerPool names, so a principal who may write those objects but not scale Deployments can have it scale another Deployment to zero. Its pods: patch is not limited to annotations. Its pod mounts a node certificate with client auth that passes the node gate on every node, though it reads only the CA bundle.
  • Fix: resourceNames on the scale rule; the operator refuses a spec naming any other Deployment; the operator gets only the CA bundle; the ADR says what pods: patch allows.
  • Held by: the chart golden under just chart, and an operator cell refusing a foreign Deployment name.

5. Any certificate from the node authority is a node, and the chart accepts a shared issuer

  • Severity: medium when the issuer is shared, otherwise none. In scope per SECURITY.md: no - it depends on the issuer an operator chooses; the chart can refuse the unsafe choice.
  • Where: crates/sutura-http/src/tls/authority.rs:53, crates/sutura-http/src/server.rs:169, charts/sutura/values.yaml:114-118, charts/sutura/templates/_helpers.tpl:213-220,246-249.
  • What: a node is any client whose certificate chains to the node authority; no name is checked, and a certificate with no extended key usage passes as a client. The chart takes issuer.kind: ClusterIssuer and any CA issuer. With a shared CA, any workload holding a certificate from it can run stages on every node, make a worker dial node addresses a plan names, and, given a stage key, read another question's stage output.
  • Fix: the chart refuses ClusterIssuer unless acknowledged and asks for a dedicated CA; the task grant (feat(exec): elastic multi-node DataFusion on demand, so a federated question too wide for one replica is answered rather than refused #828, Decision 5).
  • Held by: a refusal fixture under charts/sutura/testdata/refusals.

6. No NetworkPolicy for the worker, serving or operator pods

  • Severity: low. In scope per SECURITY.md: no - deployment hardening.
  • What: the headless workers Service is reachable from every namespace; only the certificate check of finding 5 gates it.
  • Fix: an opt-in policy in the chart admitting the serving pods and the operator to the workers, pinned by a chart golden.

7. Images are pulled by tag, and the chart cannot take a digest

  • Severity: low. In scope per SECURITY.md: no - supply-chain hardening.
  • Where: charts/sutura/templates/_helpers.tpl:286-292.
  • What: the serving, worker and operator images render as repository:tag; a repushed tag swaps the binary with no chart change.
  • Fix: an image.digest value rendered as repository@digest, pinned by a chart golden.

8. A loopback subject-token URL can reach the error log through the driver's error text

  • Severity: informational. In scope per SECURITY.md: yes in class, no practical impact today - the listener closes on drop and its secret header is still needed.
  • Where: crates/sutura-exec-bigquery/src/adbc/subject.rs:318, crates/sutura-exec-bigquery/src/adbc/driver_message.rs:45, crates/sutura-http/src/routes/v1/query.rs:414.
  • What: when fetching the subject token fails, the driver's error text can carry the loopback URL with its nonce into the logged cause chain; driver_message.rs strips only console links. The guarantee at subject.rs:71 is worded stronger than this. Unverified that the driver passes the text through verbatim.
  • Fix: strip loopback URLs in driver_message.rs, held by a cell over such a text.

9. Unbounded reads of operator-supplied files

  • Severity: low. In scope per SECURITY.md: no - the inputs are the operator's own.
  • Where: crates/sutura-cli/src/password_file.rs:28, crates/sutura-catalog-wren/src/lib.rs:77.
  • What: a password file and a wren manifest are read with read_to_string: no size cap, a FIFO hangs the read, and the password's untrimmed copy is never zeroized.
  • Fix: read both through sutura-bounded-read.

10. The deny.toml zstd advisory ignores give no reachability reason

  • Severity: informational. In scope per SECURITY.md: no - tooling.
  • Where: deny.toml:28-30 (RUSTSEC-2026-0391, -0398, -0403).
  • What: read from the registry sources, no vulnerable entry point is reached - 0391 needs stream::raw::*::with_context, 0398 the seekable feature, 0403 legacy - but the ignores say only that DataFusion pins the version, so a later bump that reaches one is hidden. Not checked with a full feature resolution.
  • Fix: a reason per advisory, and a check that seekable and legacy stay off.

Not a security finding

A question with several metrics ordered by a guarded metric is refused with a placeholder-count mismatch on every dialect but Postgres (crates/sutura-sql/src/generate.rs:727-747, generate/top.rs:17-27). It fails closed; tracked in #1679.

🤖 Generated with Claude Code

Activity

  1. geoHeil commented on Nov 1, 2026

    @geoHeil
    MemberAuthor

    Upstream issues filed for finding 1, with the owner's approval:

    Not filed yet, until CI on the vendoring branches has reproduced the panics, so each issue can paste the real output:

    • the datafusion-proto-common from_proto panic sites;
    • the arrow-ipc buffer and decompression bounds.
  2. geoHeil commented on Nov 1, 2026

    @geoHeil
    MemberAuthor

    Finding 1: the datafusion-physical-plan decode panics (an Unnest recursion with an unset column, an Explain plan with an unset plan_type) are filed upstream as apache/datafusion#26999. #1703 tracks every upstream fix this finding waits on, and the proto-common and arrow-ipc issues will be added there once their cells' red CI run shows the panic.

  3. geoHeil commented on Nov 2, 2026

    @geoHeil
    MemberAuthor

    Status, 2026-11-02.

    Findings 2-10 are merged:

    Finding PR
    2. request-head bound #1685
    3. BigQuery driver debug-log refusal #1691
    4. operator scale rule names its Deployments #1704
    5. ClusterIssuer refusal; task grant #1681; #1688 (ADR #1683)
    6. opt-in NetworkPolicy #1682
    7. image digest #1684
    8. loopback subject URL cut from driver errors #1692
    9. bounded secret-file and manifest reads #1694
    10. zstd ignore reasons #1693

    Finding 1, merged: the AvroScan arm (#1690), the Unnest/Explain unwraps (#1710), the datafusion-proto-common panics (#1719), the server's decode cap (#1695), and the client's decode cap (#1715).

    Finding 1, still open:

    The upstream waits are tracked in #1703. This issue stays open until both items above land.

  4. geoHeil commented on Nov 2, 2026

    @geoHeil
    MemberAuthor

    Closing. The two items the 2026-11-02 status left open have both merged:

    Findings 1-10 are all merged. What stays open is upstream only, and #1703 tracks it: apache/datafusion#26998, #26999, #27004; datafusion-contrib/datafusion-distributed#828; apache/arrow-rs#12030. Removing each vendored patch waits on its release.

    Limits:

    • A 60-second fuzz run is a smoke check, not a soak.
    • The peer_stage harness rebuilds the worker's decode from public parts, and nothing checks that it stays in step with the worker's own session.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions