OpenAPI: Recommend a standard User-Agent format for REST clients by rahulsmahadev · Pull Request #17727 · apache/iceberg

rahulsmahadev · 2026-08-19T20:57:26Z

What

Adds a recommended, consistent User-Agent format to the REST spec so clients identify themselves the same way, and documents that client identification is optional and informational only.

Spec-only proposal

Why

Clients identify themselves inconsistently today, and the most-deployed one doesn't by default:

Client	`User-Agent`	`X-Client-Version`
iceberg-java	none by default (opt-in `rest.client.user-agent`)	`Apache Iceberg <ver> (commit <sha>)` — library version
pyiceberg	`PyIceberg/<ver>`	`PyIceberg <ver>` — library version
iceberg-rust	`iceberg-rs/<ver>`	`0.14.1` — REST spec version
iceberg-go	`GoIceberg/<ver>`	REST spec version

A catalog operator can't reliably tell what's calling: the Java client sends no User-Agent unless configured, the naming differs across clients, and X-Client-Version carries the library version in two clients and the spec version in the other two. None of it is described in the spec.

A single recommended format gives operators usable telemetry for observability, debugging, and support, and an Iceberg library token that's always present regardless of the embedding engine.

Proposal

Use the standard User-Agent header (RFC 7231): whitespace-separated product/version tokens, most-specific first (engine → integration → Iceberg library → runtime), with a parenthesized comment as an open extension point.

Spark/4.0.0 iceberg-spark/1.9.0 iceberg-java/1.9.0 (scala/2.13.16)

The Iceberg library token (iceberg-java, pyiceberg, iceberg-rust, iceberg-go) is the piece every client can supply and should always be present.
Optional and informational: servers must not reject on missing/malformed, must not use it for auth or trust decisions, and it is not capability negotiation — that's the versioned-endpoints discussion from Core, OpenApi: Add X-Iceberg-Client-Capabilities header #16394, kept deliberately separate.

Scope / follow-ups

This PR is the spec recommendation. If the format lands, client conformance follows as separate PRs:

iceberg-java: give rest.client.user-agent a default library token and a way for integrations (Spark/Flink/…) to contribute engine tokens.
pyiceberg / iceberg-rust / iceberg-go: emit the same format.

singhpk234 · 2026-08-21T06:12:30Z

This is a nice addition, can you please open a dev DISCUSS thread as well !

rahulsmahadev · 2026-09-02T01:55:38Z

@singhpk234 thanks! Opened a DISCUSS thread on the dev list: https://lists.apache.org/thread/d6hkx66o8qc1lolh11l1lmswxlq0bpj9

adutra · 2026-09-02T16:41:07Z

+        Examples:
+
+
+        `pyiceberg/0.11.0 (cpython/3.11.4)` — a client library used directly.


Out of curiosity, does the RFC accept product identifiers surrounded by parentheses?

Good catch — no, the parens are an RFC comment, not product tokens. The runtime bits belong there as free-form context. Clarified the wording and fixed the examples so the parenthesized part reads as a comment (e.g. (jvm 17.0.9)).

adutra · 2026-09-24T15:23:21Z

-        Recommended library tokens are `iceberg-java`, `pyiceberg`,
-        `iceberg-rust`, and `iceberg-go`.
+        then any integration or connector, then the Iceberg library, then the
+        language runtime.


Language runtime is mentioned twice: here as a product token, then below (line 2212) as a comment. I think the final intent is to move language runtime info to comments.

adutra · 2026-09-24T15:36:26Z

-        and MUST NOT depend on its contents.
+        The optional trailing comment, enclosed in parentheses as defined by
+        the RFC, carries lower-value context such as build identifiers, the
+        HTTP library, the language runtime, or the operating system. Everything


It would be good to clarify:

whether all of these items should appear in different comments, or together in a single one, or both? E.g.

(apache-http 5.0; jvm 17.0.9; ubuntu 26.04) (apache-http 5.0) (jvm 17.0.9) (ubuntu 26.04)

whether comments can appear interleaved with products, or only at the end after the products.

The RFC allows all of the above, but maybe we could restrict the expected format to facilitate parsing.

rahulsmahadev

Hi @adutra Thanks for your feedback so far, addressed your comments lmk if it looks better, I can re-open the voting once we are in agreement on this PR

adutra · 2026-10-02T14:10:46Z

Thanks @rahulsmahadev LGTM!

sungwy · 2026-10-05T16:15:39Z

Thanks for the PR @rahulsmahadev - it looks like the CI is failing due to CVE-2026-68497 (#18318)

Could you please rebase it on main and let the CI run one more time?

manuzhang · 2026-10-07T16:04:01Z

@rahulsmahadev please rebase to fix CVE failures.

Adds a recommended, consistent User-Agent format to the REST spec so clients identify themselves the same way, and documents that client identification is optional and informational only (not authentication, not capability negotiation). Spec-only; no behavior change. Client libraries would be updated to emit this format as follow-ups.

- Require the Iceberg library token when the header is sent, and make the format normative rather than "recommended". - Clarify that the parenthesized part is a free-form RFC comment, not a product token, and update the examples accordingly. - Allow servers to refuse operations from client versions known to be unsafe (e.g. corrupt statistics), while still not rejecting merely on a missing or malformed header.

…nt format Address review feedback on the client User-Agent parameter: make the Iceberg library list illustrative rather than exhaustive (add iceberg-cpp, note proprietary or custom implementations use their own product token); drop the language runtime from the product-token ordering since it belongs in the trailing comment; and recommend grouping context items into a single trailing comment placed after all product tokens, semicolon-separated.

Clarify that refusing a known-bad client version applies only to a header that names that version, and is not an exception to the rule that an absent or malformed header is not a reason to refuse a request. Add an example with both a runtime and an operating system, and show that the comment format extends to any number of items.

rahulsmahadev · 2026-10-09T01:32:30Z

@rahulsmahadev please rebase to fix CVE failures.

Done

rahulsmahadev · 2026-10-13T22:31:53Z

@singhpk234 and @openinx can I get a review

rahulsmahadev · 2026-10-16T01:29:49Z

Friendly ping for review @manuzhang @singhpk234 @openinx

The user-agent header parameter was not referenced by any operation, so renderers dropped it. Fold its format rules into the Client Identification section, cite RFC 9110, and clarify the token order, how a server finds the library token, the capability negotiation rule, and how a server refuses a known-bad client version.

rahulsmahadev · 2026-10-26T22:03:48Z

@manuzhang addressed your comments, PTAL

openinx · 2026-10-28T18:05:38Z

Approved, but I will wait @manuzhang @singhpk234 for the approval before I merge it. thanks @rahulsmahadev for the work.

manuzhang · 2026-10-29T00:44:24Z

LGTM. Thanks for the PR.

rahulsmahadev · 2026-10-29T22:33:33Z

Thanks @openinx and @manuzhang! @singhpk234, could you take a look when you get a chance?

rahulsmahadev · 2026-11-02T15:28:36Z

@openinx manuzhang has LGTM'd, and singhpk234 hasn't had a chance yet. Would you be OK merging this?

singhpk234 · 2026-11-02T16:48:43Z

+    - When the header is sent, the token of the Iceberg client library MUST be
+      present, for example `iceberg-java`, `pyiceberg`, `iceberg-rust`,
+      `iceberg-go`, or `iceberg-cpp`. This list is illustrative, not


already released client don't send iceberg-java in the UA what we observe is Apache HTTP client version ? this should be a SHOULD right considering already released SDK

Right, released clients don't send it. Today the Java client only sets a User-Agent when rest.client.user-agent is configured, so servers see the Apache HttpClient default (they can still identify old Java clients from the X-Client-Version header). The spec can't change what's already shipped, which is why the advisory rules say servers SHOULD NOT reject a request whose header is absent, incomplete, or malformed. The MUST is for clients that follow this format from now on, so a server can always rely on the library token being there. With a SHOULD, it couldn't rely on it even for new clients. This is also the wording the vote passed on. The client changes that add the token (Java, Python, Go, Rust) are ready and will go out once this merges.

The MUST is for clients that follow this format from now on, so a server can always rely on the library token being there

I am not sure how does this literature defines now on for example there are already released version of the SDK which this MUST puts it into questionable situation in past we have taken same stance please ref #16822 (comment) @nastra thoughts ?

Fair point, and #16822 is a good precedent. Changed it to SHOULD. The Java, Python, Go and Rust clients will all send the token once their follow-up changes land, so servers will still see it in practice.

yes I agree, it should be updated to SHOULD

singhpk234

thanks for the ping @rahulsmahadev !
added some comments inline

Released clients do not send the library token, so a MUST would leave them out of compliance. apache#16822 made a similar client rule a SHOULD for the same reason. Co-authored-by: Isaac <[email protected]>

nastra · 2026-11-04T07:50:56Z

+      advertise the protocol features they support, and servers MUST NOT infer
+      from it which features a client supports. Servers advertise the
+      endpoints they support in the `endpoints` field of the config response.
+    - Servers MAY refuse specific operations when the header identifies a


I don't think we should have this statement here. Isn't the header purely informational?

This came from Dan's review (#17727 (comment)): a catalog should be able to block a client version known to have a critical bug, for example one that writes corrupt statistics. It was also part of the text we voted on. The header is informational for everything else, and this is kept narrow: it's a MAY, it only applies when the header names a known-bad version, and an absent or malformed header is never a reason to refuse. @danielcweeks can you weigh in?

I agree with @nastra here. We can just remove the language. We don't want to say servers can/can't do anything with it.

Makes sense. Dan agreed it's over-specifying, so I removed it.

The spec does not need to say when a server may refuse a request, so the rule only over-specified server behavior. Co-authored-by: Isaac <[email protected]>

nastra · 2026-11-06T18:08:38Z

+      system. These items SHOULD be grouped into a single comment after all
+      product tokens and separated by `; `, for example
+      `(jvm 17.0.9; ubuntu 26.04)` or
+      `(cpython 3.11.4; urllib3 2.2.1; macos 15.1)`. The comment is free-form:


nit: I feel like the second example should fit into the previous line

Co-authored-by: Isaac <[email protected]>

rahulsmahadev · 2026-11-09T15:53:16Z

@openinx nastra and Prashant have approved now. Since the vote, the text changed in two ways, both loosening it: the library token went from MUST to SHOULD, and the rule letting servers refuse known-bad versions is gone. Would you be OK merging this?

szehon-ho

this makes sense to me, the format is now pretty clear (space delimited and then comment in parens). will merge if no other comments!

laskoviymishka

The vote passed, it's ready to merge

…3668) The REST catalog spec recommends a standard User-Agent format whose library token for Rust is iceberg-rust (apache/iceberg#17727), so send iceberg-rust/<version> instead of iceberg-rs/<version>. A header.user-agent property still overrides the default.

Send iceberg-go/<version> (go <go version>) as the default User-Agent, following the format recommended by the REST catalog spec (apache/iceberg#17727). Headers configured by the caller still take precedence over the default.

* REST: Use the standard User-Agent format Send pyiceberg/<version> (<python implementation> <python version>), for example pyiceberg/0.11.0 (cpython 3.11.4), the format recommended by the REST catalog spec (apache/iceberg#17727). Co-authored-by: Isaac <[email protected]> * Use a loop for User-Agent test cases Co-authored-by: Copilot App <[email protected]> --------- Co-authored-by: Isaac <[email protected]> Co-authored-by: Fokko Driesprong <[email protected]> Co-authored-by: Kevin Liu <[email protected]> Co-authored-by: Copilot App <[email protected]>

github-actions Bot added the OPENAPI label Aug 19, 2026

rahulsmahadev marked this pull request as ready for review August 19, 2026 23:54

singhpk234 self-requested a review August 21, 2026 06:12

adutra reviewed Sep 2, 2026

View reviewed changes

danielcweeks reviewed Sep 2, 2026

View reviewed changes

Comment thread open-api/rest-catalog-open-api.yaml Resolved

adutra reviewed Sep 24, 2026

View reviewed changes

danielcweeks reviewed Sep 29, 2026

View reviewed changes

Comment thread open-api/rest-catalog-open-api.yaml Resolved

rahulsmahadev requested review from adutra and danielcweeks September 30, 2026 02:13

rahulsmahadev commented Sep 30, 2026

View reviewed changes

rahulsmahadev force-pushed the rest-user-agent-proposal branch from 617b0d5 to 5943e8f Compare September 30, 2026 05:36

adutra approved these changes Oct 2, 2026

View reviewed changes

openinx reviewed Oct 5, 2026

View reviewed changes

Comment thread open-api/rest-catalog-open-api.yaml Resolved

openinx reviewed Oct 5, 2026

View reviewed changes

Comment thread open-api/rest-catalog-open-api.yaml Resolved

rahulsmahadev added 5 commits October 8, 2026 22:49

OpenAPI: Soften User-Agent rejection rule to SHOULD NOT per review

6376a43

rahulsmahadev force-pushed the rest-user-agent-proposal branch from 5943e8f to 5534ff9 Compare October 9, 2026 00:56

dhruvarya-db mentioned this pull request Oct 12, 2026

feat(rest): send the spec's recommended User-Agent format apache/iceberg-rust#3421

Closed

rahulsmahadev requested a review from openinx October 13, 2026 22:31

manuzhang requested review from flyrain and stevenzwu October 23, 2026 15:00

openinx approved these changes Oct 28, 2026

View reviewed changes

singhpk234 reviewed Nov 2, 2026

View reviewed changes

Comment thread open-api/rest-catalog-open-api.yaml Resolved

OpenAPI: Soften User-Agent library token rule to SHOULD per review

cc19974

Released clients do not send the library token, so a MUST would leave them out of compliance. apache#16822 made a similar client rule a SHOULD for the same reason. Co-authored-by: Isaac <[email protected]>

nastra reviewed Nov 4, 2026

View reviewed changes

OpenAPI: Remove User-Agent refusal rule per review

85d78c3

The spec does not need to say when a server may refuse a request, so the rule only over-specified server behavior. Co-authored-by: Isaac <[email protected]>

nastra reviewed Nov 6, 2026

View reviewed changes

nastra approved these changes Nov 6, 2026

View reviewed changes

singhpk234 approved these changes Nov 7, 2026

View reviewed changes

OpenAPI: Keep the User-Agent comment examples on one line per review

0422985

Co-authored-by: Isaac <[email protected]>

szehon-ho approved these changes Nov 10, 2026

View reviewed changes

laskoviymishka approved these changes Nov 10, 2026

View reviewed changes

laskoviymishka merged commit 0d07c41 into apache:main Nov 10, 2026
24 checks passed

rahulsmahadev mentioned this pull request Nov 13, 2026

Send the pyiceberg User-Agent on S3 remote signing requests apache/iceberg-python#4367

Merged

		Examples:


		`pyiceberg/0.11.0 (cpython/3.11.4)` — a client library used directly.

Conversation

rahulsmahadev commented Aug 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Why

Proposal

Scope / follow-ups

Uh oh!

singhpk234 commented Aug 21, 2026

Uh oh!

rahulsmahadev commented Sep 2, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

rahulsmahadev left a comment

Choose a reason for hiding this comment

Uh oh!

adutra commented Oct 2, 2026

Uh oh!

sungwy commented Oct 5, 2026

Uh oh!

Uh oh!

Uh oh!

manuzhang commented Oct 7, 2026

Uh oh!

rahulsmahadev commented Oct 9, 2026

Uh oh!

rahulsmahadev commented Oct 13, 2026

Uh oh!

rahulsmahadev commented Oct 16, 2026

Uh oh!

rahulsmahadev commented Oct 26, 2026

Uh oh!

openinx commented Oct 28, 2026

Uh oh!

manuzhang commented Oct 29, 2026

Uh oh!

rahulsmahadev commented Oct 29, 2026

Uh oh!

rahulsmahadev commented Nov 2, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

singhpk234 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

rahulsmahadev commented Aug 19, 2026 •

edited

Loading

szehon-ho left a comment •

edited

Loading