gen-ai: add optional byte_size to multimodal content parts#143
gen-ai: add optional byte_size to multimodal content parts#143Mandark-droid wants to merge 1 commit into
Conversation
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
Adds an optional byte_size field to GenAI multimodal content part schemas to consistently report payload size across inline blobs, provider file IDs, and URIs.
Changes:
- Add
byte_size: Optional[int]toBlobPart,FilePart, andUriPartnon-normative models and regenerate the normative JSON schemas. - Update a multimodal example to include
byte_size. - Document the enhancement in
CHANGELOG.md.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| docs/gen-ai/non-normative/models.ipynb | Adds byte_size to the reference (non-normative) Pydantic models for three part types. |
| docs/gen-ai/non-normative/examples-llm-calls.md | Demonstrates byte_size usage in a multimodal URI example. |
| docs/gen-ai/gen-ai-system-instructions.json | Regenerated schema including optional byte_size on relevant part definitions. |
| docs/gen-ai/gen-ai-output-messages.json | Regenerated schema including optional byte_size on relevant part definitions. |
| docs/gen-ai/gen-ai-input-messages.json | Regenerated schema including optional byte_size on relevant part definitions. |
| CHANGELOG.md | Records the new optional field under Unreleased enhancements. |
|
Thanks @Copilot for the review — all four items addressed in b6930e0:
|
| - Add optional `byte_size` field on `BlobPart`, `FilePart`, and `UriPart` in the GenAI | ||
| input/output/system-instructions message JSON schemas. Captures the size of the payload (in | ||
| bytes) at the point of capture, useful for cost-of-capture telemetry and storage planning when | ||
| the same logical request might be carried inline, by URI, or by provider file ID. |
|
Thanks @Copilot for the second pass — good catch. Addressed in d725a44: In the first fixup I broadened the description to mention BlobPart-specific base64 semantics, but applied it uniformly across all three parts. That was wrong: FilePart and UriPart have no
The notebook and the three generated JSON schemas are updated in lockstep; |
|
@trask done — PR body now filled out against the new template. Description / Motivation (with user journey + prior art) / Prototype / Checklist all in place. One checklist item left intentionally unchecked: Reference scenarios updated for affected libraries. I flagged this in the Prototype section — the repo's existing
If you prefer (1) or (2), I'll do it as part of this PR. If you'd rather land the schema changes first and follow up with the multimodal scenario(s) as a separate PR — also happy to do that, especially since the same scenario will want to exercise Also note PR #142 was already approved by @MikeGoldsmith on a deferred-scenario basis — if the same standard applies here #143 and #144 could land first and the scenario lands in a unified follow-up. Whatever you prefer. PR #144's body has been similarly filled out against the template; @MikeGoldsmith, flagging since you reviewed #142. |
|
let's add the scenario to #142, get that merged, then you can update that scenario in this PR |
Per @trask's direction on PR open-telemetry#143, the multimodal scenario lives in open-telemetry#142 so it can carry the document modality first, then be extended in open-telemetry#143 (byte_size) and open-telemetry#144 (stripped_reason) once this lands. Adds run_chat_with_document_input_reference(client) to the openai scenario, exercising the new 'document' enum value on a UriPart in gen_ai.input.messages. The OpenAI SDK call uses the documented file content block; the canonical OTel attribute carries a TextPart plus a document-modality UriPart, mirroring how a KYC document-extraction workload surfaces in the wire shape.
| "type": "uri", | ||
| "modality": "video", | ||
| "mime_type": "video/mp4", | ||
| "byte_size": 5242880, |
There was a problem hiding this comment.
For uri and file type responses, I suspect you may not actually be able to obtain the byte_size value, since it is generally only determined when the content is downloaded.
There was a problem hiding this comment.
Good catch, and the schema does handle this case — byte_size is Optional[int] precisely so instrumentations omit it when the value isn't cheaply observable. But it's worth saying why we wanted this field in the first place, because it shapes which cases legitimately have a size and which don't.
Motivation: trace-side metadata as a join key for offline multimodal evals.
The field exists to support a pattern we ship in genai-otel-instrument: optional offload of multimodal payloads (images / audio / video / documents) to an object store — MinIO in our default deployment, configurable to any S3-compatible target — while spans carry only lightweight metadata (modality, MIME, byte size, source, SHA, store URI). The bytes themselves never go on the span. The store becomes the substrate for downstream evaluation: LLM-as-judge passes over generated images, voice-quality scoring on audio responses, OCR/grounding checks on document inputs, joint text+image evals on text-to-image flows (judge both the prompt and the produced artifact). As the industry moves toward voice and multimodal use cases this becomes the dominant eval shape, and trace ↔ artifact correlation is what makes it tractable.
byte_size plays two roles in that pattern:
- Standalone signal when the offload sink is off. Even with no object store wired up, byte size on the span gives operators a meaningful low-cost dimension — cost attribution (multimodal billing scales with payload size), spotting oversized or empty inputs, anomaly detection. It's a single integer; negligible on the wire.
- Correlation / integrity key when the sink is on. Pairs with
mime_type+ (optionally)sha256to let an evaluator join the lightweight trace to the heavyweight artifact in the store, and to detect truncation/corruption between span emission and storage.
Which UriPart / FilePart cases actually have a known size — concretely, in our reference impl:
There are three code paths that produce a UriPart-shape, and byte_size is set on two of them:
- Inline bytes → uploaded to object store → UriPart (
media/offload.py:142-147). We started with inlinedata:URIs or raw bytes from the SDK, computedlen(payload), uploaded, and emit a UriPart pointing at the upload. Size is exact because we held the bytes. This is the primary "interesting" case — it's also the case wherebyte_sizeis most load-bearing for evals, because it lets the evaluator verify the artifact in the store matches what the trace says was sent. - Inline bytes too large to upload → stripped UriPart-shape (
media/offload.py:114-120). We setmedia_byte_size = len(part.data)before dropping the buffer, then emit a stripped part withstripped_reason: "size_exceeded". Size is the entire reason this branch exists — without it, consumers can't tell why the part was stripped or estimate what was lost. - Caller-supplied external URL → UriPart (
media/offload.py:95-99). User passed an already-remote URL (e.g. OpenAIimage_urlwithhttps://…). We recordmedia_uri = part.external_urland do not setmedia_byte_size. The canonical writer atmedia/canonical.py:66-67only emitsbyte_sizewhen it's notNone, so this UriPart goes out without it — exactly what you'd want.
The same logic applies to FilePart: when an app references a provider-side file (OpenAI Files API returns bytes, Anthropic Files returns size_bytes, Gemini File.sizeBytes), the SDK response carries the size, so it's known without a download. When it isn't carried, the field stays unset.
On the example you flagged (examples-llm-calls.md:245, "vendor-specific URI" video): you're right that as written it implies an instrumentation magicked the size out of a remote URL. In our impl's terms that's the external_url path, where byte_size is not set. I'll push an example fix that drops byte_size from that specific UriPart entry and adds a brief note explaining that byte_size on UriPart is only populated when the instrumentation observed the bytes before they became a URI — typically the offload-to-object-store flow this field is designed to support. Happy to instead replace with an inline-then-offloaded UriPart (where size is genuinely known) plus a sibling URI example without byte_size if you'd prefer the canonical "size on UriPart is legitimate" case demonstrated explicitly. LMK.
Address review feedback on PR open-telemetry#143: a pre-existing remote URI's size generally is not observable to instrumentation without downloading the content, so the vendor-specific URI example should not carry byte_size. Set byte_size on a uri part only when bytes were observed before they became a URI (e.g. inline payload uploaded to an object store). Also add byte_size to the inline blob image example where the value is unambiguous.
* gen-ai: add document modality to multimodal content parts Adds the 'document' value to the Modality enum in the gen-ai input, output, and system-instructions message JSON schemas. Today PDF/DOCX and similar parts have to fall through to the free-form string branch of the modality anyOf; this change gives them a first-class enum value matching the existing image/video/audio entries. Pure addition: no removals, no changes to required fields. The notebook source under docs/gen-ai/non-normative/models.ipynb has been updated to keep the regen-from-notebook flow consistent, and a worked example was added to examples-llm-calls.md. Follow-up from open-telemetry/semantic-conventions#3673 (closed and refiled here per the new repo home for gen-ai semantic conventions). * gen-ai: add document-modality reference scenario to openai Per @trask's direction on PR #143, the multimodal scenario lives in #142 so it can carry the document modality first, then be extended in #143 (byte_size) and #144 (stripped_reason) once this lands. Adds run_chat_with_document_input_reference(client) to the openai scenario, exercising the new 'document' enum value on a UriPart in gen_ai.input.messages. The OpenAI SDK call uses the documented file content block; the canonical OTel attribute carries a TextPart plus a document-modality UriPart, mirroring how a KYC document-extraction workload surfaces in the wire shape. * gen-ai: apply ruff format to multimodal scenario block * gen-ai: address scenario-evaluation feedback on PR #142 @trask's reference-scenario SKILL evaluation flagged that the previous openai multimodal block fabricated a URI and mime_type that aren't on the chat.completions.create boundary -- file_id is opaque, neither yields a public URI or mime type without a separate Files-API roundtrip. Two fixes per the evaluation: 1. Rewrite the openai scenario to use the inline file_data shape ({'type': 'file', 'file': {'file_data': 'data:application/pdf;base64,...', 'filename': '...'}}) and emit a BlobPart instead of a UriPart. Every emitted BlobPart field now derives directly from the SDK arg: - mime_type from the data-URI prefix - content from the base64 portion - modality 'document' from the classification of application/pdf No fabricated URIs. 2. Add run_chat_with_document_input() to the anthropic scenario. Anthropic's Messages API has a first-class document content block ({'type': 'document', 'source': {'type': 'base64', 'media_type': '...', 'data': '...'}}), so the BlobPart emission is directly observable from the SDK call boundary with no derivation gymnastics. Both scenarios run cleanly against the local mock server; weaver live-check reports no after_resolution policy violations on either. The bedrock-converse demonstration trask also suggested is deferred to a follow-up (DocumentBlock has the same shape; happy to add if needed). Addresses #142 (comment)... * gen-ai: add Bedrock Converse DocumentBlock reference + ruff import-sort fix Completes @trask's three-scenario request for PR #142: - openai: inline file_data + BlobPart (previous commit, derives mime/content/modality from SDK arg) - anthropic: native document content block + BlobPart (previous commit) - aws-bedrock: native Converse DocumentBlock + BlobPart (this commit) -- the DocumentBlock 'format' / 'source.bytes' fields map 1:1 to the BlobPart mime_type/content with zero out-of-band derivation. Most direct of the three. Also applies the ruff isort fix from this run (I001 in the anthropic scenario file: import base64 should sort after import anthropic in the new function -- ruff check --fix moved it). All three scenarios validated locally against the mock LLM server; weaver live-check reports no after_resolution policy violation on any of them. ruff format + ruff check both clean. * gen-ai: drop incorrect filename claim from anthropic doc-input docstring Round-3 Copilot review on PR #142: the docstring claimed Anthropic's document content block exposes 'mime type, source bytes, and filename' on the SDK boundary, but the scenario does not include a filename field in its payload. Removing the filename claim keeps the docstring accurate to the scenario; if/when a filename field is added to the schema in a follow-up (per issue #50), the docstring can be updated alongside.
| "description": "The general modality of the data if it is known. Instrumentations SHOULD also set the mimeType field if the specific type is known.", | ||
| "title": "Modality" | ||
| }, | ||
| "byte_size": { |
There was a problem hiding this comment.
how would instrumentation capture this on FilePart and UriPart?
There was a problem hiding this comment.
The answer differs by part:
FilePart — the size comes back in the SDK response, no download needed. When an app references a provider-side file, the file-create / file-get call already returns the size: OpenAI Files bytes, Anthropic Files size_bytes, Gemini File.size_bytes. The instrumentation reads it off the same object it's already inspecting to build the part. When a provider doesn't surface it, the field stays unset (Optional).
UriPart — two sub-cases:
- Instrumentation-produced URI (the case the field is really for): the instrumentation held the bytes inline, computed
len(payload), uploaded them to an object store, and emitted a UriPart pointing at the upload. Size is exact because the bytes were in hand before they became a URI. This is the offload-to-object-store flowbyte_sizeis designed for — it lets an offline evaluator join the lightweight span to the heavyweight artifact and detect truncation. - Caller-supplied pre-existing remote URL (e.g. an
image_urlthat's alreadyhttps://…): the size generally isn't observable without downloading, sobyte_sizeis not set. That's the misleading example you and @Cirilla-zmh flagged; it's corrected in 77d45f8 — the remote-URI example no longer carriesbyte_size, with a note that it's only populated when the bytes were observed before becoming a URI, and the inline-blob example gains it where the value is unambiguous.
BlobPart — agreed it's derivable from the base64 length, so on its own it's redundant. The reason to still allow it is uniformity across the three part types, and — as you noted — it's the only surviving size signal once #144 lets content be omitted: {"type":"blob","modality":"image","stripped_reason":"size_exceeded","byte_size":2000000} tells a consumer how much was dropped. So byte_size and #144 are complementary by design.
If you'd rather land this narrowly, I can take the same approach as #144 — scope this PR to byte_size on input-messages BlobPart only and replicate to FilePart/UriPart plus the other two schemas in a follow-up.
There was a problem hiding this comment.
can you add reference scenarios for FilePart and UriPart?
| "description": "The general modality of the data if it is known. Instrumentations SHOULD also set the mimeType field if the specific type is known.", | ||
| "title": "Modality" | ||
| }, | ||
| "byte_size": { |
There was a problem hiding this comment.
on BlobPart, it's a bit redundant with content since easy to calculate from the base64 length
I think would be ok to capture on BlobPart for consistency with others, but I'm not sure about the others (see other comment), which is why I mention
There was a problem hiding this comment.
oh, I see, this would be useful with #144 which makes content optional
| "video", | ||
| "audio" | ||
| "audio", | ||
| "document" |
77d45f8 to
8d5a161
Compare
|
Rebased onto Reference-scenario emission is deferred to the follow-up that will carry document-modality, |
|
Hi! As part of #275, this repository switched to Towncrier changelog fragments to reduce merge conflicts in Please move this PR's changelog entry out of Create Add optional `byte_size` (non-negative integer) to `BlobPart`, `FilePart`, and `UriPart` in the GenAI input/output/system-instructions message JSON schemas. Lets instrumentations record the size of a captured media payload for cost-of-capture telemetry and storage planning; omitted when the size is not cheaply observable (e.g. a pre-existing remote URI).After adding the fragment, please remove this PR's direct edit to Thanks! |
Add an optional non-negative byte_size to BlobPart, FilePart, and UriPart in the gen_ai input/output/system-instructions message JSON schemas. Lets an instrumentation record the size of a captured media payload for cost-of-capture telemetry and storage planning. Omitted when the size is not cheaply observable - e.g. a pre-existing remote URI, where the bytes would have to be downloaded to measure; set only when the bytes were observed before becoming a URI. Per-part descriptions reflect this (decoded byte length for BlobPart, file size for FilePart, payload size for UriPart). Examples updated accordingly. Rebased onto main after the Towncrier migration (open-telemetry#275) and the models.ipynb -> models.py conversion (open-telemetry#226): the field is now declared in models.py and the committed JSON schemas are regenerated via make generate-json-schemas; the changelog entry moved from CHANGELOG.md to changelog.d/143.enhancement.md.
8d5a161 to
be20d01
Compare
|
@trask done — the changelog entry now lives in While rebasing onto current |
Pull request dashboard statusStatus last refreshed: 2026-07-20 23:03:39 UTC.
This automated status or its linked feedback items may be incorrect. If something looks wrong, report it with the result you expected. |
Description
Adds an optional
byte_sizefield onBlobPart,FilePart, andUriPartin the gen-ai input, output, and system-instructions message JSON schemas.byte_sizecaptures the size, in bytes, of the captured or referenced media payload as observed by the instrumentation at the time of capture.Concretely each part type carries its own focused description:
byte_sizedescriptionBlobPartcontentstring on the wire.FilePartUriPartThe Pydantic source models in
docs/gen-ai/non-normative/models.ipynbdeclarebyte_size: Optional[int] = Field(default=None, ge=0, ...)so the regenerated JSON schemas carry"minimum": 0inside theanyOfinteger branch — negative values are rejected by validators rather than silently accepted.Follow-up from the closed PR open-telemetry/semantic-conventions#3673 (split into three small PRs per the new repo's "keep PRs small" guidance: this one, the
documentmodality PR #142, and thestripped_reasonPR #144). The three are independent and additive.Motivation
User journey: cost-of-capture telemetry, storage planning, and capacity forecasting for content-capturing instrumentation.
A single logical multimodal request may be carried inline (
BlobPartwith base64content), by URI (UriPartpointing at S3 / GCS / MinIO / similar), or by provider-side file id (FilePart). Without a uniformbyte_size, an operator who wants to answer "how much media data are we ingesting per minute / per tenant / per modality?" has to:BlobPart: base64-decodecontentand count bytes (and decide whether to do that per-span, which is expensive).UriPart: dereference the URI to a backing store and HEAD it (involves an extra request and trust in the store).FilePart: ask the provider's files API for size metadata (extra API call, rate-limited, provider-specific).Putting an optional
byte_sizedirectly on the captured part lets the producing instrumentation record what it already knows (it just saw the bytes or just got back acontent-lengthheader / file metadata) so consumers don't have to re-derive it later.Prior art:
byteson every file object — instrumentation that's about to upload viaFiles.create()already has this number.genai-otel-instrument(v1.1.1) already populates an equivalent inside its dual-emission JSON whenOTEL_SEMCONV_STABILITY_OPT_IN=gen_ai— this PR upstreams the convention.Prototype
This PR ships the schema change in
docs/gen-ai/. The repo's reference scenarios underreference/scenarios/<library>/are all currently text-only — none emitBlobPart/FilePart/UriParttoday — so there is no obvious existing scenario to addbyte_sizeto without first introducing a multimodal scenario.I flagged this as an open question in the original PR body (and on the closed-and-refiled open-telemetry/semantic-conventions#3673): would maintainers prefer (a) extending an existing
<library>scenario (e.g.openai) with a multimodal call that emitsbyte_size, or (b) a dedicatedopenai-multimodal(or similar) scenario, given the SDK call shape changes? Happy to do either as a follow-up — leaving this checkbox unchecked rather than guessing the shape. Cross-referenced from #142, where the document-modality counterpart of this question was raised and approved without a scenario.End-to-end producer verification: the
genai-otel-instrumentlibrary emitsbyte_sizeon URI parts captured by its OpenAI / Anthropic / Bedrock instrumentors, validated through an OTel collector → OpenSearch pipeline against a MinIO-backed media store; the actual size value observed at capture time round-trips into thegen_ai.input.messagesspan attribute as the JSON schema in this PR describes.Checklist
UnreleasedinCHANGELOG.md