[gen-ai] Add multimodal additions: document modality, byte_size, StrippedPart#3673
[gen-ai] Add multimodal additions: document modality, byte_size, StrippedPart#3673Mandark-droid wants to merge 5 commits into
Conversation
…pedPart) to input messages schema
lmolkova
moved this from Untriaged
to Awaiting codeowners approval
in Semantic Conventions Triage
There was a problem hiding this comment.
please edit docs/gen-ai/non-normative/models.ipynb, these files should be generated from there, not manually edited.
There was a problem hiding this comment.
Thanks! Done in ae86b3f — moved the schema additions (document modality, byte_size, StrippedPart, StrippedReason) into docs/gen-ai/non-normative/models.ipynb and regenerated the JSON files from the notebook. Heads-up: regenerating with current pydantic surfaces a few emitter-style deltas vs. the previously-checked-in JSON (single-element oneOf → $ref, additionalProperties: true on BlobPart, key ordering inside anyOf branches). These aren't introduced by this PR but become visible now that the notebook is the source of truth — happy to roll them back or pin a pydantic version if you'd prefer the diff stay tighter.
Address review feedback on open-telemetry#3673: - Move the multimodal schema additions (document modality, byte_size, StrippedPart, StrippedReason) into docs/gen-ai/non-normative/models.ipynb and regenerate the JSON schemas from the notebook so they are no longer hand-edited (per @alexmojaki). - Add a worked multimodal example to docs/gen-ai/non-normative/examples-llm-calls.md demonstrating a document URI part with byte_size and a stripped video part with stripped_reason=size_exceeded (per @alexmojaki). Note: regenerating from the notebook with current pydantic surfaces some emitter-style differences vs. the previous hand-edited JSON (single-element $ref vs oneOf, additionalProperties:true on BlobPart, key ordering inside anyOf branches). These are not introduced by this change set but become visible once the notebook becomes the source of truth.
Address review feedback on open-telemetry#3673: - Move the multimodal schema additions (document modality, byte_size, StrippedPart, StrippedReason) into docs/gen-ai/non-normative/models.ipynb and regenerate the JSON schemas from the notebook so they are no longer hand-edited (per @alexmojaki). - Add a worked multimodal example to docs/gen-ai/non-normative/examples-llm-calls.md demonstrating a document URI part with byte_size and a stripped video part with stripped_reason=size_exceeded (per @alexmojaki). Note: regenerating from the notebook with current pydantic surfaces some emitter-style differences vs. the previous hand-edited JSON (single-element $ref vs oneOf, additionalProperties:true on BlobPart, key ordering inside anyOf branches). These are not introduced by this change set but become visible once the notebook becomes the source of truth.
ae86b3f to
5bdf6e6
Compare
Address review feedback on open-telemetry#3673: - Drop the new `StrippedPart` type and the `StrippedReason` enum. Instead, make `content` / `file_id` / `uri` optional on BlobPart / FilePart / UriPart and add an optional free-form `stripped_reason` field on each. This keeps the original part type (and therefore "what kind of part was sent to the model") intact while still allowing fail-closed observability when the instrumentation intentionally does not capture bytes / id / URI (per @alexmojaki). - Make `stripped_reason` a free-form short string instead of an enum, so instrumentations can describe their own capture-skipping conditions ("size_exceeded", "modality_not_allowed", etc.) without needing a spec change. Drops the previously-introduced enum values that referenced implementation-specific concepts ("redactor", "store") (per @alexmojaki). - Harmonize the `byte_size` description across BlobPart / FilePart / UriPart to a single shared phrasing (per @alexmojaki). - Update examples-llm-calls.md to demonstrate a `document` URI part with `byte_size` and a `blob` part stripped via `stripped_reason: size_exceeded`. - Update chloggen entry to reflect the narrower scope.
| " byte_size: Optional[int] = Field(default=None, ge=0, description=_BYTE_SIZE_DESC)\n", | ||
| " stripped_reason: Optional[str] = Field(default=None, description=_STRIPPED_REASON_DESC)\n", |
There was a problem hiding this comment.
it feels like we've reached the point where we need a common base class for the 3 media parts
There was a problem hiding this comment.
Done in f627524d — extracted MediaPart as a common base for BlobPart/FilePart/UriPart holding mime_type, modality, byte_size, and stripped_reason. Each part now contributes only its type literal and its own content-bearing field (content / file_id / uri).
Regen findings, flagging proactively: pydantic flattens MediaPart cleanly — no MediaPart entry appears in $defs, and no allOf/$ref indirection is introduced in the three child schemas. The only wire-format diff is property ordering within each child: inherited fields (mime_type, modality, byte_size, stripped_reason) now appear before the subclass-specific fields (type, content/file_id/uri), per pydantic's MRO. Happy to override field ordering if you'd rather keep type first in the JSON.
Address review feedback to introduce a common base class for the three media-attachment parts. MediaPart holds the fields shared across BlobPart, FilePart, and UriPart (mime_type, modality, byte_size, stripped_reason). Each subclass now contributes only its 'type' literal and its own content-bearing field (content / file_id / uri). Notebook is shorter and the family relationship between the three parts is explicit. Regenerated JSON schemas are substantively identical to the previous version: no allOf/$ref indirection is introduced (pydantic flattens MediaPart, no MediaPart entry appears in $defs), and the only diff is property ordering within each subclass (inherited fields appear before subclass-specific fields, per pydantic's MRO).
|
Hello @Mandark-droid! With the merging of Move GenAI semantic conventions to its own dedicated repository, the Our ask is that you move this PR to the new repo here: https://github.com/open-telemetry/semantic-conventions-genai The contribution guide for the new repo can be found here: https://github.com/open-telemetry/semantic-conventions-genai/blob/main/CONTRIBUTING.md. A major difference is making sure there's a representative scenario and reference outlined in:
You may also find that you need to additional updates to you proposal as the new repo use the V2 Schema. If you would like to discuss or have questions you can reply here, post in slack here, or add an agenda item for the Gen AI SIG. Thank you so much! |
lmolkova
moved this from Awaiting codeowners approval
to Blocked
in Semantic Conventions Triage
|
Closing here, but looking forward to seeing this in the new repo! |
|
Thanks @wolfgangcodes for the redirect and @trask for closing — refiled in the new repo. Per the new
All three carry forward the design pivots from the review on this PR (notebook is source of truth, harmonized One open question I've flagged on each new PR: there are currently no multimodal scenarios under |
* gen-ai: add document modality to multimodal content parts Adds the 'document' value to the Modality enum in the gen-ai input, output, and system-instructions message JSON schemas. Today PDF/DOCX and similar parts have to fall through to the free-form string branch of the modality anyOf; this change gives them a first-class enum value matching the existing image/video/audio entries. Pure addition: no removals, no changes to required fields. The notebook source under docs/gen-ai/non-normative/models.ipynb has been updated to keep the regen-from-notebook flow consistent, and a worked example was added to examples-llm-calls.md. Follow-up from open-telemetry/semantic-conventions#3673 (closed and refiled here per the new repo home for gen-ai semantic conventions). * gen-ai: add document-modality reference scenario to openai Per @trask's direction on PR #143, the multimodal scenario lives in #142 so it can carry the document modality first, then be extended in #143 (byte_size) and #144 (stripped_reason) once this lands. Adds run_chat_with_document_input_reference(client) to the openai scenario, exercising the new 'document' enum value on a UriPart in gen_ai.input.messages. The OpenAI SDK call uses the documented file content block; the canonical OTel attribute carries a TextPart plus a document-modality UriPart, mirroring how a KYC document-extraction workload surfaces in the wire shape. * gen-ai: apply ruff format to multimodal scenario block * gen-ai: address scenario-evaluation feedback on PR #142 @trask's reference-scenario SKILL evaluation flagged that the previous openai multimodal block fabricated a URI and mime_type that aren't on the chat.completions.create boundary -- file_id is opaque, neither yields a public URI or mime type without a separate Files-API roundtrip. Two fixes per the evaluation: 1. Rewrite the openai scenario to use the inline file_data shape ({'type': 'file', 'file': {'file_data': 'data:application/pdf;base64,...', 'filename': '...'}}) and emit a BlobPart instead of a UriPart. Every emitted BlobPart field now derives directly from the SDK arg: - mime_type from the data-URI prefix - content from the base64 portion - modality 'document' from the classification of application/pdf No fabricated URIs. 2. Add run_chat_with_document_input() to the anthropic scenario. Anthropic's Messages API has a first-class document content block ({'type': 'document', 'source': {'type': 'base64', 'media_type': '...', 'data': '...'}}), so the BlobPart emission is directly observable from the SDK call boundary with no derivation gymnastics. Both scenarios run cleanly against the local mock server; weaver live-check reports no after_resolution policy violations on either. The bedrock-converse demonstration trask also suggested is deferred to a follow-up (DocumentBlock has the same shape; happy to add if needed). Addresses #142 (comment)... * gen-ai: add Bedrock Converse DocumentBlock reference + ruff import-sort fix Completes @trask's three-scenario request for PR #142: - openai: inline file_data + BlobPart (previous commit, derives mime/content/modality from SDK arg) - anthropic: native document content block + BlobPart (previous commit) - aws-bedrock: native Converse DocumentBlock + BlobPart (this commit) -- the DocumentBlock 'format' / 'source.bytes' fields map 1:1 to the BlobPart mime_type/content with zero out-of-band derivation. Most direct of the three. Also applies the ruff isort fix from this run (I001 in the anthropic scenario file: import base64 should sort after import anthropic in the new function -- ruff check --fix moved it). All three scenarios validated locally against the mock LLM server; weaver live-check reports no after_resolution policy violation on any of them. ruff format + ruff check both clean. * gen-ai: drop incorrect filename claim from anthropic doc-input docstring Round-3 Copilot review on PR #142: the docstring claimed Anthropic's document content block exposes 'mime type, source bytes, and filename' on the SDK boundary, but the scenario does not include a filename field in its payload. Removing the filename claim keeps the docstring accurate to the scenario; if/when a filename field is added to the schema in a follow-up (per issue #50), the docstring can be updated alongside.
5 participants
Summary
Narrow additions to
gen-ai-input-messages.jsonandgen-ai-output-messages.jsonto better support multimodal LLM workloads:documentto theModalityenum — needed for PDF / DOCX workloads (BFSI KYC extraction is a high-volume real example).byte_sizefield toBlobPart,FilePart, andUriPart— useful for cost-of-capture telemetry and storage planning. Optional, additive.StrippedParttype for fail-closed observability. Records that the instrumentation detected a content part but intentionally did not capture its bytes (size cap exceeded, modality disallowed by configuration, redactor failed, store unavailable, no_store_configured). Without this, consumers cannot distinguish "no media in this turn" from "media was deliberately stripped" — which matters for compliance auditing and for diagnosing capture-pipeline regressions.Why
Discussed in #3672. Original framing on that issue overstated the gap; on closer reading the existing schemas already define
BlobPart,FilePart,UriPartand aModalityenum (image,video,audio). The substantive remaining gap is the three pieces above. The original issue body and follow-up comments now reflect this corrected scope.Backwards compatibility
Purely additive — no removals, no changes to required fields on existing types. Consumers that don't recognise
StrippedPartwill fall through to theGenericPartmatcher inparts.items.anyOf. Consumers that don't recognisedocumentin theModalityenum can fall through to thestringbranch of theanyOfalready permitted.Reference implementation
genai-otel-instrumentv1.1.1 emits the canonical JSON shape viagen_ai.input.messages/gen_ai.output.messageswhenOTEL_SEMCONV_STABILITY_OPT_IN=gen_ai, including:BlobPartfor inline base64 bytesUriPartwithbyte_sizefor offloaded objects (S3 / MinIO)StrippedPartwithstripped_reasonfor fail-closed casesEnd-to-end validated through OTel collector → OpenSearch with bytes resolvable from MinIO via the
urifield.Files
docs/gen-ai/gen-ai-input-messages.json—Modalityenum, three part types gainbyte_size, newStrippedPart,ChatMessage.parts.items.anyOfacceptsStrippedPartdocs/gen-ai/gen-ai-output-messages.json— same changes mirrored onOutputMessage.parts.chloggen/gen-ai-multimodal-narrow.yaml— changelog entryOpen questions / scope notes
make checklocally; happy to iterate on whatever the CI surfaces.data_urlenum value formedia_sourcediscussed in the issue is not included here because the existingBlobPart(with inline base64content) already covers that case — keeping the diff minimal.ToolCallResponsePart.Resolves open-telemetry/semantic-conventions-genai#304 (reviewers: feel free to keep the issue open if you'd like it to remain the umbrella discussion).