gen-ai: add document modality to multimodal content parts by Mandark-droid · Pull Request #142 · open-telemetry/semantic-conventions-genai

Mandark-droid · 2026-05-11T16:14:25Z

Description

Adds document as a value of the Modality enum on BlobPart, FilePart, and UriPart in the gen-ai input, output, and system-instructions message JSON schemas. PDFs, DOCX, and other office-document payloads currently fall through to the free-form string branch of the modality anyOf; this change gives them a first-class enum value alongside image / video / audio.

Concrete shape on the wire:

{
  "type": "blob",
  "modality": "document",
  "mime_type": "application/pdf",
  "content": "<base64 of raw PDF bytes>"
}

The Pydantic source model in docs/gen-ai/non-normative/models.ipynb is the source of truth; the three JSON schemas are regenerated from it.

Follow-up from the closed open-telemetry/semantic-conventions#3673. Closes (or significantly advances) #50. Sister PRs in this repo: byte_size #143 and stripped_reason #144. The three are independent and additive.

Motivation

User journey: content auditability and cost-attribution for document-processing GenAI workloads.

A high-volume real workload that drove this: BFSI KYC (Know Your Customer) extraction — a customer uploads a passport or utility bill as a PDF, the GenAI app extracts identity fields, and the instrumentation needs to record what kind of content was sent. Today the only way to convey "this was a PDF document, not an image" is either (a) leak the mime type alone and ask consumers to classify, or (b) fall through to the free-form string branch of modality, fragmenting how producers describe documents.

Both options block:

Cost attribution by content type — operators want per-tenant per-modality token-usage rollups; document is a distinct class with different per-token cost characteristics (PDFs are commonly OCR'd or embedded by the model, which has cost implications different from raw images).
Compliance dashboards that need to answer "how many document-class payloads were captured this quarter?" — a stable, normative enum value makes this a single-attribute query rather than a mime-type classification expression in every consumer.

Prior art:

Anthropic's messages API has a first-class document content-block type today.
OpenAI Chat Completions consumes PDFs via a file content block with inline file_data (data URI) or file_id.
AWS Bedrock Converse has a native DocumentBlock (format + source.bytes + name).
The reference implementation in genai-otel-instrument (v1.1.1) already emits modality: "document" on PDFs captured by its OpenAI / Anthropic / Bedrock instrumentors, validated end-to-end through MinIO + OTel collector + OpenSearch on KYC extraction workloads.

Issue #50 enumerates three follow-ups that this PR partially addresses (the modality enum) and that the broader split (byte_size in #143, optional filename as a future PR) will round out.

Prototype

Three matching reference scenarios were added (per @trask's evaluation on the first iteration: the emitted fields must derive from the SDK call boundary, no fabricated URIs or out-of-band lookups):

Scenario	SDK boundary	BlobPart derivation
`reference/scenarios/openai/run_chat_with_document_input_reference`	`{"type":"file", "file": {"file_data": "data:application/pdf;base64,...", "filename": "..."}}`	mime_type from data-URI prefix; content from base64 portion; modality from mime classification
`reference/scenarios/anthropic/run_chat_with_document_input`	`{"type":"document", "source": {"type":"base64", "media_type":"application/pdf", "data":"..."}}`	mime_type ← `source.media_type`; content ← `source.data`; modality from mime classification
`reference/scenarios/aws-bedrock/run_converse_with_document_input_reference`	`{"document": {"format":"pdf", "name":"...", "source": {"bytes": <raw>}}}`	mime_type from `format` (`pdf` → `application/pdf`); content = base64 of `source.bytes`; modality from mime classification

All three are runnable end-to-end against the local mock LLM server (uv run run-scenario <library>); weaver live-check reports ✔ No after_resolution policy violation on each. Every emitted BlobPart field on gen_ai.input.messages is derivable from a value present on a single synchronous client.X.create(...) call — no Files-API roundtrip required, no application-side URI fabrication.

A worked example was also added to docs/gen-ai/non-normative/examples-llm-calls.md under the existing Multimodal-inputs block.

make check-policies and make generate-all both run clean against this branch on a local Docker host (Weaver v0.23.0); generate-all produces zero working-tree drift.

Checklist

Motivation section filled in above
Reference scenarios updated for affected libraries
Changelog entry added under Unreleased in CHANGELOG.md

Adds the 'document' value to the Modality enum in the gen-ai input, output, and system-instructions message JSON schemas. Today PDF/DOCX and similar parts have to fall through to the free-form string branch of the modality anyOf; this change gives them a first-class enum value matching the existing image/video/audio entries. Pure addition: no removals, no changes to required fields. The notebook source under docs/gen-ai/non-normative/models.ipynb has been updated to keep the regen-from-notebook flow consistent, and a worked example was added to examples-llm-calls.md. Follow-up from open-telemetry/semantic-conventions#3673 (closed and refiled here per the new repo home for gen-ai semantic conventions).

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Copilot

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

trask · 2026-05-11T20:11:06Z

@Mandark-droid can you fill out the new PR template? thanks! https://raw.githubusercontent.com/open-telemetry/semantic-conventions-genai/refs/heads/main/.github/PULL_REQUEST_TEMPLATE.md

@trask

Per @trask's direction on PR open-telemetry#143, the multimodal scenario lives in open-telemetry#142 so it can carry the document modality first, then be extended in open-telemetry#143 (byte_size) and open-telemetry#144 (stripped_reason) once this lands. Adds run_chat_with_document_input_reference(client) to the openai scenario, exercising the new 'document' enum value on a UriPart in gen_ai.input.messages. The OpenAI SDK call uses the documented file content block; the canonical OTel attribute carries a TextPart plus a document-modality UriPart, mirroring how a KYC document-extraction workload surfaces in the wire shape.

@trask

@trask's reference-scenario SKILL evaluation flagged that the previous openai multimodal block fabricated a URI and mime_type that aren't on the chat.completions.create boundary -- file_id is opaque, neither yields a public URI or mime type without a separate Files-API roundtrip. Two fixes per the evaluation: 1. Rewrite the openai scenario to use the inline file_data shape ({'type': 'file', 'file': {'file_data': 'data:application/pdf;base64,...', 'filename': '...'}}) and emit a BlobPart instead of a UriPart. Every emitted BlobPart field now derives directly from the SDK arg: - mime_type from the data-URI prefix - content from the base64 portion - modality 'document' from the classification of application/pdf No fabricated URIs. 2. Add run_chat_with_document_input() to the anthropic scenario. Anthropic's Messages API has a first-class document content block ({'type': 'document', 'source': {'type': 'base64', 'media_type': '...', 'data': '...'}}), so the BlobPart emission is directly observable from the SDK call boundary with no derivation gymnastics. Both scenarios run cleanly against the local mock server; weaver live-check reports no after_resolution policy violations on either. The bedrock-converse demonstration trask also suggested is deferred to a follow-up (DocumentBlock has the same shape; happy to add if needed). Addresses open-telemetry#142 (comment)...

Mandark-droid · 2026-05-12T03:23:36Z

Thanks @trask for running the SKILL — the central capture-gap critique on the URI/mime fabrication was exactly right, and the inline-data path is the cleaner fix. Addressed in 5b6905c:

1. OpenAI scenario — switched to inline `file_data` + `BlobPart`

The new shape:

# SDK boundary
{"type": "file", "file": {"file_data": "data:application/pdf;base64,...", "filename": "sample-kyc.pdf"}}

# Emitted BlobPart
{"type": "blob", "modality": "document", "mime_type": "application/pdf", "content": "<base64>"}

Every emitted field traces directly to the SDK arg:

mime_type is parsed from the data-URI prefix
content is the base64 portion of the data-URI
modality: "document" is the classification of application/pdf under the new enum value — the actual thing this PR is demonstrating

No fabricated URI, no Files-API roundtrip, no cross-call. Native chat.completions.create instrumentation can produce all of this from the call boundary alone.

2. Anthropic — added a native `document` content block scenario

# SDK boundary (Anthropic Messages API)
{"type": "document",
 "source": {"type": "base64", "media_type": "application/pdf", "data": "..."}}

# Emitted BlobPart
{"type": "blob", "modality": "document", "mime_type": "application/pdf", "content": "<base64>"}

Anthropic's API exposes this most directly of all three providers — media_type and data map 1:1 to BlobPart fields with zero derivation gymnastics. Agreed this is the most credible demonstration of the new enum value, which is exactly why your PR-body cross-reference flagged it.

Both scenarios run cleanly against the local mock LLM server; weaver live-check reports ✔ No after_resolution policy violation on each. Local make check-policies continues to pass.

3. Bedrock Converse — deferred to follow-up

The DocumentBlock shape ({"document": {"format": "pdf", "name": ..., "source": {"bytes": ...}}}) is structurally identical to the Anthropic and OpenAI inline cases, so it would be repetition of the same pattern. Happy to add it if you'd like the third demonstration in this PR — or land #142 with OpenAI+Anthropic and put Bedrock in a follow-up so this PR doesn't keep growing. Slight preference for the follow-up since the Anthropic scenario already covers the "first-class document block on a provider that has one" case, but defer to your call.

(One nit on the SKILL output for future runs: the "credible reference coverage" tally still credited the previous OpenAI UriPart emission for the gen_ai.input.messages shape — that span covers a TextPart + UriPart(modality=document, mime_type, uri) combination, and the credible-coverage check would catch the fabrication on a second pass given the call-boundary analysis in the "central capture gap" section. Helpful tool — caught a real problem and articulated why it's a problem clearly.)

@trask

…rt fix Completes @trask's three-scenario request for PR open-telemetry#142: - openai: inline file_data + BlobPart (previous commit, derives mime/content/modality from SDK arg) - anthropic: native document content block + BlobPart (previous commit) - aws-bedrock: native Converse DocumentBlock + BlobPart (this commit) -- the DocumentBlock 'format' / 'source.bytes' fields map 1:1 to the BlobPart mime_type/content with zero out-of-band derivation. Most direct of the three. Also applies the ruff isort fix from this run (I001 in the anthropic scenario file: import base64 should sort after import anthropic in the new function -- ruff check --fix moved it). All three scenarios validated locally against the mock LLM server; weaver live-check reports no after_resolution policy violation on any of them. ruff format + ruff check both clean.

Mandark-droid · 2026-05-12T03:29:55Z

Done — Bedrock scenario added in 66f53f6 so the PR now covers all three native producers you flagged:

Scenario	SDK boundary	BlobPart derivation
`openai/run_chat_with_document_input_reference`	`{"type":"file", "file": {"file_data": "data:application/pdf;base64,...", "filename": "..."}}`	mime from data-URI prefix; content from base64 portion; modality from mime classification
`anthropic/run_chat_with_document_input`	`{"type":"document", "source": {"type":"base64", "media_type":"application/pdf", "data":"..."}}`	mime ← `media_type`; content ← `data` directly; modality from mime classification
`aws-bedrock/run_converse_with_document_input_reference`	`{"document": {"format":"pdf", "name":"...", "source": {"bytes": <raw>}}}`	mime from `format` classification (`pdf` → `application/pdf`); content = base64 of `source.bytes`; modality from mime classification

All three pass the SKILL's "credible reference coverage" bar — every emitted field on the BlobPart traces back to a value present on the synchronous client.X.create(...) call boundary, no Files-API roundtrips, no application-side URI fabrication.

All three scenarios run cleanly locally:

uv run run-scenario openai → ✔ No after_resolution policy violation, chat_document block executes, mock LLM responds.
uv run run-scenario anthropic → same.
uv run run-scenario aws-bedrock → same.

ruff check and ruff format are both clean after this commit (the earlier I001 import-sort error from ruff check is fixed too).

Ready for re-review.

…nce scenarios; will rebase out when open-telemetry#142 lands # Conflicts: # CHANGELOG.md

Adds byte_size to the BlobPart emitted on gen_ai.input.messages in all three multimodal reference scenarios introduced by PR open-telemetry#142 (openai, anthropic, aws-bedrock). The byte_size value is len(pdf_bytes) -- the size of the raw payload the SDK saw on the call boundary, directly derivable without any Files-API roundtrip: - openai: pdf_bytes is the source for both the data-URI on the SDK side and the BlobPart.content + byte_size on the OTel side. - anthropic: pdf_bytes is base64-encoded into the document block's source.data; byte_size is len(pdf_bytes) before encoding. - aws-bedrock: source.bytes IS the raw bytes; byte_size is len() of the same value the SDK passed. This is the byte_size companion to the document-modality demo from PR open-telemetry#142 -- exactly the pattern an instrumentation library should expose. ruff format and ruff check both clean. Currently sits on top of a merge of the open-telemetry#142 branch; once open-telemetry#142 lands this PR rebases cleanly onto main and the open-telemetry#142 commits drop out, leaving only the byte_size schema + this scenario extension.

Copilot

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.

Round-3 Copilot review on PR open-telemetry#142: the docstring claimed Anthropic's document content block exposes 'mime type, source bytes, and filename' on the SDK boundary, but the scenario does not include a filename field in its payload. Removing the filename claim keeps the docstring accurate to the scenario; if/when a filename field is added to the schema in a follow-up (per issue open-telemetry#50), the docstring can be updated alongside.

…nce scenarios; will rebase out when open-telemetry#142 lands # Conflicts: # CHANGELOG.md # docs/gen-ai/non-normative/examples-llm-calls.md

Adds a 'stripped document' multimodal block to each of the three reference scenarios introduced by PR open-telemetry#142 (openai, anthropic, aws-bedrock). The SDK call shape is identical to the captured case in each scenario -- a real document content block on the wire -- but the OTel emission demonstrates the fail-closed observability path: - type, modality, mime_type are kept (all directly observable on the SDK boundary) - content is omitted; stripped_reason = 'size_exceeded' is set This is the canonical 'instrumentation chose not to capture' shape that motivates the schema changes in this PR. The top-level anyOf constraint added to BlobPart/FilePart/UriPart ensures these parts validate (stripped_reason branch matches), while a part with neither content nor stripped_reason would still be rejected. All three scenarios run cleanly locally; ruff format + ruff check both pass. Currently sits on top of a merge of the open-telemetry#142 branch; once open-telemetry#142 lands this PR rebases cleanly onto main, the open-telemetry#142 commits drop, and only the schema changes plus this scenario extension remain.

Add an optional non-negative byte_size to BlobPart, FilePart, and UriPart in the gen_ai input/output/system-instructions message JSON schemas (and the reference models.ipynb). Lets an instrumentation record the size of a captured media payload for cost-of-capture telemetry and storage planning. Omitted when the size is not cheaply observable - e.g. a pre-existing remote URI, where the bytes would have to be downloaded to measure; set only when the bytes were observed before becoming a URI. Per-part descriptions reflect this (decoded byte length for BlobPart, file size for FilePart, payload size for UriPart). Examples updated accordingly. Rebased onto main after open-telemetry#142 merged; reference-scenario emission deferred to the follow-up that will also carry document-modality and stripped_reason.

Add 'document' modality for application/* MIME types in DeriveModalityFromMediaType(), aligning with the updated Modality enum in the upstream gen-ai input/output message schemas (open-telemetry/semantic-conventions-genai#142). Co-authored-by: Copilot <[email protected]>

Add document modality detection in OtelMessageSerializer.DeriveModalityFromMediaType to support the new 'document' modality value added to the GenAI semantic conventions (open-telemetry/semantic-conventions-genai#142). PDF, Office, and ODF file types (application/pdf, application/msword, application/vnd.openxmlformats-officedocument.*, application/vnd.ms-excel, application/vnd.ms-powerpoint, application/vnd.oasis.opendocument.*, application/rtf, text/rtf) now emit modality: 'document' in message-part serialization. Also migrate doc-comment wording in all OpenTelemetry* instrumentation client files from 'Semantic Conventions for Generative AI systems v1.41' to 'GenAI Semantic Conventions latest' per the new standalone conventions repo. Co-authored-by: Copilot <[email protected]>

- Add 'document' modality in DeriveModalityFromMediaType for PDF and Office document MIME types (application/pdf, application/msword, application/vnd.openxmlformats-officedocument.*, application/vnd.ms-*) per open-telemetry/semantic-conventions-genai#142 - Migrate doc-comment wording from 'Semantic Conventions for Generative AI systems v1.41' to 'GenAI Semantic Conventions latest' across all six OpenTelemetry* instrumentation client files - Augment AllOfficialOtelContentPartTypes_SerializedCorrectly test to assert document modality for application/pdf content Co-authored-by: Copilot <[email protected]>

Copilot AI review requested due to automatic review settings May 11, 2026 16:14

Mandark-droid requested a review from a team as a code owner May 11, 2026 16:14

Mandark-droid mentioned this pull request May 11, 2026

[gen-ai] Add multimodal additions: document modality, byte_size, StrippedPart open-telemetry/semantic-conventions#3673

Closed

Copilot AI reviewed May 11, 2026

Copilot started reviewing on behalf of Mandark-droid May 11, 2026 16:42 View session

github-actions Bot mentioned this pull request May 11, 2026

Pull Request Dashboard #102

Closed

trask requested a review from Copilot May 11, 2026 18:08

Copilot started reviewing on behalf of trask May 11, 2026 18:09 View session

Copilot AI reviewed May 11, 2026

View reviewed changes

Comment thread docs/gen-ai/non-normative/examples-llm-calls.md

MikeGoldsmith approved these changes May 11, 2026

View reviewed changes

This was referenced May 11, 2026

gen-ai: add optional byte_size to multimodal content parts #143

Open

gen-ai: make input-messages BlobPart content optional and add stripped_reason #144

Open

Mandark-droid added 2 commits May 12, 2026 08:36

gen-ai: apply ruff format to multimodal scenario block

c12e2e6

trask reviewed May 12, 2026

View reviewed changes

Comment thread reference/scenarios/openai/scenario.py

trask requested a review from Copilot May 12, 2026 03:32

Mandark-droid added a commit to Mandark-droid/semantic-conventions-genai that referenced this pull request May 12, 2026

Merge open-telemetry#142 branch to pick up document modality + refere…

9533903

…nce scenarios; will rebase out when open-telemetry#142 lands # Conflicts: # CHANGELOG.md

Copilot AI reviewed May 12, 2026

View reviewed changes

Comment thread reference/scenarios/openai/scenario.py

Comment thread reference/scenarios/anthropic/scenario.py Outdated

Mandark-droid added a commit to Mandark-droid/semantic-conventions-genai that referenced this pull request May 12, 2026

Pull anthropic docstring fix from open-telemetry#142

c49338e

Mandark-droid mentioned this pull request May 12, 2026

Support for GenAI document/pdf inputs #50

Closed

alexmojaki approved these changes May 12, 2026

View reviewed changes

trask added this pull request to the merge queue May 16, 2026

Merged via the queue into open-telemetry:main with commit c4e27eb May 16, 2026
37 checks passed

jeffhandley mentioned this pull request Jun 26, 2026

Update OpenTelemetry GenAI conventions to latest dotnet/extensions#7589

Closed

github-actions Bot mentioned this pull request Jun 27, 2026

Update OpenTelemetry GenAI conventions to latest jeffhandley/extensions#14

Closed

This was referenced Jun 28, 2026

Update OpenTelemetry GenAI conventions to latest jeffhandley/extensions#15

Closed

Update OpenTelemetry GenAI conventions to latest jeffhandley/extensions#16

Closed

This was referenced Jun 28, 2026

Update OpenTelemetry GenAI conventions to latest jeffhandley/extensions#17

Closed

Update OpenTelemetry GenAI conventions to latest jeffhandley/extensions#18

Draft

Update OpenTelemetry GenAI conventions to latest jeffhandley/extensions#19

Closed

github-actions Bot mentioned this pull request Jul 19, 2026

Update open-telemetry/semantic-conventions-genai to latest dotnet/extensions#7642

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

gen-ai: add document modality to multimodal content parts#142

gen-ai: add document modality to multimodal content parts#142
trask merged 6 commits into
open-telemetry:mainfrom
Mandark-droid:gen-ai/multimodal-document-modality

Mandark-droid commented May 11, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

trask commented May 11, 2026

Uh oh!

Uh oh!

Mandark-droid commented May 12, 2026

Uh oh!

Mandark-droid commented May 12, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

Mandark-droid commented May 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Motivation

Prototype

Checklist

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

trask commented May 11, 2026

Uh oh!

Uh oh!

Mandark-droid commented May 12, 2026

1. OpenAI scenario — switched to inline file_data + BlobPart

2. Anthropic — added a native document content block scenario

3. Bedrock Converse — deferred to follow-up

Uh oh!

Mandark-droid commented May 12, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Mandark-droid commented May 11, 2026 •

edited

Loading

1. OpenAI scenario — switched to inline `file_data` + `BlobPart`

2. Anthropic — added a native `document` content block scenario