gen-ai: make input-messages BlobPart content optional and add stripped_reason#144
Conversation
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
Makes multimodal part payload fields nullable/optional and adds a stripped_reason marker so instrumentations can indicate observed-but-not-captured media while preserving the original part type/modality.
Changes:
- Make
BlobPart.content,FilePart.file_id, andUriPart.urioptional/nullable in models and regenerated JSON schemas. - Add optional free-form
stripped_reasonto each part type. - Update non-normative examples and changelog to document the stripped-media pattern.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| docs/gen-ai/non-normative/models.ipynb | Updates non-normative Pydantic models: optional payload fields + new stripped_reason. |
| docs/gen-ai/non-normative/examples-llm-calls.md | Adds an example of a stripped video blob part. |
| docs/gen-ai/gen-ai-system-instructions.json | Regenerated schema: payload fields nullable + stripped_reason added, payload removed from required. |
| docs/gen-ai/gen-ai-output-messages.json | Regenerated schema updates for output message parts (same pattern). |
| docs/gen-ai/gen-ai-input-messages.json | Regenerated schema updates for input message parts (same pattern). |
| CHANGELOG.md | Documents the enhancement under Unreleased. |
Comments suppressed due to low confidence (3)
docs/gen-ai/non-normative/models.ipynb:1
- The
stripped_reasondescription says “did not capture its payload bytes” even forFilePartandUriPart, where the omitted payload is an identifier/URI rather than bytes. Consider rewording to something type-agnostic (e.g., “did not capture the content-bearing field”) or tailoring the wording per part type so it remains accurate.
{
docs/gen-ai/non-normative/models.ipynb:1
- The
stripped_reasondescription says “did not capture its payload bytes” even forFilePartandUriPart, where the omitted payload is an identifier/URI rather than bytes. Consider rewording to something type-agnostic (e.g., “did not capture the content-bearing field”) or tailoring the wording per part type so it remains accurate.
{
docs/gen-ai/non-normative/models.ipynb:1
- The
stripped_reasondescription says “did not capture its payload bytes” even forFilePartandUriPart, where the omitted payload is an identifier/URI rather than bytes. Consider rewording to something type-agnostic (e.g., “did not capture the content-bearing field”) or tailoring the wording per part type so it remains accurate.
{
Adds an anyOf constraint to BlobPart, FilePart, and UriPart so any instance must carry either the content-bearing field (`content` / `file_id` / `uri`) or `stripped_reason` -- or both. Without this constraint, the schema permitted structurally empty parts (everything nullable), which would defeat the 'no media in this turn' vs 'media was deliberately stripped' distinction the previous commit set up. Implementation: declared via `Config.json_schema_extra` on each of the three part classes in models.ipynb so the notebook remains the source of truth, and applied surgically to the three JSON schemas under docs/gen-ai/ in the same shape Pydantic emits (anyOf at the top level of each part def, alongside additionalProperties / description / properties / required / title / type). BlobPart did not previously carry a `class Config` block; one is added here purely to hold the json_schema_extra. No change to extra-fields behaviour on BlobPart -- still implicit Pydantic default. Addresses Copilot review on PR open-telemetry#144.
|
Thanks @Copilot for the review — both items addressed in a9107c4:
|
| " )\n", | ||
| " file_id: str = Field(description=\"An identifier referencing a file that was pre-uploaded to the provider.\")\n", | ||
| " file_id: Optional[str] = Field(default=None, description=\"An identifier referencing a file that was pre-uploaded to the provider. MAY be omitted when `stripped_reason` is set.\")\n", | ||
| " stripped_reason: Optional[str] = Field(default=None, description=\"If set, indicates that the instrumentation observed a content part of this type but intentionally did not capture its payload bytes. The value is a short free-form string describing why capture was skipped (examples: `size_exceeded`, `modality_not_allowed`, `redactor_error`, `upload_error`, `no_store_configured`). When `stripped_reason` is set the content-bearing field (`content` / `file_id` / `uri`) MAY be omitted.\")\n", |
| " \"MAY be omitted when `stripped_reason` is set.\"\n", | ||
| " )\n", | ||
| " )\n", | ||
| " stripped_reason: Optional[str] = Field(default=None, description=\"If set, indicates that the instrumentation observed a content part of this type but intentionally did not capture its payload bytes. The value is a short free-form string describing why capture was skipped (examples: `size_exceeded`, `modality_not_allowed`, `redactor_error`, `upload_error`, `no_store_configured`). When `stripped_reason` is set the content-bearing field (`content` / `file_id` / `uri`) MAY be omitted.\")\n", |
| " class Config:\n", | ||
| " # An instance must carry either the payload (`content`) or `stripped_reason` \u2014 both\n", | ||
| " # absent would defeat the 'no media' vs 'media stripped' distinction.\n", | ||
| " json_schema_extra = {\n", | ||
| " \"anyOf\": [\n", | ||
| " {\"properties\": {\"content\": {\"not\": {\"type\": \"null\"}}}, \"required\": [\"content\"]},\n", | ||
| " {\"properties\": {\"stripped_reason\": {\"not\": {\"type\": \"null\"}}}, \"required\": [\"stripped_reason\"]},\n", | ||
| " ]\n", | ||
| " }\n", |
|
Thanks @Copilot for the second pass — good catch. The first-pass Fixed in b00a96f. Each branch now also constrains the corresponding property to non-null: "anyOf": [
{
"properties": {"content": {"not": {"type": "null"}}},
"required": ["content"]
},
{
"properties": {"stripped_reason": {"not": {"type": "null"}}},
"required": ["stripped_reason"]
}
]So an instance must now carry either a non-null payload ( Functional check with
|
Round-3 Copilot review on PR open-telemetry#144: - The previous stripped_reason description said the instrumentation 'did not capture its payload bytes', but only BlobPart's payload is bytes. FilePart carries a file_id (provider identifier, not bytes) and UriPart carries a URI string (reference, not bytes). Each part now has its own focused stripped_reason wording. - Reworded the Config block code comments to clarify the anyOf constraint is a JSON-schema-level guarantee enforced by validators on the consumer side via json_schema_extra. The Pydantic *runtime* model still allows both fields to be None (no model_validator added; these models are documentation, not production validators). The constraint statement is correct on the wire, not at the Python layer -- the comment now says so.
|
Thanks @Copilot for the third pass. Addressed in 107f1be: 1 & 2.
Each description also tailors the example reason list — e.g. 3. Notebook code comment vs Pydantic runtime. Fair — the old comment said the model "must carry" one or the other, but class Config:
# JSON-schema-level constraint: serialized instances must carry either a non-null
# `content` or a non-null `stripped_reason` (or both). The Pydantic *runtime* model
# still allows both fields to be None; the constraint is exported via
# `json_schema_extra` and enforced by JSON schema validators on the consumer side.
json_schema_extra = {
"anyOf": [
{"properties": {"content": {"not": {"type": "null"}}}, "required": ["content"]},
{"properties": {"stripped_reason": {"not": {"type": "null"}}}, "required": ["stripped_reason"]},
]
}Same clarification applied to FilePart and UriPart's Config blocks. Happy to add a PR body has also been filled out against the new PR template (per @trask's request on #143). |
Per @trask's direction on PR open-telemetry#143, the multimodal scenario lives in open-telemetry#142 so it can carry the document modality first, then be extended in open-telemetry#143 (byte_size) and open-telemetry#144 (stripped_reason) once this lands. Adds run_chat_with_document_input_reference(client) to the openai scenario, exercising the new 'document' enum value on a UriPart in gen_ai.input.messages. The OpenAI SDK call uses the documented file content block; the canonical OTel attribute carries a TextPart plus a document-modality UriPart, mirroring how a KYC document-extraction workload surfaces in the wire shape.
* gen-ai: add document modality to multimodal content parts Adds the 'document' value to the Modality enum in the gen-ai input, output, and system-instructions message JSON schemas. Today PDF/DOCX and similar parts have to fall through to the free-form string branch of the modality anyOf; this change gives them a first-class enum value matching the existing image/video/audio entries. Pure addition: no removals, no changes to required fields. The notebook source under docs/gen-ai/non-normative/models.ipynb has been updated to keep the regen-from-notebook flow consistent, and a worked example was added to examples-llm-calls.md. Follow-up from open-telemetry/semantic-conventions#3673 (closed and refiled here per the new repo home for gen-ai semantic conventions). * gen-ai: add document-modality reference scenario to openai Per @trask's direction on PR #143, the multimodal scenario lives in #142 so it can carry the document modality first, then be extended in #143 (byte_size) and #144 (stripped_reason) once this lands. Adds run_chat_with_document_input_reference(client) to the openai scenario, exercising the new 'document' enum value on a UriPart in gen_ai.input.messages. The OpenAI SDK call uses the documented file content block; the canonical OTel attribute carries a TextPart plus a document-modality UriPart, mirroring how a KYC document-extraction workload surfaces in the wire shape. * gen-ai: apply ruff format to multimodal scenario block * gen-ai: address scenario-evaluation feedback on PR #142 @trask's reference-scenario SKILL evaluation flagged that the previous openai multimodal block fabricated a URI and mime_type that aren't on the chat.completions.create boundary -- file_id is opaque, neither yields a public URI or mime type without a separate Files-API roundtrip. Two fixes per the evaluation: 1. Rewrite the openai scenario to use the inline file_data shape ({'type': 'file', 'file': {'file_data': 'data:application/pdf;base64,...', 'filename': '...'}}) and emit a BlobPart instead of a UriPart. Every emitted BlobPart field now derives directly from the SDK arg: - mime_type from the data-URI prefix - content from the base64 portion - modality 'document' from the classification of application/pdf No fabricated URIs. 2. Add run_chat_with_document_input() to the anthropic scenario. Anthropic's Messages API has a first-class document content block ({'type': 'document', 'source': {'type': 'base64', 'media_type': '...', 'data': '...'}}), so the BlobPart emission is directly observable from the SDK call boundary with no derivation gymnastics. Both scenarios run cleanly against the local mock server; weaver live-check reports no after_resolution policy violations on either. The bedrock-converse demonstration trask also suggested is deferred to a follow-up (DocumentBlock has the same shape; happy to add if needed). Addresses #142 (comment)... * gen-ai: add Bedrock Converse DocumentBlock reference + ruff import-sort fix Completes @trask's three-scenario request for PR #142: - openai: inline file_data + BlobPart (previous commit, derives mime/content/modality from SDK arg) - anthropic: native document content block + BlobPart (previous commit) - aws-bedrock: native Converse DocumentBlock + BlobPart (this commit) -- the DocumentBlock 'format' / 'source.bytes' fields map 1:1 to the BlobPart mime_type/content with zero out-of-band derivation. Most direct of the three. Also applies the ruff isort fix from this run (I001 in the anthropic scenario file: import base64 should sort after import anthropic in the new function -- ruff check --fix moved it). All three scenarios validated locally against the mock LLM server; weaver live-check reports no after_resolution policy violation on any of them. ruff format + ruff check both clean. * gen-ai: drop incorrect filename claim from anthropic doc-input docstring Round-3 Copilot review on PR #142: the docstring claimed Anthropic's document content block exposes 'mime type, source bytes, and filename' on the SDK boundary, but the scenario does not include a filename field in its payload. Removing the filename claim keeps the docstring accurate to the scenario; if/when a filename field is added to the schema in a follow-up (per issue #50), the docstring can be updated alongside.
| "video", | ||
| "audio" | ||
| "audio", | ||
| "document" |
There was a problem hiding this comment.
would you mind splitting this piece out into a separate PR?
and then reducing this PR to just cover input messages BlobPart?
You can mention in the PR description that if accepted, a follow-up PR would add replicate the change across XYZ
I think that would make it easier to get reviewers engaged on this. Thanks!
There was a problem hiding this comment.
Done. Force-pushed a rebase onto main (the merged #142 commits drop out) and reduced this PR to just BlobPart on gen_ai.input.messages: content becomes optional, stripped_reason is added, and a top-level anyOf requires one or the other. The diff is now 4 files (schema + the reference model in models.ipynb + one example + CHANGELOG).
FilePart/UriPart and the output-messages / system-instructions schemas are pulled out — I've written the follow-up plan into the PR description so the sequencing is on record. The FilePart/UriPart capture-feasibility question is the same one on #143; I answered it there.
5be8b02 to
ea3a174
Compare
lmolkova
left a comment
There was a problem hiding this comment.
I like where it's going, but Blob part is just one of the parts that has large content problem.
I'd prefer if we
- documented a configuration option that limits size of any content in any part
- if size is exceeded we wouldn't capture it; or, for text, we would rather trim to the size
- we could let users know that content was there by including its size on each affected part (especially blob) as you're proposing in the #143, but also on the text part
- we could potentially mimic OTLP 'dropped attributes' property and record something like that on a generic part (e.g. content_dropped: true) if that's necessary (presence of non-0 size would be an indication on its own)
Having a reason seems interesting, but it's not a common approach in OTel - if something is dropped, we don't record why, it could introduce interesting problems on its own.
| @@ -156,7 +156,20 @@ | |||
| " modality: Union[Modality, str] = Field(\n", | |||
There was a problem hiding this comment.
this file was converted to python in #226, sorry for the inconvenience!
There was a problem hiding this comment.
Ported in the rebase, thanks — the BlobPart change now lives in docs/gen-ai/non-normative/models.py (input-messages variant classes) and gen-ai-input-messages.json is regenerated via make generate-json-schemas.
|
Hi! As part of #275, this repository switched to Towncrier changelog fragments to reduce merge conflicts in Please move this PR's changelog entry out of Create Make `BlobPart.content` optional and add a free-form `stripped_reason` marker on the `gen_ai.input.messages` `BlobPart`, so an instrumentation can record that it observed an inline blob but intentionally did not capture the payload bytes (size cap exceeded, modality not allowed, redactor failure, etc.) while preserving the original part type and modality. A schema-level `anyOf` requires either a non-null `content` or a non-null `stripped_reason`. Scoped to input-messages `BlobPart`; if accepted, follow-up PRs replicate the change to `FilePart`/`UriPart` and to the output-messages and system-instructions schemas.After adding the fragment, please remove this PR's direct edit to Thanks! |
…ssages Make BlobPart.content optional and add a free-form stripped_reason marker on the gen_ai.input.messages BlobPart, so an instrumentation can record that it observed an inline blob but intentionally did not capture the payload bytes (size cap exceeded, modality not allowed, redactor failure, etc.) while preserving the original part type and modality. A schema-level anyOf requires either a non-null content or a non-null stripped_reason. Scoped to input-messages BlobPart per review feedback; if accepted, follow-up PRs replicate the change to FilePart/UriPart and to the output-messages and system-instructions schemas. Rebased onto main after the Towncrier migration (open-telemetry#275) and the models.ipynb -> models.py conversion (open-telemetry#226). The input-only scoping is expressed via input-messages variants of BlobPart/ChatMessage in models.py that reuse the shared class names, so the generated $defs keys and wire format stay identical; JSON schemas are regenerated via make generate-json-schemas. The changelog entry moved from CHANGELOG.md to changelog.d/144.enhancement.md.
ea3a174 to
6244073
Compare
|
@trask done — the changelog entry now lives in The rebase onto current If maintainers would rather not carry the input-only variant classes in |
|
Thanks @lmolkova — I think we agree on most of this, and the pieces you list map onto the current PR stack:
On "presence of non-0 size would be an indication on its own": If the WG preference lands on size-limit-config + |
Pull request dashboard status
|
Description
Makes
BlobPart.contentoptional and adds an optional free-formstripped_reasonfield on thegen_ai.input.messagesBlobPart. This enables fail-closed observability for inline blobs: an instrumentation that detected a blob but intentionally did not capture its payload bytes can record that fact while preserving the original parttypeandmodality.{ "type": "blob", "modality": "video", "mime_type": "video/mp4", "stripped_reason": "size_exceeded" }A top-level
anyOfonBlobPartenforces that an instance must carry either a non-nullcontentor a non-nullstripped_reason— both absent (or bothnull) is rejected, so structurally-empty parts cannot validate.Scope
This PR is intentionally narrowed to
BlobPartongen_ai.input.messagesonly, per @trask's review request. If the shape is accepted, follow-up PRs replicate it to:FilePart(file_idoptional +stripped_reason) andUriPart(urioptional +stripped_reason) on input messages — thestripped_reasonwording there refers to the file reference / external URI rather than payload bytes; see the discussion on the capture feasibility of those parts in gen-ai: add optional byte_size to multimodal content parts #143.gen_ai.output.messagesandgen_ai.system_instructionsschemas, which reuse the same part definitions.Follow-up from the closed PR open-telemetry/semantic-conventions#3673. Sister PRs:
documentmodality #142 (merged) andbyte_size#143. All independent and additive.Motivation
User journey: compliance auditing and capture-pipeline diagnostics for content-capturing instrumentations.
Today the gen-ai message schemas have no way to record a part the instrumentation observed but intentionally did not capture. The instrumentation can either:
content). This works only when the part is captureable — under the configured size cap, permitted by the configured modality allow-list, approved by the configured redactor (if any), and the configured offload-to-store path succeeded.In case (2) the downstream consumer sees
parts: [...]with no media — indistinguishable from a turn that simply had none. Two consumer use cases break:stripped_reason: "upload_error"an SRE can dashboard-count strippeds per modality / per reason and page on a step-change.Prior art:
InvokeModelrequest logs annotate "guardrail intervened" + reason rather than dropping content silently._comment+contentshape where an inline body can be a placeholder noting "body too large; not captured" while preservingmimeTypeandsize.genai-otel-instrumentemits this stripped-blob marker on size-cap-exceeded media.Prototype
This PR ships the schema change in
docs/gen-ai/gen-ai-input-messages.jsonand the matching reference model indocs/gen-ai/non-normative/models.ipynb. As with #143, the existingreference/scenarios/<library>/scenarios are text-only — there is no existing scenario into which to emit a stripped blob as-is, and a new multimodal scenario is its own chunk of work I'd prefer to scope under a follow-up so it can carry #143'sbyte_sizeand this PR'sstripped_reasontogether.Functional schema check: jsonschema validation against the schema in this PR confirms the
anyOfrejects structurally-empty parts —{"type":"blob","modality":"image","content":null}(no stripped_reason)Nonenot valid under{"type":"null"}— both anyOf branches fail{"type":"blob","modality":"image"}(neither){"type":"blob","modality":"image","stripped_reason":"size_exceeded"}{"type":"blob","modality":"image","content":"aGVsbG8="}End-to-end producer verification: the
genai-otel-instrumentlibrary emits this stripped-blob shape on its OpenAI / Anthropic / Bedrock instrumentors when a configured size cap is exceeded, validated against MinIO + OTel collector + OpenSearch. The library already ships the shape proposed here (the original separateStrippedParttype was dropped in v1.3.0 in favour of extending the existing part with a free-formstripped_reasonstring), so the producer side is aligned with this PR rather than pending on it.Checklist
UnreleasedinCHANGELOG.md