Skip to content

gen-ai: make input-messages BlobPart content optional and add stripped_reason#144

Open
Mandark-droid wants to merge 1 commit into
open-telemetry:mainfrom
Mandark-droid:gen-ai/multimodal-stripped-reason
Open

gen-ai: make input-messages BlobPart content optional and add stripped_reason#144
Mandark-droid wants to merge 1 commit into
open-telemetry:mainfrom
Mandark-droid:gen-ai/multimodal-stripped-reason

Conversation

@Mandark-droid

@Mandark-droid Mandark-droid commented May 11, 2026

Copy link
Copy Markdown
Contributor

Description

Makes BlobPart.content optional and adds an optional free-form stripped_reason field on the gen_ai.input.messages BlobPart. This enables fail-closed observability for inline blobs: an instrumentation that detected a blob but intentionally did not capture its payload bytes can record that fact while preserving the original part type and modality.

{
  "type": "blob",
  "modality": "video",
  "mime_type": "video/mp4",
  "stripped_reason": "size_exceeded"
}

A top-level anyOf on BlobPart enforces that an instance must carry either a non-null content or a non-null stripped_reason — both absent (or both null) is rejected, so structurally-empty parts cannot validate.

Scope

This PR is intentionally narrowed to BlobPart on gen_ai.input.messages only, per @trask's review request. If the shape is accepted, follow-up PRs replicate it to:

  1. FilePart (file_id optional + stripped_reason) and UriPart (uri optional + stripped_reason) on input messages — the stripped_reason wording there refers to the file reference / external URI rather than payload bytes; see the discussion on the capture feasibility of those parts in gen-ai: add optional byte_size to multimodal content parts #143.
  2. The gen_ai.output.messages and gen_ai.system_instructions schemas, which reuse the same part definitions.

Follow-up from the closed PR open-telemetry/semantic-conventions#3673. Sister PRs: document modality #142 (merged) and byte_size #143. All independent and additive.

Motivation

User journey: compliance auditing and capture-pipeline diagnostics for content-capturing instrumentations.

Today the gen-ai message schemas have no way to record a part the instrumentation observed but intentionally did not capture. The instrumentation can either:

  1. Emit the full part with its payload (content). This works only when the part is captureable — under the configured size cap, permitted by the configured modality allow-list, approved by the configured redactor (if any), and the configured offload-to-store path succeeded.
  2. Emit nothing at all when any of the above fails.

In case (2) the downstream consumer sees parts: [...] with no media — indistinguishable from a turn that simply had none. Two consumer use cases break:

  • Compliance audits that need to prove media was not exfiltrated to a logging pipeline (because the redactor or size cap held). The auditor needs telemetry that says "the instrumentation acknowledged a video here and intentionally dropped its bytes," not silence.
  • Capture-pipeline regression detection. If a deploy starts silently dropping media (e.g. an upload-to-MinIO bucket policy broke), there is no signal to alert on — the spans look fine, the bytes just aren't there. With stripped_reason: "upload_error" an SRE can dashboard-count strippeds per modality / per reason and page on a step-change.

Prior art:

  • AWS bedrock-runtime's InvokeModel request logs annotate "guardrail intervened" + reason rather than dropping content silently.
  • HAR (HTTP Archive) format has a _comment + content shape where an inline body can be a placeholder noting "body too large; not captured" while preserving mimeType and size.
  • The reference implementation in genai-otel-instrument emits this stripped-blob marker on size-cap-exceeded media.

Prototype

This PR ships the schema change in docs/gen-ai/gen-ai-input-messages.json and the matching reference model in docs/gen-ai/non-normative/models.ipynb. As with #143, the existing reference/scenarios/<library>/ scenarios are text-only — there is no existing scenario into which to emit a stripped blob as-is, and a new multimodal scenario is its own chunk of work I'd prefer to scope under a follow-up so it can carry #143's byte_size and this PR's stripped_reason together.

Functional schema check: jsonschema validation against the schema in this PR confirms the anyOf rejects structurally-empty parts —

Instance Validates? Why
{"type":"blob","modality":"image","content":null} (no stripped_reason) None not valid under {"type":"null"} — both anyOf branches fail
{"type":"blob","modality":"image"} (neither) both branches fail
{"type":"blob","modality":"image","stripped_reason":"size_exceeded"} second branch matches
{"type":"blob","modality":"image","content":"aGVsbG8="} first branch matches

End-to-end producer verification: the genai-otel-instrument library emits this stripped-blob shape on its OpenAI / Anthropic / Bedrock instrumentors when a configured size cap is exceeded, validated against MinIO + OTel collector + OpenSearch. The library already ships the shape proposed here (the original separate StrippedPart type was dropped in v1.3.0 in favour of extending the existing part with a free-form stripped_reason string), so the producer side is aligned with this PR rather than pending on it.

Checklist

  • Motivation section filled in above
  • Reference scenarios updated for affected libraries — deferred pending maintainer guidance; see Prototype section
  • Changelog entry added under Unreleased in CHANGELOG.md

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Makes multimodal part payload fields nullable/optional and adds a stripped_reason marker so instrumentations can indicate observed-but-not-captured media while preserving the original part type/modality.

Changes:

  • Make BlobPart.content, FilePart.file_id, and UriPart.uri optional/nullable in models and regenerated JSON schemas.
  • Add optional free-form stripped_reason to each part type.
  • Update non-normative examples and changelog to document the stripped-media pattern.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
docs/gen-ai/non-normative/models.ipynb Updates non-normative Pydantic models: optional payload fields + new stripped_reason.
docs/gen-ai/non-normative/examples-llm-calls.md Adds an example of a stripped video blob part.
docs/gen-ai/gen-ai-system-instructions.json Regenerated schema: payload fields nullable + stripped_reason added, payload removed from required.
docs/gen-ai/gen-ai-output-messages.json Regenerated schema updates for output message parts (same pattern).
docs/gen-ai/gen-ai-input-messages.json Regenerated schema updates for input message parts (same pattern).
CHANGELOG.md Documents the enhancement under Unreleased.
Comments suppressed due to low confidence (3)

docs/gen-ai/non-normative/models.ipynb:1

  • The stripped_reason description says “did not capture its payload bytes” even for FilePart and UriPart, where the omitted payload is an identifier/URI rather than bytes. Consider rewording to something type-agnostic (e.g., “did not capture the content-bearing field”) or tailoring the wording per part type so it remains accurate.
{

docs/gen-ai/non-normative/models.ipynb:1

  • The stripped_reason description says “did not capture its payload bytes” even for FilePart and UriPart, where the omitted payload is an identifier/URI rather than bytes. Consider rewording to something type-agnostic (e.g., “did not capture the content-bearing field”) or tailoring the wording per part type so it remains accurate.
{

docs/gen-ai/non-normative/models.ipynb:1

  • The stripped_reason description says “did not capture its payload bytes” even for FilePart and UriPart, where the omitted payload is an identifier/URI rather than bytes. Consider rewording to something type-agnostic (e.g., “did not capture the content-bearing field”) or tailoring the wording per part type so it remains accurate.
{

Comment thread docs/gen-ai/gen-ai-input-messages.json
Comment thread docs/gen-ai/non-normative/examples-llm-calls.md
@github-actions github-actions Bot mentioned this pull request May 11, 2026
Mandark-droid added a commit to Mandark-droid/semantic-conventions-genai that referenced this pull request May 11, 2026
Adds an anyOf constraint to BlobPart, FilePart, and UriPart so any
instance must carry either the content-bearing field (`content` /
`file_id` / `uri`) or `stripped_reason` -- or both. Without this
constraint, the schema permitted structurally empty parts (everything
nullable), which would defeat the 'no media in this turn' vs 'media
was deliberately stripped' distinction the previous commit set up.

Implementation: declared via `Config.json_schema_extra` on each of
the three part classes in models.ipynb so the notebook remains the
source of truth, and applied surgically to the three JSON schemas
under docs/gen-ai/ in the same shape Pydantic emits (anyOf at the
top level of each part def, alongside additionalProperties /
description / properties / required / title / type).

BlobPart did not previously carry a `class Config` block; one is
added here purely to hold the json_schema_extra. No change to
extra-fields behaviour on BlobPart -- still implicit Pydantic default.

Addresses Copilot review on PR open-telemetry#144.
@Mandark-droid

Copy link
Copy Markdown
Contributor Author

Thanks @Copilot for the review — both items addressed in a9107c4:

  • anyOf constraint — {content|file_id|uri} OR stripped_reason required — good catch. With the payload fields removed from required it was indeed possible to construct a structurally empty part, which would defeat the whole "no media" vs "media stripped" distinction this PR introduces. Fixed by declaring json_schema_extra with the appropriate anyOf on the Config of each of BlobPart/FilePart/UriPart so the notebook stays the source of truth, and applying the matching shape to the three JSON schemas. Now a part must carry at least one of the payload-bearing field or stripped_reason (both is also fine, e.g. for a redactor that captured a hash but not the bytes).
  • json → jsonc fence for the multimodal-inputs example — this one was actually a false-positive against the head commit: the multimodal-inputs and multimodal-output blocks under ### Multimodal inputs example / ### Multimodal output example are already fenced as ```jsonc in the file (lines 224 and 289). The new stripped-part example this PR adds (around line 275 of the diff) lives inside that already-jsonc block, so the // comments are valid in the fence. No change required.

make check-policies re-validated locally on the updated branch.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 4 comments.

Comment thread docs/gen-ai/non-normative/models.ipynb Outdated
Comment thread docs/gen-ai/gen-ai-input-messages.json
Comment thread docs/gen-ai/gen-ai-output-messages.json Outdated
Comment thread docs/gen-ai/gen-ai-system-instructions.json Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.

Comment thread docs/gen-ai/non-normative/models.ipynb Outdated
" )\n",
" file_id: str = Field(description=\"An identifier referencing a file that was pre-uploaded to the provider.\")\n",
" file_id: Optional[str] = Field(default=None, description=\"An identifier referencing a file that was pre-uploaded to the provider. MAY be omitted when `stripped_reason` is set.\")\n",
" stripped_reason: Optional[str] = Field(default=None, description=\"If set, indicates that the instrumentation observed a content part of this type but intentionally did not capture its payload bytes. The value is a short free-form string describing why capture was skipped (examples: `size_exceeded`, `modality_not_allowed`, `redactor_error`, `upload_error`, `no_store_configured`). When `stripped_reason` is set the content-bearing field (`content` / `file_id` / `uri`) MAY be omitted.\")\n",
Comment thread docs/gen-ai/non-normative/models.ipynb Outdated
" \"MAY be omitted when `stripped_reason` is set.\"\n",
" )\n",
" )\n",
" stripped_reason: Optional[str] = Field(default=None, description=\"If set, indicates that the instrumentation observed a content part of this type but intentionally did not capture its payload bytes. The value is a short free-form string describing why capture was skipped (examples: `size_exceeded`, `modality_not_allowed`, `redactor_error`, `upload_error`, `no_store_configured`). When `stripped_reason` is set the content-bearing field (`content` / `file_id` / `uri`) MAY be omitted.\")\n",
Comment thread docs/gen-ai/non-normative/models.ipynb Outdated
Comment on lines +161 to +169
" class Config:\n",
" # An instance must carry either the payload (`content`) or `stripped_reason` \u2014 both\n",
" # absent would defeat the 'no media' vs 'media stripped' distinction.\n",
" json_schema_extra = {\n",
" \"anyOf\": [\n",
" {\"properties\": {\"content\": {\"not\": {\"type\": \"null\"}}}, \"required\": [\"content\"]},\n",
" {\"properties\": {\"stripped_reason\": {\"not\": {\"type\": \"null\"}}}, \"required\": [\"stripped_reason\"]},\n",
" ]\n",
" }\n",
@Mandark-droid

Copy link
Copy Markdown
Contributor Author

Thanks @Copilot for the second pass — good catch. The first-pass anyOf only checked key presence; since the payload fields are nullable, {..., "content": null} would still satisfy {required: ["content"]} and validate even without stripped_reason. Exactly the structurally-empty case the constraint was supposed to forbid.

Fixed in b00a96f. Each branch now also constrains the corresponding property to non-null:

"anyOf": [
    {
        "properties": {"content": {"not": {"type": "null"}}},
        "required": ["content"]
    },
    {
        "properties": {"stripped_reason": {"not": {"type": "null"}}},
        "required": ["stripped_reason"]
    }
]

So an instance must now carry either a non-null payload (content / file_id / uri) or a non-null stripped_reason (or both).

Functional check with jsonschema against the regenerated schema:

Instance Result
{"type": "blob", "modality": "image", "content": null} (no stripped_reason) ❌ rejected — None should not be valid under {'type': 'null'}
{"type": "blob", "modality": "image", "stripped_reason": "size_exceeded"} ✅ accepted
{"type": "blob", "modality": "image", "content": "aGVsbG8="} ✅ accepted

Config.json_schema_extra on each of BlobPart/FilePart/UriPart in models.ipynb updated to match, so the notebook stays the source of truth. make check-policies re-validated.

Mandark-droid added a commit to Mandark-droid/semantic-conventions-genai that referenced this pull request May 11, 2026
Round-3 Copilot review on PR open-telemetry#144:

- The previous stripped_reason description said the instrumentation
  'did not capture its payload bytes', but only BlobPart's payload is
  bytes. FilePart carries a file_id (provider identifier, not bytes)
  and UriPart carries a URI string (reference, not bytes). Each part
  now has its own focused stripped_reason wording.

- Reworded the Config block code comments to clarify the anyOf
  constraint is a JSON-schema-level guarantee enforced by validators
  on the consumer side via json_schema_extra. The Pydantic *runtime*
  model still allows both fields to be None (no model_validator
  added; these models are documentation, not production validators).
  The constraint statement is correct on the wire, not at the Python
  layer -- the comment now says so.
@Mandark-droid

Copy link
Copy Markdown
Contributor Author

Thanks @Copilot for the third pass. Addressed in 107f1be:

1 & 2. stripped_reason description was BlobPart-centric. Same issue as PR #143 round-2 (broadening wording leaked BlobPart-specific bytes terminology onto FilePart / UriPart). Now per-part:

Part stripped_reason description
BlobPart "did not capture its payload bytes" (unchanged — correct here)
FilePart "did not capture the file identifier" (the payload is a provider file id, not bytes)
UriPart "did not capture the URI (or did not resolve it to retrieve the underlying bytes)"

Each description also tailors the example reason list — e.g. size_exceeded doesn't really apply to a file_id capture, so it's been dropped from FilePart's example list.

3. Notebook code comment vs Pydantic runtime. Fair — the old comment said the model "must carry" one or the other, but Optional[...] with default=None means a Python instance with both content=None and stripped_reason=None constructs fine at the Pydantic layer; the constraint only kicks in when the schema is exported and a JSON schema validator runs on a serialized instance. Rather than add a @model_validator (the notebook is intended as documentation rather than a production validator), the comment is now explicit:

class Config:
    # JSON-schema-level constraint: serialized instances must carry either a non-null
    # `content` or a non-null `stripped_reason` (or both). The Pydantic *runtime* model
    # still allows both fields to be None; the constraint is exported via
    # `json_schema_extra` and enforced by JSON schema validators on the consumer side.
    json_schema_extra = {
        "anyOf": [
            {"properties": {"content": {"not": {"type": "null"}}}, "required": ["content"]},
            {"properties": {"stripped_reason": {"not": {"type": "null"}}}, "required": ["stripped_reason"]},
        ]
    }

Same clarification applied to FilePart and UriPart's Config blocks. Happy to add a @model_validator instead if maintainers prefer the runtime enforcement — let me know.

PR body has also been filled out against the new PR template (per @trask's request on #143). make check-policies re-validated locally.

Mandark-droid added a commit to Mandark-droid/semantic-conventions-genai that referenced this pull request May 12, 2026
Per @trask's direction on PR open-telemetry#143, the multimodal scenario lives in
open-telemetry#142 so it can carry the document modality first, then be extended
in open-telemetry#143 (byte_size) and open-telemetry#144 (stripped_reason) once this lands.

Adds run_chat_with_document_input_reference(client) to the openai
scenario, exercising the new 'document' enum value on a UriPart in
gen_ai.input.messages. The OpenAI SDK call uses the documented file
content block; the canonical OTel attribute carries a TextPart plus a
document-modality UriPart, mirroring how a KYC document-extraction
workload surfaces in the wire shape.
github-merge-queue Bot pushed a commit that referenced this pull request May 16, 2026
* gen-ai: add document modality to multimodal content parts

Adds the 'document' value to the Modality enum in the gen-ai input,
output, and system-instructions message JSON schemas. Today PDF/DOCX
and similar parts have to fall through to the free-form string branch
of the modality anyOf; this change gives them a first-class enum value
matching the existing image/video/audio entries.

Pure addition: no removals, no changes to required fields. The notebook
source under docs/gen-ai/non-normative/models.ipynb has been updated to
keep the regen-from-notebook flow consistent, and a worked example was
added to examples-llm-calls.md.

Follow-up from open-telemetry/semantic-conventions#3673 (closed and
refiled here per the new repo home for gen-ai semantic conventions).

* gen-ai: add document-modality reference scenario to openai

Per @trask's direction on PR #143, the multimodal scenario lives in
#142 so it can carry the document modality first, then be extended
in #143 (byte_size) and #144 (stripped_reason) once this lands.

Adds run_chat_with_document_input_reference(client) to the openai
scenario, exercising the new 'document' enum value on a UriPart in
gen_ai.input.messages. The OpenAI SDK call uses the documented file
content block; the canonical OTel attribute carries a TextPart plus a
document-modality UriPart, mirroring how a KYC document-extraction
workload surfaces in the wire shape.

* gen-ai: apply ruff format to multimodal scenario block

* gen-ai: address scenario-evaluation feedback on PR #142

@trask's reference-scenario SKILL evaluation flagged that the previous
openai multimodal block fabricated a URI and mime_type that aren't on
the chat.completions.create boundary -- file_id is opaque, neither
yields a public URI or mime type without a separate Files-API roundtrip.

Two fixes per the evaluation:

1. Rewrite the openai scenario to use the inline file_data shape
   ({'type': 'file', 'file': {'file_data': 'data:application/pdf;base64,...',
   'filename': '...'}}) and emit a BlobPart instead of a UriPart. Every
   emitted BlobPart field now derives directly from the SDK arg:
     - mime_type from the data-URI prefix
     - content from the base64 portion
     - modality 'document' from the classification of application/pdf
   No fabricated URIs.

2. Add run_chat_with_document_input() to the anthropic scenario.
   Anthropic's Messages API has a first-class document content block
   ({'type': 'document', 'source': {'type': 'base64',
   'media_type': '...', 'data': '...'}}), so the BlobPart emission is
   directly observable from the SDK call boundary with no derivation
   gymnastics.

Both scenarios run cleanly against the local mock server; weaver
live-check reports no after_resolution policy violations on either.
The bedrock-converse demonstration trask also suggested is deferred to
a follow-up (DocumentBlock has the same shape; happy to add if needed).

Addresses #142 (comment)...

* gen-ai: add Bedrock Converse DocumentBlock reference + ruff import-sort fix

Completes @trask's three-scenario request for PR #142:

- openai: inline file_data + BlobPart (previous commit, derives mime/content/modality from SDK arg)
- anthropic: native document content block + BlobPart (previous commit)
- aws-bedrock: native Converse DocumentBlock + BlobPart (this commit) -- the
  DocumentBlock 'format' / 'source.bytes' fields map 1:1 to the BlobPart
  mime_type/content with zero out-of-band derivation. Most direct of the three.

Also applies the ruff isort fix from this run (I001 in the anthropic scenario
file: import base64 should sort after import anthropic in the new function
-- ruff check --fix moved it).

All three scenarios validated locally against the mock LLM server; weaver
live-check reports no after_resolution policy violation on any of them.
ruff format + ruff check both clean.

* gen-ai: drop incorrect filename claim from anthropic doc-input docstring

Round-3 Copilot review on PR #142: the docstring claimed Anthropic's
document content block exposes 'mime type, source bytes, and filename'
on the SDK boundary, but the scenario does not include a filename
field in its payload. Removing the filename claim keeps the docstring
accurate to the scenario; if/when a filename field is added to the
schema in a follow-up (per issue #50), the docstring can be updated
alongside.
This was referenced May 26, 2026
"video",
"audio"
"audio",
"document"

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would you mind splitting this piece out into a separate PR?

and then reducing this PR to just cover input messages BlobPart?

You can mention in the PR description that if accepted, a follow-up PR would add replicate the change across XYZ

I think that would make it easier to get reviewers engaged on this. Thanks!

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. Force-pushed a rebase onto main (the merged #142 commits drop out) and reduced this PR to just BlobPart on gen_ai.input.messages: content becomes optional, stripped_reason is added, and a top-level anyOf requires one or the other. The diff is now 4 files (schema + the reference model in models.ipynb + one example + CHANGELOG).

FilePart/UriPart and the output-messages / system-instructions schemas are pulled out — I've written the follow-up plan into the PR description so the sequencing is on record. The FilePart/UriPart capture-feasibility question is the same one on #143; I answered it there.

@Mandark-droid
Mandark-droid force-pushed the gen-ai/multimodal-stripped-reason branch from 5be8b02 to ea3a174 Compare June 1, 2026 11:29
@Mandark-droid Mandark-droid changed the title gen-ai: make multimodal content optional and add stripped_reason gen-ai: make input-messages BlobPart content optional and add stripped_reason Jun 1, 2026

@lmolkova lmolkova left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like where it's going, but Blob part is just one of the parts that has large content problem.

I'd prefer if we

  • documented a configuration option that limits size of any content in any part
  • if size is exceeded we wouldn't capture it; or, for text, we would rather trim to the size
  • we could let users know that content was there by including its size on each affected part (especially blob) as you're proposing in the #143, but also on the text part
  • we could potentially mimic OTLP 'dropped attributes' property and record something like that on a generic part (e.g. content_dropped: true) if that's necessary (presence of non-0 size would be an indication on its own)

Having a reason seems interesting, but it's not a common approach in OTel - if something is dropped, we don't record why, it could introduce interesting problems on its own.

Comment thread docs/gen-ai/non-normative/models.ipynb Outdated
@@ -156,7 +156,20 @@
" modality: Union[Modality, str] = Field(\n",

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this file was converted to python in #226, sorry for the inconvenience!

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ported in the rebase, thanks — the BlobPart change now lives in docs/gen-ai/non-normative/models.py (input-messages variant classes) and gen-ai-input-messages.json is regenerated via make generate-json-schemas.

@trask

trask commented Jun 10, 2026

Copy link
Copy Markdown
Member

Hi! As part of #275, this repository switched to Towncrier changelog fragments to reduce merge conflicts in CHANGELOG.md.

Please move this PR's changelog entry out of CHANGELOG.md and into this Towncrier fragment:

Create changelog.d/144.enhancement.md containing the change log entry, e.g.:

Make `BlobPart.content` optional and add a free-form `stripped_reason` marker on the `gen_ai.input.messages` `BlobPart`, so an instrumentation can record that it observed an inline blob but intentionally did not capture the payload bytes (size cap exceeded, modality not allowed, redactor failure, etc.) while preserving the original part type and modality. A schema-level `anyOf` requires either a non-null `content` or a non-null `stripped_reason`. Scoped to input-messages `BlobPart`; if accepted, follow-up PRs replicate the change to `FilePart`/`UriPart` and to the output-messages and system-instructions schemas.

After adding the fragment, please remove this PR's direct edit to CHANGELOG.md. Towncrier will add the PR link from the fragment filename when release notes are generated.

Thanks!

…ssages

Make BlobPart.content optional and add a free-form stripped_reason marker
on the gen_ai.input.messages BlobPart, so an instrumentation can record
that it observed an inline blob but intentionally did not capture the
payload bytes (size cap exceeded, modality not allowed, redactor failure,
etc.) while preserving the original part type and modality. A schema-level
anyOf requires either a non-null content or a non-null stripped_reason.

Scoped to input-messages BlobPart per review feedback; if accepted,
follow-up PRs replicate the change to FilePart/UriPart and to the
output-messages and system-instructions schemas.

Rebased onto main after the Towncrier migration (open-telemetry#275) and the
models.ipynb -> models.py conversion (open-telemetry#226). The input-only scoping is
expressed via input-messages variants of BlobPart/ChatMessage in
models.py that reuse the shared class names, so the generated $defs keys
and wire format stay identical; JSON schemas are regenerated via
make generate-json-schemas. The changelog entry moved from CHANGELOG.md
to changelog.d/144.enhancement.md.
@Mandark-droid
Mandark-droid force-pushed the gen-ai/multimodal-stripped-reason branch from ea3a174 to 6244073 Compare June 10, 2026 19:55
@Mandark-droid

Copy link
Copy Markdown
Contributor Author

@trask done — the changelog entry now lives in changelog.d/144.enhancement.md and the direct CHANGELOG.md edit is removed (commit 6244073).

The rebase onto current main also ports the PR to the #226 notebook-to-Python conversion. Since docs/gen-ai/non-normative/models.py now generates all message schemas from one shared part union, the input-only scoping is expressed via input-messages variants of BlobPart/ChatMessage (a small namespace class in models.py that reuses the shared class names, so the generated $defs keys and the wire format stay identical across schemas). gen-ai-input-messages.json is regenerated via make generate-json-schemas; the resulting JSON diff is the same shape that was previously reviewed — top-level anyOf, nullable content, and the stripped_reason property, with no changes to the output-messages or system-instructions schemas.

If maintainers would rather not carry the input-only variant classes in models.py, the alternative is to apply the change to the shared BlobPart (which widens scope back to all three schemas) — happy to go either way.

@Mandark-droid

Copy link
Copy Markdown
Contributor Author

Thanks @lmolkova — I think we agree on most of this, and the pieces you list map onto the current PR stack:

  • A documented capture-size limit option — agreed this is needed and complementary. It is instrumentation configuration rather than schema shape, so I see it as a docs follow-up that this PR's schema change enables (the marker is what an instrumentation sets when that limit fires).
  • Size on each affected part — that is exactly gen-ai: add optional byte_size to multimodal content parts #143 (byte_size on BlobPart/FilePart/UriPart). Extending it to TextPart is a one-line addition there; happy to fold it in if you want it.
  • Trim-to-size for text — agreed, trimming works for text because a prefix is still useful. It does not transfer to binary parts: a truncated blob is not decodable, so for BlobPart the real choice is captured vs not captured. That is why this PR makes content omittable rather than truncatable.
  • content_dropped: true mimicking OTLP dropped attributes vs a reason string — I am open to converging on the boolean form if that is the more OTel-consistent shape; the core of this PR is "content optional + explicit stripped marker", and the marker's type is a one-line change. Two things made me start with a free-form string: (a) it is deliberately non-enumerated, so there is no reason taxonomy for the spec to maintain (the failure-mode concern you mention with recording "why"); and (b) in practice operators do need to distinguish "size cap fired" from "redactor errored" without correlating collector logs — the string costs nothing when unused.

On "presence of non-0 size would be an indication on its own": byte_size alone cannot carry that signal, because it is optional and not always cheaply observable (e.g. a pre-existing remote URI), and absence of content alone is ambiguous against providers that legitimately reference blobs without inline bytes. An explicit marker keeps "instrumentation stripped this" distinguishable from "this was never inline".

If the WG preference lands on size-limit-config + byte_size everywhere + a boolean marker, I can reshape this PR to that in one pass — the schema mechanics (optional content + anyOf constraint) stay the same either way.

@opentelemetry-pr-dashboard

Copy link
Copy Markdown

Pull request dashboard status

  • Waiting on: Reviewers
  • Next step: Review the latest changes.

@open-telemetry open-telemetry deleted a comment from otelbot Bot Jul 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

4 participants