Skip to content

fix(otel): omit absent fields from error exception events - #901

Merged
zhongkechen merged 3 commits into
mainfrom
issue-894-partial-error-events
Nov 2, 2026
Merged

zhongkechen merged 3 commits into
mainfrom
issue-894-partial-error-events

Conversation

@zhongkechen

@zhongkechen zhongkechen commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #894

Problem

A callback or other operation that fails with only ErrorMessage or only
ErrorType produced an OTel exception event with an invented value for the
missing field: a message-only error reported exception.type = "Exception",
and a type-only error reported an empty exception.message. A synthetic
exception.stacktrace was emitted too. Consumers could not distinguish absent
metadata from generated values.

exception_events.py built a plain Exception carrier and called
Span.record_exception, which derives attributes from the carrier before
merging explicit attributes. Omitting a field from that explicit mapping cannot
remove the generated value. Core ErrorObject already preserves absent fields
as None.

Change

record_error_object_exception emits the event directly with
span.add_event("exception", attributes=..., timestamp=...), using the existing
mapping of checkpointed fields. The carrier exception, exception_from_error_object,
and _DEFAULT_MESSAGE fallback are removed.

  • When a type or message is present, emit only the checkpointed fields. Explicit
    empty strings remain distinct from absent fields.
  • When both type and message are absent, emit no exception event, including when
    a stacktrace is present. The OTel exception conventions
    require exception.message if exception.type is unset, and vice versa;
    exception.stacktrace alone does not satisfy that requirement. This guard
    avoids both malformed events and invented diagnostics. The span's error
    status and durable.operation.status still report the failure.
  • Stop generating exception.escaped and synthetic stacktraces from the
    carrier. A stored, nonempty stacktrace is still included when an exception
    event is emitted. Removing synthetic stacktraces also addresses that symptom
    from [Bug]: Align OTel exception stacktrace emission with OpenTelemetry JS #886.
  • Document partial fields, explicit empty strings, and event suppression in
    the README's status-mapping section.

Core checkpoint serialization, CallbackExternalError diagnostics, and the
normalized event timestamp are unchanged. No conformance requirements or
handlers are changed.

Tests

  • tests/test_status_mapping.py checks the exact exception attributes for five
    partial-field combinations: message-only, type-only, and three combinations
    of explicit empty strings. These cover operation and failed-attempt recorders
    in both views; failed-attempt cases also cover replayed and non-replayed
    delivery. Exact attribute assertions exclude synthetic fields.
  • test_error_without_type_or_message_records_no_event covers 12 combinations:
    both views, operation and attempt recorders, and absent, empty, or nonempty
    stacktraces. It asserts no exception events and retained span ERROR status.
  • tests/e2e/test_partial_callback_error_int.py adds 12 callback cases through
    DurableFunctionTestRunner, context.wait_for_callback, and
    send_callback_failure. These cover message-only and type-only errors in
    both views, same-invocation and post-resume completion, and caught and uncaught
    failures. They check that the submitter step runs once, caught failures allow
    handler success, and uncaught failures preserve the failure statuses.

The former message-only fallback test is replaced by the partial-field tests.
test_failed_operation_without_type_or_message_uses_fallback is replaced by
test_error_without_type_or_message_records_no_event, which checks suppression
and retained error status. The carrier's fallback-message test is removed with
the function it tested.

Validation

At current head ad4e452a25e0c51ece143cda0f8aba7cd711c1cd:

  • hatch run dev-otel:python -m pytest packages/aws-durable-execution-sdk-python-otel/tests/test_status_mapping.py -q:
    198 passed on Python 3.13.18.
  • A direct comparison with earlier revision a2e9c7f2 reproduced an empty
    exception event for missing type/message with absent or empty stacktraces,
    and a stacktrace-only event for a nonempty stacktrace. The current code emits
    no event in all three cases.

Earlier validation reported passing callback e2e tests, formatting, linting,
and changed-file type checks. Its full OTel-suite run reported 12 failures in
tests/e2e/test_view_registration.py, also reproduced with the change reverted.
Those broader checks were not rerun during the current-head validation above.

Validation is local; Lambda and X-Ray behavior has not been verified.

A checkpointed error that stores only a message or only a type gained an
invented value for the other field. `Span.record_exception` derives
`exception.type`, `exception.message` and `exception.stacktrace` from the
carrier exception and only then merges the explicit attributes, so the plain
`Exception` carrier supplied a type of `Exception`, an empty message and a
synthetic stack trace that the explicit attributes could not delete.

Emit the event with `add_event` directly from the checkpointed fields. The
carrier exception and its default-message fallback are no longer needed, so
both are removed.
@zhongkechen
zhongkechen deployed to ai-pr-review-runtime November 2, 2026 19:39 — with GitHub Actions Active
@zhongkechen
zhongkechen deployed to ai-pr-review-runtime November 2, 2026 19:40 — with GitHub Actions Active
@github-actions

This comment has been minimized.

@zhongkechen
zhongkechen deployed to ai-pr-review-runtime November 2, 2026 21:27 — with GitHub Actions Active
The OTel exception conventions make `exception.type` and `exception.message`
each required when the other is absent, so an event carrying neither is
malformed. Omitting both, as the previous commit did, traded one convention
violation for another.

Record no event in that case instead. Inventing a value is the defect this
module exists to avoid, and the span's error status and
`durable.operation.status` already report the failure. A stack trace alone
does not satisfy the requirement, since `exception.stacktrace` is only
recommended.
@zhongkechen
zhongkechen force-pushed the issue-894-partial-error-events branch from 0b63e51 to ad4e452 Compare November 2, 2026 21:32
@zhongkechen
zhongkechen deployed to ai-pr-review-runtime November 2, 2026 21:33 — with GitHub Actions Active
@github-actions

github-actions Bot commented Nov 2, 2026

Copy link
Copy Markdown
Contributor

Codex AI review

Two approach findings:

  • Rule 7: packages/aws-durable-execution-sdk-python-otel/src/aws_durable_execution_sdk_python_otel/exception_events.py:49
  • Rule 4: packages/aws-durable-execution-sdk-python-otel/tests/e2e/test_partial_callback_error_int.py:117

No SDK correctness regression confirmed. Tests were inspected but not executed, as requested.

Reviewed commit ad4e452a25e0c51ece143cda0f8aba7cd711c1cd. Workflow run

@zhongkechen
zhongkechen merged commit 7a7996d into main Nov 2, 2026
50 checks passed
@zhongkechen
zhongkechen deleted the issue-894-partial-error-events branch November 2, 2026 21:48

This branch was successfully deployed

1 active deployment
ai-pr-review-runtime — ad4e452a Deployed Nov 2, 2026 by zhongkechen via ai-pr-review / Codex review / Generate Codex review #2743
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: OTel adds missing type/message to partial callback errors

3 participants