Repository navigation
Commit 631f714
authored
fix(email): partition an attached email (message/rfc822) instead of raising AssertionError (Unstructured-IO#4597)
## Summary
`partition_email()` processes attachments by default. When a message has
a
`message/rfc822` part, the whole call fails with a bare `AssertionError`
and
the body elements are lost too. Common inputs have such a part:
- an email forwarded "as attachment" (Outlook, Thunderbird),
- a delivery-failure bounce (`multipart/report` with the returned
message),
- a `multipart/digest`, where every part is `message/rfc822`.
The repository's own fixture
`example-docs/eml/mime-multipart-digest.eml`
fails this way, through `partition()` too.
Cause: for a `message/rfc822` part, `get_content()` returns the enclosed
`EmailMessage`, not `str` or `bytes`, so
`_AttachmentPartitioner._file_bytes`
fails its `assert isinstance(content, bytes)`. The assert sits outside
the
`try`/`except` that skips attachments it cannot partition.
Fix, in `unstructured/partition/email.py`:
- `_file_bytes` serializes an enclosed message with `as_bytes()`.
- For a `message/rfc822` part, `content_type="message/rfc822"` is added
to the
kwargs of the nested `partition()` call. Without it the bytes go through
libmagic, which reports a message as `text/plain` when its first header
is
e.g. `MIME-Version`, `Message-ID` or `To` and not
`From`/`Received`/`Date`.
The attachment's raw headers then come out as `Title` elements. Python's
`EmailMessage` writes such messages: the forwarded message in the new
test
starts with `Content-Type`, which libmagic does not recognize either.
The
hint is merged into the kwargs only for these parts, so a
`content_type=`
kwarg passed to `partition_email()` directly still reaches the other
attachments as before instead of colliding with an explicit keyword.
Related:
- Unstructured-IO#4424 is the sibling fix in the same `_file_bytes` function (the
`KeyError`
on a `multipart/*` attachment). Its comment already lists
`message/rfc822`
among the types `get_content()` handles, but that path still hit the
assert.
- Unstructured-IO#3604 (closed in October 2024 with no fix linked) reported
`process_attachments=True` failing on a nested `.eml`, with a
`TypeError` in
the implementation of that time. A maintainer asked there for an email
that
reproduces it so a test could guard the behavior. The tests here use two
such emails: the existing digest fixture and a forwarded message built
in
the test.
## Validation
Environment: `main` at c74e6c4, Python 3.12,
`uv sync --locked --group test` (no extras; libmagic installed).
New tests in `test_unstructured/partition/test_email.py` (attachments
section):
- `test_partition_email_partitions_the_messages_of_a_multipart_digest`:
uses
the existing `mime-multipart-digest.eml` fixture. It checks the three
messages' text, their `subject` metadata and `attached_to_filename`.
-
`test_partition_email_partitions_a_message_forwarded_as_an_attachment`:
an
`EmailMessage` forwarded with `add_attachment()` whose first header is
`Content-Type`. It checks the body, the forwarded text, `subject` and
`sent_from`.
Results:
- New tests without the fix:
`pytest test_unstructured/partition/test_email.py -k "multipart_digest
or forwarded_as_an_attachment"`
gives 2 failed, both `AssertionError` at
`unstructured/partition/email.py:485`
(`assert isinstance(content, bytes)`). The whole file without the fix
gives
2 failed, 77 passed, 1 skipped.
- New tests with the fix: 2 passed.
- With only the `as_bytes()` change and no `content_type` hint, the
digest
test passes and the forwarded-message test fails with 6 extra `Title`
elements (the forwarded message's raw headers). That is why the hint is
part of the fix.
- `pytest test_unstructured/partition/test_email.py`: 79 passed, 1
skipped.
The skip is the WEBP test that needs `unstructured_inference`.
- `pytest test_unstructured/file_utils/test_filetype.py
test_unstructured/partition/test_msg.py`:
362 passed, 1 failed. The failure is
`test_partition_msg_can_process_attachments`, which fails the same way
on
`main` because the `pptx` extra is not installed. `msg.py` has its own
attachment partitioner and is not changed.
- Manual check of a bounce (`multipart/report` with `text/plain`,
`message/delivery-status` and a `message/rfc822` part whose first header
is
`MIME-Version`): returns the bounce text, the delivery-status lines and
the
returned message's body with `subject == "Lunch"`. Before the fix it
raised
`AssertionError`. A forward of a forward is also partitioned at every
level.
- `make check` (ruff 0.15.10 from the `lint` group): `All checks
passed!`,
`355 files already formatted`.
- `bash scripts/test-check-new-release-version.sh`: `Passed 27 version
guard tests.`
- `VERSION_BASE_REF=upstream/main bash
scripts/check-new-release-version.sh`:
`PASS: unchanged development version 0.27.35-dev (no tag lookup).`
- No `CHANGELOG.md` or `unstructured/__version__.py` changes.
## Release note
`partition_email()`, and `partition()` on an `.eml` file, now partition
an
attached email (a message forwarded as an attachment, the returned
message in
a bounce, each message of a `multipart/digest`) like any other
attachment.
Before, they raised `AssertionError`. Each element from the attached
message
carries that message's own `subject`, `sent_from` and other email
metadata.
## Checklist
- [x] The PR title is a user-facing release-note entry with a
conventional commit prefix.
- [ ] Apply a category label (`enhancement`, `bug`, or an appropriate
existing label), or ask a maintainer to apply it. A maintainer needs to
apply `bug`.
- [ ] For breaking changes, use `!` in the title and explain impact and
migration above. (Not a breaking change.)
---------
Co-authored-by: sinnercode228 <[email protected]>1 parent 968c415 commit 631f714
2 files changed
Lines changed: 214 additions & 4 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
4 | 4 | | |
5 | 5 | | |
6 | 6 | | |
| 7 | + | |
7 | 8 | | |
8 | 9 | | |
9 | 10 | | |
10 | 11 | | |
11 | | - | |
| 12 | + | |
12 | 13 | | |
13 | 14 | | |
14 | 15 | | |
| |||
23 | 24 | | |
24 | 25 | | |
25 | 26 | | |
| 27 | + | |
26 | 28 | | |
27 | 29 | | |
28 | 30 | | |
| |||
31 | 33 | | |
32 | 34 | | |
33 | 35 | | |
34 | | - | |
| 36 | + | |
35 | 37 | | |
36 | 38 | | |
37 | 39 | | |
| |||
490 | 492 | | |
491 | 493 | | |
492 | 494 | | |
| 495 | + | |
| 496 | + | |
| 497 | + | |
| 498 | + | |
| 499 | + | |
| 500 | + | |
| 501 | + | |
| 502 | + | |
| 503 | + | |
| 504 | + | |
| 505 | + | |
| 506 | + | |
| 507 | + | |
| 508 | + | |
| 509 | + | |
| 510 | + | |
| 511 | + | |
| 512 | + | |
| 513 | + | |
| 514 | + | |
| 515 | + | |
| 516 | + | |
| 517 | + | |
| 518 | + | |
| 519 | + | |
| 520 | + | |
| 521 | + | |
| 522 | + | |
| 523 | + | |
| 524 | + | |
| 525 | + | |
| 526 | + | |
| 527 | + | |
| 528 | + | |
| 529 | + | |
| 530 | + | |
| 531 | + | |
| 532 | + | |
| 533 | + | |
| 534 | + | |
| 535 | + | |
| 536 | + | |
| 537 | + | |
| 538 | + | |
| 539 | + | |
| 540 | + | |
| 541 | + | |
| 542 | + | |
| 543 | + | |
| 544 | + | |
| 545 | + | |
| 546 | + | |
| 547 | + | |
| 548 | + | |
| 549 | + | |
| 550 | + | |
| 551 | + | |
| 552 | + | |
| 553 | + | |
| 554 | + | |
| 555 | + | |
| 556 | + | |
| 557 | + | |
| 558 | + | |
| 559 | + | |
| 560 | + | |
| 561 | + | |
| 562 | + | |
| 563 | + | |
| 564 | + | |
| 565 | + | |
| 566 | + | |
| 567 | + | |
| 568 | + | |
| 569 | + | |
| 570 | + | |
| 571 | + | |
| 572 | + | |
| 573 | + | |
| 574 | + | |
| 575 | + | |
| 576 | + | |
| 577 | + | |
| 578 | + | |
| 579 | + | |
493 | 580 | | |
494 | 581 | | |
495 | 582 | | |
| |||
787 | 874 | | |
788 | 875 | | |
789 | 876 | | |
| 877 | + | |
| 878 | + | |
| 879 | + | |
| 880 | + | |
| 881 | + | |
| 882 | + | |
| 883 | + | |
| 884 | + | |
| 885 | + | |
| 886 | + | |
| 887 | + | |
| 888 | + | |
| 889 | + | |
| 890 | + | |
| 891 | + | |
| 892 | + | |
| 893 | + | |
| 894 | + | |
| 895 | + | |
| 896 | + | |
| 897 | + | |
| 898 | + | |
| 899 | + | |
| 900 | + | |
| 901 | + | |
| 902 | + | |
| 903 | + | |
| 904 | + | |
| 905 | + | |
| 906 | + | |
| 907 | + | |
| 908 | + | |
| 909 | + | |
| 910 | + | |
| 911 | + | |
| 912 | + | |
| 913 | + | |
| 914 | + | |
| 915 | + | |
| 916 | + | |
| 917 | + | |
| 918 | + | |
| 919 | + | |
| 920 | + | |
| 921 | + | |
| 922 | + | |
| 923 | + | |
| 924 | + | |
| 925 | + | |
| 926 | + | |
| 927 | + | |
| 928 | + | |
| 929 | + | |
| 930 | + | |
| 931 | + | |
| 932 | + | |
| 933 | + | |
| 934 | + | |
| 935 | + | |
| 936 | + | |
| 937 | + | |
| 938 | + | |
| 939 | + | |
| 940 | + | |
| 941 | + | |
| 942 | + | |
| 943 | + | |
| 944 | + | |
| 945 | + | |
| 946 | + | |
| 947 | + | |
| 948 | + | |
| 949 | + | |
| 950 | + | |
| 951 | + | |
| 952 | + | |
| 953 | + | |
| 954 | + | |
| 955 | + | |
| 956 | + | |
| 957 | + | |
| 958 | + | |
| 959 | + | |
| 960 | + | |
790 | 961 | | |
791 | 962 | | |
792 | 963 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
8 | 8 | | |
9 | 9 | | |
10 | 10 | | |
| 11 | + | |
11 | 12 | | |
12 | 13 | | |
13 | 14 | | |
14 | 15 | | |
15 | 16 | | |
16 | | - | |
| 17 | + | |
17 | 18 | | |
18 | 19 | | |
19 | 20 | | |
| |||
428 | 429 | | |
429 | 430 | | |
430 | 431 | | |
| 432 | + | |
| 433 | + | |
| 434 | + | |
| 435 | + | |
| 436 | + | |
| 437 | + | |
431 | 438 | | |
432 | 439 | | |
433 | 440 | | |
434 | 441 | | |
435 | 442 | | |
436 | 443 | | |
437 | | - | |
| 444 | + | |
438 | 445 | | |
439 | 446 | | |
440 | 447 | | |
| |||
482 | 489 | | |
483 | 490 | | |
484 | 491 | | |
| 492 | + | |
| 493 | + | |
| 494 | + | |
| 495 | + | |
| 496 | + | |
485 | 497 | | |
486 | 498 | | |
487 | 499 | | |
| 500 | + | |
| 501 | + | |
| 502 | + | |
| 503 | + | |
| 504 | + | |
| 505 | + | |
| 506 | + | |
| 507 | + | |
| 508 | + | |
| 509 | + | |
| 510 | + | |
| 511 | + | |
| 512 | + | |
| 513 | + | |
| 514 | + | |
| 515 | + | |
| 516 | + | |
| 517 | + | |
| 518 | + | |
| 519 | + | |
| 520 | + | |
| 521 | + | |
| 522 | + | |
| 523 | + | |
| 524 | + | |
| 525 | + | |
| 526 | + | |
488 | 527 | | |
489 | 528 | | |
490 | 529 | | |
| |||
0 commit comments