Skip to content

Commit 631f714

Browse files
fix(email): partition an attached email (message/rfc822) instead of raising AssertionError (Unstructured-IO#4597)
## Summary `partition_email()` processes attachments by default. When a message has a `message/rfc822` part, the whole call fails with a bare `AssertionError` and the body elements are lost too. Common inputs have such a part: - an email forwarded "as attachment" (Outlook, Thunderbird), - a delivery-failure bounce (`multipart/report` with the returned message), - a `multipart/digest`, where every part is `message/rfc822`. The repository's own fixture `example-docs/eml/mime-multipart-digest.eml` fails this way, through `partition()` too. Cause: for a `message/rfc822` part, `get_content()` returns the enclosed `EmailMessage`, not `str` or `bytes`, so `_AttachmentPartitioner._file_bytes` fails its `assert isinstance(content, bytes)`. The assert sits outside the `try`/`except` that skips attachments it cannot partition. Fix, in `unstructured/partition/email.py`: - `_file_bytes` serializes an enclosed message with `as_bytes()`. - For a `message/rfc822` part, `content_type="message/rfc822"` is added to the kwargs of the nested `partition()` call. Without it the bytes go through libmagic, which reports a message as `text/plain` when its first header is e.g. `MIME-Version`, `Message-ID` or `To` and not `From`/`Received`/`Date`. The attachment's raw headers then come out as `Title` elements. Python's `EmailMessage` writes such messages: the forwarded message in the new test starts with `Content-Type`, which libmagic does not recognize either. The hint is merged into the kwargs only for these parts, so a `content_type=` kwarg passed to `partition_email()` directly still reaches the other attachments as before instead of colliding with an explicit keyword. Related: - Unstructured-IO#4424 is the sibling fix in the same `_file_bytes` function (the `KeyError` on a `multipart/*` attachment). Its comment already lists `message/rfc822` among the types `get_content()` handles, but that path still hit the assert. - Unstructured-IO#3604 (closed in October 2024 with no fix linked) reported `process_attachments=True` failing on a nested `.eml`, with a `TypeError` in the implementation of that time. A maintainer asked there for an email that reproduces it so a test could guard the behavior. The tests here use two such emails: the existing digest fixture and a forwarded message built in the test. ## Validation Environment: `main` at c74e6c4, Python 3.12, `uv sync --locked --group test` (no extras; libmagic installed). New tests in `test_unstructured/partition/test_email.py` (attachments section): - `test_partition_email_partitions_the_messages_of_a_multipart_digest`: uses the existing `mime-multipart-digest.eml` fixture. It checks the three messages' text, their `subject` metadata and `attached_to_filename`. - `test_partition_email_partitions_a_message_forwarded_as_an_attachment`: an `EmailMessage` forwarded with `add_attachment()` whose first header is `Content-Type`. It checks the body, the forwarded text, `subject` and `sent_from`. Results: - New tests without the fix: `pytest test_unstructured/partition/test_email.py -k "multipart_digest or forwarded_as_an_attachment"` gives 2 failed, both `AssertionError` at `unstructured/partition/email.py:485` (`assert isinstance(content, bytes)`). The whole file without the fix gives 2 failed, 77 passed, 1 skipped. - New tests with the fix: 2 passed. - With only the `as_bytes()` change and no `content_type` hint, the digest test passes and the forwarded-message test fails with 6 extra `Title` elements (the forwarded message's raw headers). That is why the hint is part of the fix. - `pytest test_unstructured/partition/test_email.py`: 79 passed, 1 skipped. The skip is the WEBP test that needs `unstructured_inference`. - `pytest test_unstructured/file_utils/test_filetype.py test_unstructured/partition/test_msg.py`: 362 passed, 1 failed. The failure is `test_partition_msg_can_process_attachments`, which fails the same way on `main` because the `pptx` extra is not installed. `msg.py` has its own attachment partitioner and is not changed. - Manual check of a bounce (`multipart/report` with `text/plain`, `message/delivery-status` and a `message/rfc822` part whose first header is `MIME-Version`): returns the bounce text, the delivery-status lines and the returned message's body with `subject == "Lunch"`. Before the fix it raised `AssertionError`. A forward of a forward is also partitioned at every level. - `make check` (ruff 0.15.10 from the `lint` group): `All checks passed!`, `355 files already formatted`. - `bash scripts/test-check-new-release-version.sh`: `Passed 27 version guard tests.` - `VERSION_BASE_REF=upstream/main bash scripts/check-new-release-version.sh`: `PASS: unchanged development version 0.27.35-dev (no tag lookup).` - No `CHANGELOG.md` or `unstructured/__version__.py` changes. ## Release note `partition_email()`, and `partition()` on an `.eml` file, now partition an attached email (a message forwarded as an attachment, the returned message in a bounce, each message of a `multipart/digest`) like any other attachment. Before, they raised `AssertionError`. Each element from the attached message carries that message's own `subject`, `sent_from` and other email metadata. ## Checklist - [x] The PR title is a user-facing release-note entry with a conventional commit prefix. - [ ] Apply a category label (`enhancement`, `bug`, or an appropriate existing label), or ask a maintainer to apply it. A maintainer needs to apply `bug`. - [ ] For breaking changes, use `!` in the title and explain impact and migration above. (Not a breaking change.) --------- Co-authored-by: sinnercode228 <[email protected]>
1 parent 968c415 commit 631f714

2 files changed

Lines changed: 214 additions & 4 deletions

File tree

‎test_unstructured/partition/test_email.py‎

Lines changed: 173 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,11 +4,12 @@
44

55
from __future__ import annotations
66

7+
import base64
78
import email.policy
89
import io
910
import tempfile
1011
from email.message import EmailMessage
11-
from typing import Any
12+
from typing import IO, Any
1213

1314
import pytest
1415

@@ -23,6 +24,7 @@
2324
from unstructured.chunking.title import chunk_by_title
2425
from unstructured.documents.elements import (
2526
CompositeElement,
27+
Element,
2628
ListItem,
2729
NarrativeText,
2830
Table,
@@ -31,7 +33,7 @@
3133
Title,
3234
)
3335
from unstructured.file_utils.model import FileType
34-
from unstructured.partition.auto import _PartitionerLoader
36+
from unstructured.partition.auto import _PartitionerLoader, partition
3537
from unstructured.partition.email import EmailPartitioningContext, partition_email
3638

3739
EXPECTED_OUTPUT = [
@@ -490,6 +492,91 @@ def test_partition_email_does_not_raise_on_multipart_attachment():
490492
assert elements[1].metadata.attached_to_filename == "mime-attach-multipart-signed.eml"
491493

492494

495+
def test_partition_email_partitions_the_messages_of_a_multipart_digest():
496+
"""Each part of a `multipart/digest` is a `message/rfc822` part, i.e. an email in its own right.
497+
498+
`get_content()` returns an `EmailMessage` for such a part, not `str` or `bytes`.
499+
"""
500+
elements = partition_email(
501+
example_doc_path("eml/mime-multipart-digest.eml"), process_attachments=True
502+
)
503+
504+
assert elements == [
505+
NarrativeText("This is the first message in the digest."),
506+
NarrativeText("This is the second message in the digest."),
507+
NarrativeText("This is the third message in the digest."),
508+
]
509+
assert [e.metadata.subject for e in elements] == [
510+
"First Message",
511+
"Second Message",
512+
"Third Message",
513+
]
514+
assert all(e.metadata.attached_to_filename == "mime-multipart-digest.eml" for e in elements)
515+
516+
517+
def test_partition_email_partitions_a_message_forwarded_as_an_attachment():
518+
forwarded = EmailMessage()
519+
forwarded.set_content("Are we still on for lunch tomorrow at noon?")
520+
# -- these headers follow "Content-Type:", so libmagic alone would take this for plain text --
521+
forwarded["From"] = "[email protected]"
522+
forwarded["To"] = "[email protected]"
523+
forwarded["Subject"] = "Lunch"
524+
msg = EmailMessage()
525+
msg.set_content("See the message below.")
526+
msg.add_attachment(forwarded)
527+
528+
elements = partition_email(file=io.BytesIO(msg.as_bytes()), process_attachments=True)
529+
530+
assert elements == [
531+
NarrativeText("See the message below."),
532+
NarrativeText("Are we still on for lunch tomorrow at noon?"),
533+
]
534+
assert elements[1].metadata.subject == "Lunch"
535+
assert elements[1].metadata.sent_from == ["[email protected]"]
536+
537+
538+
# -- a PNG signature, then a bare CR, a bare LF, a CRLF and non-ASCII bytes --
539+
ATTACHMENT_BYTES = b"\x89PNG\r\n\x1a\n" + b"a\rb\nc\r\nd" + "é".encode()
540+
541+
542+
@pytest.mark.parametrize(
543+
"file_name",
544+
[
545+
# -- email > level-1.eml > attachment --
546+
"binary-1.png",
547+
"7bit-1.png",
548+
"base64-1.png",
549+
# -- email > level-1.eml > level-2.eml > attachment --
550+
"binary-2.png",
551+
"7bit-2.png",
552+
"base64-2.png",
553+
],
554+
)
555+
def test_partition_email_passes_on_the_exact_bytes_of_an_attachment_of_an_attached_email(
556+
request: FixtureRequest, nested_attached_emails: io.BytesIO, file_name: str
557+
):
558+
"""A non-Base64 attachment keeps its CR and LF bytes, whatever its Content-Transfer-Encoding.
559+
560+
They are data in such a part, not line breaks, as when its email is partitioned on its own.
561+
The Base64-encoded attachments are the control.
562+
"""
563+
received: dict[str | None, bytes] = {}
564+
565+
def partition_(file: IO[bytes], **kwargs: Any) -> list[Element]:
566+
received[kwargs["metadata_filename"]] = file.read()
567+
file.seek(0)
568+
# -- partition an attached email for real, so its own attachments get here too --
569+
if kwargs.get("content_type") == "message/rfc822":
570+
return partition(file=file, **kwargs)
571+
return []
572+
573+
function_mock(request, "unstructured.partition.auto.partition", side_effect=partition_)
574+
575+
partition_email(file=nested_attached_emails, process_attachments=True)
576+
577+
assert received[file_name] == ATTACHMENT_BYTES
578+
579+
493580
def test_partition_email_skips_a_WEBP_attachment_when_image_dependencies_are_not_installed(
494581
request: FixtureRequest, caplog: LogCaptureFixture, webp_attachment_email: io.BytesIO
495582
):
@@ -787,6 +874,90 @@ def get_last_modified_date_(self, request: FixtureRequest) -> Mock:
787874
# ================================================================================================
788875

789876

877+
@pytest.fixture()
878+
def nested_attached_emails() -> io.BytesIO:
879+
"""An email with an email attached, which in turn has an email attached.
880+
881+
Each attached email has `ATTACHMENT_BYTES` attached three times: sent with
882+
Content-Transfer-Encoding "binary", with no Content-Transfer-Encoding header (which means
883+
"7bit"; these bytes are not, but such parts are found in real mail) and in Base64. Built from
884+
bytes, since `EmailMessage` would choose its own transfer-encoding.
885+
"""
886+
attachment_base64 = base64.b64encode(ATTACHMENT_BYTES)
887+
level_2 = (
888+
b"From: [email protected]\r\n"
889+
b"Subject: Level 2\r\n"
890+
b"MIME-Version: 1.0\r\n"
891+
b'Content-Type: multipart/mixed; boundary="level-2"\r\n'
892+
b"\r\n"
893+
b"--level-2\r\n"
894+
b"Content-Type: text/plain\r\n"
895+
b"\r\n"
896+
b"This is the level 2 email.\r\n"
897+
b"--level-2\r\n"
898+
b"Content-Type: image/png\r\n"
899+
b'Content-Disposition: attachment; filename="binary-2.png"\r\n'
900+
b"Content-Transfer-Encoding: binary\r\n"
901+
b"\r\n" + ATTACHMENT_BYTES + b"\r\n"
902+
b"--level-2\r\n"
903+
b"Content-Type: image/png\r\n"
904+
b'Content-Disposition: attachment; filename="7bit-2.png"\r\n'
905+
b"\r\n" + ATTACHMENT_BYTES + b"\r\n"
906+
b"--level-2\r\n"
907+
b"Content-Type: image/png\r\n"
908+
b'Content-Disposition: attachment; filename="base64-2.png"\r\n'
909+
b"Content-Transfer-Encoding: base64\r\n"
910+
b"\r\n" + attachment_base64 + b"\r\n"
911+
b"--level-2--\r\n"
912+
)
913+
level_1 = (
914+
b"From: [email protected]\r\n"
915+
b"Subject: Level 1\r\n"
916+
b"MIME-Version: 1.0\r\n"
917+
b'Content-Type: multipart/mixed; boundary="level-1"\r\n'
918+
b"\r\n"
919+
b"--level-1\r\n"
920+
b"Content-Type: text/plain\r\n"
921+
b"\r\n"
922+
b"This is the level 1 email.\r\n"
923+
b"--level-1\r\n"
924+
b"Content-Type: image/png\r\n"
925+
b'Content-Disposition: attachment; filename="binary-1.png"\r\n'
926+
b"Content-Transfer-Encoding: binary\r\n"
927+
b"\r\n" + ATTACHMENT_BYTES + b"\r\n"
928+
b"--level-1\r\n"
929+
b"Content-Type: image/png\r\n"
930+
b'Content-Disposition: attachment; filename="7bit-1.png"\r\n'
931+
b"\r\n" + ATTACHMENT_BYTES + b"\r\n"
932+
b"--level-1\r\n"
933+
b"Content-Type: image/png\r\n"
934+
b'Content-Disposition: attachment; filename="base64-1.png"\r\n'
935+
b"Content-Transfer-Encoding: base64\r\n"
936+
b"\r\n" + attachment_base64 + b"\r\n"
937+
b"--level-1\r\n"
938+
b"Content-Type: message/rfc822\r\n"
939+
b'Content-Disposition: attachment; filename="level-2.eml"\r\n'
940+
b"\r\n" + level_2 + b"\r\n"
941+
b"--level-1--\r\n"
942+
)
943+
return io.BytesIO(
944+
b"From: [email protected]\r\n"
945+
b"Subject: Level 0\r\n"
946+
b"MIME-Version: 1.0\r\n"
947+
b'Content-Type: multipart/mixed; boundary="level-0"\r\n'
948+
b"\r\n"
949+
b"--level-0\r\n"
950+
b"Content-Type: text/plain\r\n"
951+
b"\r\n"
952+
b"This is the email.\r\n"
953+
b"--level-0\r\n"
954+
b"Content-Type: message/rfc822\r\n"
955+
b'Content-Disposition: attachment; filename="level-1.eml"\r\n'
956+
b"\r\n" + level_1 + b"\r\n"
957+
b"--level-0--\r\n"
958+
)
959+
960+
790961
@pytest.fixture()
791962
def webp_attachment_email() -> io.BytesIO:
792963
"""An email with a plain-text body and the example WEBP image attached."""

‎unstructured/partition/email.py‎

Lines changed: 41 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -8,12 +8,13 @@
88
import copy
99
import datetime as dt
1010
import email
11+
import email.encoders
1112
import email.policy
1213
import email.utils
1314
import io
1415
import os
1516
import re
16-
from email.message import EmailMessage, MIMEPart
17+
from email.message import EmailMessage, Message, MIMEPart
1718
from functools import cached_property
1819
from typing import IO, Any, Final, Iterator, cast
1920

@@ -428,13 +429,19 @@ def _iter_elements(self) -> Iterator[Element]:
428429

429430
file = io.BytesIO(self._file_bytes)
430431

432+
partitioning_kwargs = self._ctx.partitioning_kwargs
433+
# -- a `message/rfc822` part is an email whatever its first header is; libmagic reports
434+
# -- e.g. a message starting with "MIME-Version:" as "text/plain" --
435+
if self._attachment.get_content_type() == "message/rfc822":
436+
partitioning_kwargs = {**partitioning_kwargs, "content_type": "message/rfc822"}
437+
431438
# -- partition the attachment --
432439
try:
433440
elements = partition(
434441
file=file,
435442
metadata_filename=self._attachment_file_name,
436443
metadata_last_modified=self._ctx.metadata_last_modified,
437-
**self._ctx.partitioning_kwargs,
444+
**partitioning_kwargs,
438445
)
439446
except BaseException as e:
440447
if not isinstance(e, EXPECTED_ATTACHMENT_ERRORS):
@@ -482,9 +489,41 @@ def _file_bytes(self) -> bytes:
482489
if isinstance(content, str):
483490
return content.encode("utf-8")
484491

492+
# -- a `message/rfc822` part (a message forwarded as an attachment, a returned message in a
493+
# -- bounce, each part of a `multipart/digest`) gets the enclosed message, not bytes --
494+
if isinstance(content, Message):
495+
return self._serialize_message(content)
496+
485497
assert isinstance(content, bytes)
486498
return content
487499

500+
@staticmethod
501+
def _serialize_message(message: Message) -> bytes:
502+
"""Serialized bytes of `message`, the email enclosed in a `message/rfc822` attachment.
503+
504+
`as_bytes()` writes the body of each part as is, except that it rewrites each CR in it,
505+
alone or in a CRLF, as LF. Unless the part is in Base64, which ignores line breaks, that
506+
corrupts binary data such as a PNG image, whatever the Content-Transfer-Encoding says
507+
("binary", "8bit", or none, i.e. "7bit"). So re-encode each part whose decoded bytes hold a
508+
CR in Base64, from those bytes, on a *copy* first; `message` belongs to the parsed email
509+
and is left as is. Each part then reaches its partitioner with the same bytes as when the
510+
enclosed email is partitioned on its own.
511+
"""
512+
enclosed = copy.deepcopy(message)
513+
for part in enclosed.walk():
514+
if part.is_multipart():
515+
continue
516+
# -- the same test for Base64 that `get_payload(decode=True)` makes --
517+
if str(part.get("Content-Transfer-Encoding", "")).lower() == "base64":
518+
continue
519+
data = part.get_payload(decode=True)
520+
if not data or b"\r" not in data:
521+
continue
522+
del part["Content-Transfer-Encoding"]
523+
part.set_payload(data)
524+
email.encoders.encode_base64(part)
525+
return enclosed.as_bytes()
526+
488527
@cached_property
489528
def _serialized_multipart_bytes(self) -> bytes:
490529
"""Raw serialized bytes of a `multipart/*` attachment, re-parseable as a document.

0 commit comments

Comments
 (0)