Skip to content

Commit 911083b

Browse files
authored
fix: partition_email raises KeyError on a multipart/* attachment (#4424)
Fixes #3922 Multipart email attachments could raise `KeyError` and stop email partitioning. This PR serializes multipart attachments as standalone documents so the email and attachment bodies can be processed. ### Behavior - Convert a copy of each multipart attachment to a standalone MIME document with its top-level `Content-Disposition` removed. Preserve the original part and its filename metadata. - Add a valid multipart/signed fixture and regression assertions for the outer body, signed body, attachment filename, and parent email filename. ### Integration and validation Merged current main, retained both upstream WEBP attachment tests, and moved the changelog entries to `0.27.27-dev4` with a matching package version. The merge with main required no additional production code changes. - Focused email tests: 3 passed, 1 skipped (image dependencies are not installed). - `make check-version`: passed. BCE Junior reviewed the narrow repair; its section-divider finding was corrected before commit. Opus 4.6 reviewed this description. GitHub CI results are reported separately from these local checks. (authored by codex)
1 parent 0b7f806 commit 911083b

5 files changed

Lines changed: 94 additions & 2 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,10 @@
1-
## 0.27.27-dev3
1+
## 0.27.27-dev4
22

33
### Fixes
44

5+
- **`partition_email()` no longer crashes on a `multipart/*` attachment**: a multipart sub-part surfaced as an "attachment" (e.g. a PGP/MIME-signed forwarded message) previously raised an uncaught `KeyError` from `email.contentmanager`, which has no `get_content()` handler for any `multipart/*` content-type. `_AttachmentPartitioner._file_bytes` now falls back to the part's raw serialized bytes for this case instead.
6+
- **`partition_email()` no longer drops a `multipart/*` attachment's own body**: serializing such an attachment verbatim left its own `Content-Disposition: attachment` header intact, so re-parsing it as the root of a new message made `EmailMessage.get_body()` skip it entirely instead of finding its body part. The attachment's raw bytes are now serialized from a copy with that header stripped.
7+
58
- **Make uncertain text decoding observable (#4466).** Automatic legacy and BOM-less UTF-16/32 detection now emits `EncodingDetectionWarning`, which callers can promote to an exception with Python warning filters. Unicode BOMs, then strict UTF-8 validity, are honored before statistical detection, so valid UTF-8 whose only non-ASCII content is an invisible character such as U+200B is no longer decoded as `mac_iceland` or `cp850`. Input containing NUL bytes skips the UTF-8 check, because BOM-less UTF-16/32 is often also valid UTF-8. File-type detection's 4,096-byte stream prefix tolerates a multi-byte sequence split at its end, including a UTF-16 surrogate pair after a BOM; UTF-8/ASCII remains the default convention. Inconclusive detection ranks unrestricted candidates instead of accepting the first decodable codec (usually Latin-1), and raises when no text candidate survives. Correctly detected codecs outside `COMMON_ENCODINGS` remain supported. `partition_md(encoding=...)` now strictly decodes local and URL bytes using the supplied encoding. Automatic detection remains heuristic; this change does not guarantee recovery of the original text without a known encoding.
69

710
- **Normalize missing text in text elements.** `Text` and its subclasses, including `Image`, now store `text=None` as an empty string. This prevents string conversion from failing for images with missing detected text. String conversion also returns an empty string if text is later set to `None`. Newly constructed elements serialize this text as `""` instead of `null`; recalculated content hashes change for formerly null text. Supplied IDs and ordinary string text are unchanged.
Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
2+
3+
Subject: Test with PGP-signed forwarded message attachment
4+
MIME-Version: 1.0
5+
Content-Type: multipart/mixed; boundary="===============6075316962646898834=="
6+
7+
--===============6075316962646898834==
8+
Content-Type: text/plain; charset="utf-8"
9+
Content-Transfer-Encoding: 7bit
10+
11+
This is the main email body.
12+
13+
--===============6075316962646898834==
14+
Content-Type: multipart/signed; protocol="application/pgp-signature";
15+
micalg="pgp-sha256"; boundary="===============3567714482309737971=="
16+
Content-Disposition: attachment; filename="signed-message.eml"
17+
MIME-Version: 1.0
18+
19+
--===============3567714482309737971==
20+
Content-Type: text/plain; charset="utf-8"
21+
Content-Transfer-Encoding: 7bit
22+
MIME-Version: 1.0
23+
24+
This is the signed forwarded message body.
25+
26+
--===============3567714482309737971==
27+
MIME-Version: 1.0
28+
Content-Type: application/pgp-signature
29+
30+
-----BEGIN PGP SIGNATURE-----
31+
fakefakefake
32+
-----END PGP SIGNATURE-----
33+
34+
--===============3567714482309737971==--
35+
36+
--===============6075316962646898834==--

‎test_unstructured/partition/test_email.py‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -412,6 +412,32 @@ def test_partition_email_silently_skips_attachments_it_cannot_partition():
412412
]
413413

414414

415+
def test_partition_email_does_not_raise_on_multipart_attachment():
416+
"""A multipart/* "attachment" (e.g. a PGP/MIME-signed forwarded message) must not crash.
417+
418+
`email.contentmanager` has no `get_content()` handler for any `multipart/*` content-type, so
419+
without special-casing this in `_AttachmentPartitioner._file_bytes`, resolving such an
420+
attachment's bytes raises `KeyError` -- crashing the whole `partition_email()` call instead
421+
of gracefully processing or skipping just that one attachment. Regression test for #3922.
422+
423+
The fixture's attachment is itself a genuine boundary-bearing `multipart/signed` MIME part
424+
(a PGP-signed forwarded message), so this also guards a second, subtler bug: serializing
425+
that part verbatim (with its own `Content-Disposition: attachment` header intact) and
426+
re-parsing it as a new root message made `EmailMessage.get_body()` skip it entirely, i.e.
427+
the attachment partitioned to zero elements instead of surfacing its signed body text.
428+
"""
429+
elements = partition_email(
430+
example_doc_path("eml/mime-attach-multipart-signed.eml"), process_attachments=True
431+
)
432+
433+
# -- No exception; the outer email body is partitioned ... --
434+
assert elements[0] == NarrativeText("This is the main email body.")
435+
# -- ... and the signed attachment's own body is also partitioned, not silently dropped. --
436+
assert elements[1] == NarrativeText("This is the signed forwarded message body.")
437+
assert elements[1].metadata.filename == "signed-message.eml"
438+
assert elements[1].metadata.attached_to_filename == "mime-attach-multipart-signed.eml"
439+
440+
415441
def test_partition_email_skips_a_WEBP_attachment_when_image_dependencies_are_not_installed(
416442
request: FixtureRequest, caplog: LogCaptureFixture, webp_attachment_email: io.BytesIO
417443
):

‎unstructured/__version__.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
__version__ = "0.27.27-dev3" # pragma: no cover
1+
__version__ = "0.27.27-dev4" # pragma: no cover

‎unstructured/partition/email.py‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@
55

66
from __future__ import annotations
77

8+
import copy
89
import datetime as dt
910
import email
1011
import email.policy
@@ -438,10 +439,36 @@ def _attachment_file_name(self) -> str | None:
438439
@cached_property
439440
def _file_bytes(self) -> bytes:
440441
"""The bytes of the attached file."""
442+
# -- `email.contentmanager` has no `get_content()` handler registered for any
443+
# -- `multipart/*` content-type (handlers exist for text/*, application/*, image/*,
444+
# -- message/rfc822, etc., but multipart sub-parts are normally consumed via
445+
# -- `.iter_parts()`, not `.get_content()`). A multipart sub-part can still appear as an
446+
# -- "attachment" though, e.g. a PGP/MIME-signed forwarded message (`multipart/signed`)
447+
# -- nested inside a `multipart/mixed` envelope. Fall back to the part's raw serialized
448+
# -- bytes in that case rather than letting `get_content()` raise `KeyError`.
449+
if self._attachment.get_content_type().startswith("multipart/"):
450+
return self._serialized_multipart_bytes
451+
441452
content = self._attachment.get_content()
442453

443454
if isinstance(content, str):
444455
return content.encode("utf-8")
445456

446457
assert isinstance(content, bytes)
447458
return content
459+
460+
@cached_property
461+
def _serialized_multipart_bytes(self) -> bytes:
462+
"""Raw serialized bytes of a `multipart/*` attachment, re-parseable as a document.
463+
464+
`EmailMessage.get_body()` skips any candidate part -- including a `multipart/*` root
465+
message -- whose own `Content-Disposition` is "attachment". This MIME part carries
466+
exactly that header (that's how it surfaced as an attachment via `iter_attachments()`
467+
in the first place), so serializing it verbatim and re-parsing it as the root of a new
468+
message would make `partition_email()` find no body at all. Strip that header on a
469+
*copy* before serializing -- `self._attachment` itself is left untouched since
470+
`_attachment_file_name` still needs its original `Content-Disposition` filename param.
471+
"""
472+
part = copy.deepcopy(self._attachment)
473+
del part["Content-Disposition"]
474+
return part.as_bytes()

0 commit comments

Comments
 (0)