Skip to content

Commit 4fe4097

Browse files
badGarnetclaude
andauthored
fix: preserve attachment elements' own filetype in auto.partition() (#4460)
## Summary When a document with attachments is partitioned, every element in the output was labelled with the **containing** file's MIME type. Content extracted from an email's attached PDF reported `filetype: message/rfc822`, so downstream consumers could not filter or route by document type. Not format-specific to Outlook — it reproduces for any email with attachments, on both the `.eml` and `.msg` paths. ## Root cause The `augment_metadata` closure in `partition()` (`unstructured/partition/auto.py`) re-stamped `metadata.filetype` on every element it was handed, unconditionally. Attachments are partitioned by a nested `partition()` call, so each attachment's elements are stamped with their correct filetype at their own recursion depth, then flattened into the containing document's element list. The outermost call then ran its own `augment_metadata` over the whole flattened list and overwrote all of them. ## Fix Skip the filetype re-stamp for elements carrying `attached_to_filename`. This matches guards the codebase already applies for exactly this case in `partition/common/metadata.py` and in two places in `file_utils/filetype.py` — so the convention is established, it just was not used by this hand-rolled closure. ### One deliberate narrowing `url` and `data_source` are still applied to attachment elements; only the filetype block is guarded. Skipping the element outright would have been the broader change, but the nested `partition()` call receives neither of those values, so attachment elements would come back with `data_source = None` — silently dropping source lineage (record locators, source URLs) for attachment-derived content. The defect here is filetype-only, so the guard is too. ## Verification An `.eml` built with a PDF and a DOCX attachment: | elements | before | after | |---|---|---| | email body (1) | `message/rfc822` | `message/rfc822` | | PDF attachment (25) | `message/rfc822` | `application/pdf` | | DOCX attachment (8) | `message/rfc822` | `…wordprocessingml.document` | ## Tests Adds `test_auto_partition_preserves_the_filetype_of_attachment_elements`, parametrized over the `.eml` and `.msg` paths using existing fixtures (no new binaries). Both cases were confirmed to fail without the fix and pass with it. `test_auto.py`, `test_email.py`, `test_msg.py` and `common/test_metadata.py`: 327 passed, 1 xfailed. Lint and format clean. ## Note The CHANGELOG diff includes a one-character trailing-whitespace removal on the pre-existing 0.27.3 entry, applied by the repo's own `trailing-whitespace` pre-commit hook. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated description by cubic. --> <a href="https://cubic.dev/pr/Unstructured-IO/unstructured/pull/4460?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a> <!-- End of auto-generated description by cubic. --> --------- Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
1 parent e2732cf commit 4fe4097

7 files changed

Lines changed: 89 additions & 4 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,9 @@
1+
## 0.27.5-dev0
2+
3+
### Fixes
4+
5+
- **Attachment elements keep their own filetype.** When partitioning an email with attachments, `auto.partition()` re-stamped `metadata.filetype` on every element with the containing message's type, so content extracted from an attached PDF was labelled `message/rfc822`. Attachment elements now retain the filetype assigned by the nested `partition()` call that produced them, including when the containing document is a file-like object with no known file-name.
6+
17
## 0.27.4
28

39
### Fixes
@@ -8,7 +14,7 @@
814

915
### Fixes
1016

11-
- **Updated README:** Added Sign up Link for Transform MCP in README.
17+
- **Updated README:** Added Sign up Link for Transform MCP in README.
1218

1319
## 0.27.2
1420

‎test_unstructured/partition/test_auto.py‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,7 @@
4848
from unstructured.file_utils.model import FileType, create_file_type, register_partitioner
4949
from unstructured.partition.auto import _PartitionerLoader, partition
5050
from unstructured.partition.common import UnsupportedFileFormatError
51+
from unstructured.partition.common.metadata import is_attachment_element
5152
from unstructured.partition.utils.constants import PartitionStrategy
5253
from unstructured.staging.base import elements_from_json, elements_to_dicts, elements_to_json
5354

@@ -1417,6 +1418,46 @@ def test_auto_partition_applies_the_correct_filetype_for_all_filetypes(
14171418
)
14181419

14191420

1421+
@pytest.mark.parametrize(
1422+
"file_name",
1423+
["eml/fake-email-attachment.eml", "fake-email-attachment.msg"],
1424+
)
1425+
def test_auto_partition_preserves_the_filetype_of_attachment_elements(file_name: str):
1426+
"""Attachment elements keep their own filetype, not the containing message's.
1427+
1428+
Their filetype was assigned by the nested `partition()` call that produced them, so the
1429+
outer call must not re-stamp them with e.g. `message/rfc822`.
1430+
"""
1431+
elements = partition(example_doc_path(file_name), process_attachments=True)
1432+
1433+
attachment_elements = [e for e in elements if is_attachment_element(e)]
1434+
assert attachment_elements
1435+
assert all(e.metadata.filetype == FileType.TXT.mime_type for e in attachment_elements)
1436+
1437+
1438+
@pytest.mark.parametrize(
1439+
"file_name",
1440+
["eml/fake-email-attachment.eml", "fake-email-attachment.msg"],
1441+
)
1442+
def test_auto_partition_preserves_attachment_filetype_when_container_filename_is_unknown(
1443+
file_name: str,
1444+
):
1445+
"""The attachment guard must not depend on `.metadata.attached_to_filename`.
1446+
1447+
That field is `None` when the containing document's file-name is unknown -- partitioning a
1448+
file-like object with no `metadata_filename` -- so keying the guard on it would let the
1449+
containing document's filetype overwrite the attachment's in exactly that case.
1450+
"""
1451+
with open(example_doc_path(file_name), "rb") as f:
1452+
elements = partition(file=f, process_attachments=True)
1453+
1454+
attachment_elements = [e for e in elements if is_attachment_element(e)]
1455+
assert attachment_elements
1456+
# -- precondition: this is the case the `attached_to_filename` guard cannot see --
1457+
assert all(e.metadata.attached_to_filename is None for e in attachment_elements)
1458+
assert all(e.metadata.filetype == FileType.TXT.mime_type for e in attachment_elements)
1459+
1460+
14201461
def test_detect_filetype_maps_file_to_bytes_io_when_spooled_temp_file_used(mocker):
14211462
detect_filetype_mock = MagicMock(return_value=FileType.JSON)
14221463
mocker.patch("unstructured.file_utils.filetype._FileTypeDetector", detect_filetype_mock)

‎unstructured/__version__.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
__version__ = "0.27.4" # pragma: no cover
1+
__version__ = "0.27.5-dev0" # pragma: no cover

‎unstructured/partition/auto.py‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@
1616
from unstructured.partition.common import UnsupportedFileFormatError
1717
from unstructured.partition.common.common import exactly_one
1818
from unstructured.partition.common.lang import check_language_args
19+
from unstructured.partition.common.metadata import is_attachment_element
1920
from unstructured.partition.utils.constants import PartitionStrategy
2021
from unstructured.safe_http import safe_get
2122
from unstructured.telemetry import partition_runtime_telemetry, set_partition_document_type
@@ -201,6 +202,13 @@ def augment_metadata(elements: list[Element]) -> list[Element]:
201202
for element in elements:
202203
element.metadata.url = url
203204
element.metadata.data_source = data_source_metadata
205+
# -- an attachment's elements were assigned their own (correct) filetype by the
206+
# -- nested `partition()` call that produced them; don't re-stamp them with the
207+
# -- containing document's type (e.g. an attached PDF is not `message/rfc822`).
208+
# -- Note this cannot key on `.metadata.attached_to_filename`, which is `None`
209+
# -- whenever the containing document's file-name is unknown. --
210+
if is_attachment_element(element):
211+
continue
204212
if content_type is not None:
205213
out_filetype = FileType.from_mime_type(content_type)
206214
element.metadata.filetype = out_filetype.mime_type if out_filetype else None

‎unstructured/partition/common/metadata.py‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,28 @@
1717

1818
_P = ParamSpec("_P")
1919

20+
# -- Name of a transient, in-process attribute used to identify elements produced by partitioning
21+
# -- an attachment. It is deliberately NOT an `ElementMetadata` field: it must not appear in
22+
# -- dict/JSON output, and it must work when the containing document's file-name is unknown --
23+
# -- which is exactly when `.metadata.attached_to_filename` is `None` and so cannot serve as the
24+
# -- marker (e.g. `partition(file=f)` with no `metadata_filename`).
25+
_ATTACHMENT_ELEMENT_ATTR = "_produced_by_attachment_partitioning"
26+
27+
28+
def mark_as_attachment_element(element: Element) -> None:
29+
"""Record that `element` was produced by partitioning an attachment.
30+
31+
Attachment elements are partitioned by a nested `partition()` call, which assigns them their
32+
own (correct) source-document metadata. Marking them lets the containing document's partitioner
33+
avoid overwriting that metadata with its own.
34+
"""
35+
setattr(element, _ATTACHMENT_ELEMENT_ATTR, True)
36+
37+
38+
def is_attachment_element(element: Element) -> bool:
39+
"""True when `element` was produced by partitioning an attachment."""
40+
return getattr(element, _ATTACHMENT_ELEMENT_ATTR, False) is True
41+
2042

2143
def get_last_modified_date(filename: str) -> str | None:
2244
"""Modification time of file at path `filename`, if it exists.

‎unstructured/partition/email.py‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,10 @@
2121
from unstructured.file_utils.model import FileType
2222
from unstructured.logger import logger
2323
from unstructured.partition.common import EXPECTED_ATTACHMENT_ERRORS
24-
from unstructured.partition.common.metadata import get_last_modified_date
24+
from unstructured.partition.common.metadata import (
25+
get_last_modified_date,
26+
mark_as_attachment_element,
27+
)
2528
from unstructured.partition.html import partition_html
2629
from unstructured.partition.text import partition_text
2730
from unstructured.telemetry import partition_runtime_telemetry
@@ -416,6 +419,7 @@ def _iter_elements(self) -> Iterator[Element]:
416419

417420
for e in elements:
418421
e.metadata.attached_to_filename = self._attached_to_filename
422+
mark_as_attachment_element(e)
419423
yield e
420424

421425
@cached_property

‎unstructured/partition/msg.py‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,10 @@
1313
from unstructured.file_utils.model import FileType
1414
from unstructured.logger import logger
1515
from unstructured.partition.common import EXPECTED_ATTACHMENT_ERRORS
16-
from unstructured.partition.common.metadata import get_last_modified_date
16+
from unstructured.partition.common.metadata import (
17+
get_last_modified_date,
18+
mark_as_attachment_element,
19+
)
1720
from unstructured.partition.html import partition_html
1821
from unstructured.partition.text import partition_text
1922
from unstructured.telemetry import partition_runtime_telemetry
@@ -282,6 +285,7 @@ def _iter_elements(self) -> Iterator[Element]:
282285

283286
for e in elements:
284287
e.metadata.attached_to_filename = self._opts.metadata_file_path
288+
mark_as_attachment_element(e)
285289
yield e
286290

287291
@cached_property

0 commit comments

Comments
 (0)