feat: custom fallback for language detection by claytonlin1110 · Pull Request #4238 · Unstructured-IO/unstructured

claytonlin1110 · 2026-02-12T15:46:29Z

Implements custom fallback for language detection so short text is not forced to English and callers can control or disable detection.

Changes:

language_fallback
Optional callable used when text is short (<5 words) and ASCII. It receives the text and can return a list of ISO 639-3 codes or None to leave language unspecified. If not provided, short text still defaults to ["eng"] (backward compatible).
detect_languages() / apply_lang_metadata()
New parameter language_fallback; applied in the short-text path only.
partition() (auto)
New parameter language_fallback; passed through to all partitioners via the metadata decorator.
partition_md()
New parameter languages so callers can pass languages=[""] to disable language detection (aligned with other partitioners).

Usage:

Return None for short text: partition(..., language_fallback=lambda text: None)
Custom short-text language: partition(..., language_fallback=my_detector)
Disable detection: partition_md(..., languages=[""]) or partition(..., languages=[""])

claytonlin1110 · 2026-02-12T15:48:42Z

@badGarnet Would you please review this since this is my first contribution here.

claytonlin1110 · 2026-02-18T04:22:42Z

@badGarnet Sorry for pinging you since I'm a new contributor... Please review

claytonlin1110 · 2026-02-19T01:30:15Z

@PastelStorm I'm a new contributor here i don't know who i need to ping to get this PR reviewed.

PastelStorm · 2026-02-19T04:15:14Z

@PastelStorm I'm a new contributor here i don't know who i need to ping to get this PR reviewed.

Would you mind making a feature branch instead of forking the repo please? Then I will review. Thank you.

claytonlin1110 · 2026-02-19T10:41:02Z

@PastelStorm Forking is the normal workflow on public git repo to contribute.

claytonlin1110 · 2026-02-19T10:49:49Z

@PastelStorm Do you want to set the target branch "feature" rather than "main"?

claytonlin1110 · 2026-02-19T10:53:31Z

@PastelStorm Please review as I'm an external contributor.

PastelStorm · 2026-02-19T17:04:25Z

@PastelStorm Forking is the normal workflow on public git repo to contribute.

I apologize, you're correct here. I'll review your PR in a moment. Thank you for the contribution :)

PastelStorm

There is no integration test verifying that language_fallback flows correctly through the full partition() call chain. The parameter passes through three layers of **kwargs unpacking (partition() -> partition_md() -> partition_html() -> @apply_metadata -> apply_lang_metadata() -> detect_languages()). If any intermediate function changes its signature to capture this kwarg explicitly, the feature silently breaks. An integration test like partition(filename=..., language_fallback=lambda t: None) asserting element.metadata.languages is None would guard against this.

PastelStorm · 2026-02-19T17:39:49Z

unstructured/partition/common/lang.py

+        if language_fallback is not None:
+            result = language_fallback(text)
+            return result
        logger.debug(f'short text: "{text}". Defaulting to English.')


Two issues here:

Unnecessary temp variable. result = language_fallback(text); return result can be simplified to return language_fallback(text).

No validation of the fallback return value. The normal detection path validates language codes through _get_iso639_language_object(), but the fallback bypasses all validation. A buggy callback returning ["not_a_language"], [42], or "" would produce invalid metadata that passes silently. Consider at minimum a type/value check, or document explicitly that the caller is responsible for returning valid ISO 639-3 codes.

PastelStorm · 2026-02-19T17:40:42Z

unstructured/partition/common/lang.py

    """Detect language and apply it to metadata.languages for each element in `elements`.
    If languages is None, default to auto detection.
-    If languages is and empty string, skip."""
+    If languages is and empty string, skip.


typo: and should be an

PastelStorm · 2026-02-19T17:41:45Z

unstructured/partition/auto.py


    partitioning_kwargs = copy.deepcopy(kwargs)
    partitioning_kwargs["detect_language_per_element"] = detect_language_per_element
+    partitioning_kwargs["language_fallback"] = language_fallback


language_fallback is added to partitioning_kwargs (used for "all other file types"), but it is not passed to the PDF/image code paths (lines 215-253). This means partition(filename="doc.pdf", language_fallback=my_fn) silently ignores the callback. If intentional, the docstring for language_fallback (lines 102-105) should mention this limitation.

PastelStorm · 2026-02-19T17:43:00Z

unstructured/partition/auto.py

+                Optional callable for short text (e.g. when detection defaults to English).
+                Called with the text; return a list of ISO 639-3 codes or None to leave
+                language unspecified.
    pdf_infer_table_structure


The language_fallback docstring is indented as a sub-parameter of languages, but it's actually an independent top-level parameter. This is a pre-existing style issue (from detect_language_per_element) that this PR perpetuates.

PastelStorm · 2026-02-19T17:44:17Z

unstructured/partition/md.py

    url: str | None = None,
    metadata_filename: str | None = None,
    metadata_last_modified: str | None = None,
+    languages: Optional[list[str]] = None,


Inconsistent parameter exposure. languages is surfaced explicitly, but detect_language_per_element and language_fallback (which also flow through **kwargs to partition_html -> @apply_metadata) remain implicit. If the goal is to make language-related parameters discoverable, they should be treated consistently.

PastelStorm · 2026-02-19T17:45:24Z

test_unstructured/partition/common/test_lang.py

+def test_detect_languages_short_text_fallback_returns_custom():
+    """Short ASCII text with language_fallback returns custom language."""
+    result = detect_languages(
+        text="Bonjour monde.",
+        language_fallback=lambda t: ["fra"],
+    )
+    assert result == ["fra"]


Slightly misleading test. "Bonjour monde." is pure ASCII and has < 5 words, so the fallback is invoked purely based on length/charset, not because the text is French. The hardcoded ["fra"] return is what's being asserted. The test would behave identically with text="Hello world.". Consider using neutral text or adding a clarifying comment.

claytonlin1110 · 2026-02-19T19:39:49Z

@PastelStorm Fixed everything.

PastelStorm · 2026-02-19T21:37:55Z

@claytonlin1110 you need to bump the project version and update the changelog file.

claytonlin1110 · 2026-02-19T21:43:33Z

@claytonlin1110 you need to bump the project version and update the changelog file.

Just added and pushed

claytonlin1110 · 2026-02-19T22:11:24Z

@PastelStorm Feel free to review

claytonlin1110 · 2026-02-19T22:34:01Z

@PastelStorm Would you help me to fix lint issue? Would be great if you can guide where to see the lint error and fix it.

PastelStorm · 2026-02-19T22:51:05Z

@PastelStorm Would you help me to fix lint issue? Would be great if you can guide where to see the lint error and fix it.

Take a look inside Makefile, you will find all sorts of commands there that will help you debug your code locally. For example to run a lint check you can execute either make check or make check ruff from the project root.

claytonlin1110 · 2026-02-19T23:06:13Z

@PastelStorm Done

PastelStorm · 2026-02-19T23:21:14Z

@PastelStorm Done

Running the CI, hold on :)

PastelStorm · 2026-02-19T23:33:46Z

@claytonlin1110 you might want to run the tests locally, so we have less of this back-and-forth. These tests will fail locally too for example:
https://github.com/Unstructured-IO/unstructured/actions/runs/22203791837/job/64224646667?pr=4238

claytonlin1110 · 2026-02-19T23:34:17Z

@claytonlin1110 you might want to run the tests locally, so we have less of this back-and-forth. These tests will fail locally too for example: https://github.com/Unstructured-IO/unstructured/actions/runs/22203791837/job/64224646667?pr=4238

By make test?

claytonlin1110 · 2026-02-19T23:40:09Z

@PastelStorm would you help fixing test?

PastelStorm · 2026-02-19T23:42:08Z

@PastelStorm would you help fixing test?

I would encourage you to do the research and fix it yourself. This way you will also learn how the project works under the hood.

claytonlin1110 · 2026-02-20T00:36:55Z

@PastelStorm make test correct?

claytonlin1110 · 2026-02-20T02:02:40Z

@PastelStorm let me know the test workflow...
Even I'm on main branch, make test failed

claytonlin1110 · 2026-02-20T18:03:33Z

@PastelStorm Could you please review again?

…tition_md

…on partition_html/partition_md

claytonlin1110 · 2026-02-23T03:26:12Z

@PastelStorm Everything is passed, now just one issue with test_ingest_src.
Any suggestion to fix this?

PastelStorm · 2026-02-23T03:44:42Z

@PastelStorm Everything is passed, now just one issue with test_ingest_src. Any suggestion to fix this?

You needed to regenerate the fixtures but I think my colleagues already did it in #4256 . I started a new CI workflow, let's see if it passes.

There are differences from the previously checked-in structured outputs.

If these differences are acceptable, overwrite by the fixtures by setting the env var:

  export OVERWRITE_FIXTURES=true

and then rerun this script.

NOTE: You'll likely just want to run scripts/ingest-test-fixtures-update.sh on x86_64 hardware
to update fixtures for CI.

https://github.com/Unstructured-IO/unstructured/actions/runs/22235316041/job/64373485696#step:6:9156

PastelStorm · 2026-02-23T03:51:31Z

@claytonlin1110 same issue, you need to regenerate the fixtures first, then you can run the tests:
https://github.com/Unstructured-IO/unstructured/actions/runs/22291662617/job/64481268558?pr=4238#step:6:9182

claytonlin1110 · 2026-02-23T04:41:32Z

@PastelStorm Please review again

claytonlin1110 · 2026-02-23T04:52:03Z

@PastelStorm FYI, in UDHR_first_article_all.txt.json file, languages, modes and path values are changed automatically when I run the command. But i ignored path values since it's just replacing my local project path, so i just pushed languages and modes changes only. Let me know if it's okay.

claytonlin1110 · 2026-02-23T04:59:13Z

@PastelStorm Seems like mode values shouldnt be changed. I just pushed, please re-run CI. Thanks.

claytonlin1110 · 2026-02-23T05:31:38Z

@PastelStorm All CI passed, thanks for your support.

claytonlin1110 force-pushed the feat/language-detection-custom-fallback branch from 1d0c2bc to 8fcbf39 Compare February 19, 2026 10:41

PastelStorm requested changes Feb 19, 2026

View reviewed changes

claytonlin1110 force-pushed the feat/language-detection-custom-fallback branch from 267533b to 9da0547 Compare February 19, 2026 19:39

claytonlin1110 requested a review from PastelStorm February 19, 2026 19:39

claytonlin1110 force-pushed the feat/language-detection-custom-fallback branch 2 times, most recently from adecfb8 to 7507698 Compare February 19, 2026 21:42

claytonlin1110 force-pushed the feat/language-detection-custom-fallback branch from 1ce845c to cac7c0f Compare February 19, 2026 22:08

claytonlin1110 added 7 commits February 22, 2026 21:14

feat: add language_fallback for short text and languages param to par…

5e977ae

…tition_md

fix: add language_fallback for short text and expose language params …

740f2f6

…on partition_html/partition_md

fix: test

bfbc9e8

fix: add change log

2aadbe8

lint fix

dd174d9

fix: lint

e765cda

fix: test

dc445a9

claytonlin1110 force-pushed the feat/language-detection-custom-fallback branch from b2ba912 to dc445a9 Compare February 23, 2026 03:16

fix: fixture

58aef85

fix: fixture

f45251a

PastelStorm approved these changes Feb 23, 2026

View reviewed changes

PastelStorm added this pull request to the merge queue Feb 23, 2026

Merged via the queue into Unstructured-IO:main with commit afbda95 Feb 23, 2026
51 checks passed

Conversation

claytonlin1110 commented Feb 12, 2026

Changes:

Usage:

Uh oh!

claytonlin1110 commented Feb 12, 2026

Uh oh!

claytonlin1110 commented Feb 18, 2026

Uh oh!

claytonlin1110 commented Feb 19, 2026

Uh oh!

PastelStorm commented Feb 19, 2026

Uh oh!

claytonlin1110 commented Feb 19, 2026

Uh oh!

claytonlin1110 commented Feb 19, 2026

Uh oh!

claytonlin1110 commented Feb 19, 2026

Uh oh!

PastelStorm commented Feb 19, 2026

Uh oh!

PastelStorm left a comment

Choose a reason for hiding this comment

Uh oh!

PastelStorm Feb 19, 2026

Choose a reason for hiding this comment

Uh oh!

PastelStorm Feb 19, 2026

Choose a reason for hiding this comment

Uh oh!

PastelStorm Feb 19, 2026

Choose a reason for hiding this comment

Uh oh!

PastelStorm Feb 19, 2026

Choose a reason for hiding this comment

Uh oh!

PastelStorm Feb 19, 2026

Choose a reason for hiding this comment

Uh oh!

PastelStorm Feb 19, 2026

Choose a reason for hiding this comment

Uh oh!

claytonlin1110 commented Feb 19, 2026

Uh oh!

PastelStorm commented Feb 19, 2026

Uh oh!

claytonlin1110 commented Feb 19, 2026

Uh oh!

claytonlin1110 commented Feb 19, 2026

Uh oh!

claytonlin1110 commented Feb 19, 2026

Uh oh!

PastelStorm commented Feb 19, 2026

Uh oh!

claytonlin1110 commented Feb 19, 2026

Uh oh!

PastelStorm commented Feb 19, 2026

Uh oh!

PastelStorm commented Feb 19, 2026

Uh oh!

claytonlin1110 commented Feb 19, 2026

Uh oh!

claytonlin1110 commented Feb 19, 2026

Uh oh!

PastelStorm commented Feb 19, 2026

Uh oh!

claytonlin1110 commented Feb 20, 2026

Uh oh!

claytonlin1110 commented Feb 20, 2026

Uh oh!

claytonlin1110 commented Feb 20, 2026

Uh oh!

claytonlin1110 commented Feb 23, 2026

Uh oh!

PastelStorm commented Feb 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

PastelStorm commented Feb 23, 2026

Uh oh!

claytonlin1110 commented Feb 23, 2026

PastelStorm commented Feb 23, 2026 •

edited

Loading