Skip to content

fix(lang): treat an empty language selection as automatic - #4704

Merged
cragwolfe merged 2 commits into
Unstructured-IO:mainfrom
Afloat16:Afloat16/empty-language-auto-detection-20261112
Nov 12, 2026
Merged

cragwolfe merged 2 commits into
Unstructured-IO:mainfrom
Afloat16:Afloat16/empty-language-auto-detection-20261112

Conversation

@Afloat16

@Afloat16 Afloat16 commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Passing languages=[] to language detection or metadata application raises IndexError at languages[0], before either empty-text handling or automatic detection can run. An empty selection is already treated as a default by check_language_args and by the later automatic-detection branch.

Normalize an empty language list to the existing automatic mode, just like None. This also uses the normal short-ASCII fallback. Explicit languages, the empty-string skip sentinel, and validation of non-list inputs retain their existing behavior.

Validation

  • Added 18 native cases: 11 fail and 7 pass on the unchanged base.
  • New and existing language tests: 83 passed, 4 skipped. The skips are optional backend tests in the existing suite.
  • Covers actual langdetect output for English/French/Korean, empty/whitespace text, short-text fallback, list/iterator metadata application in whole-document and per-element modes, unchanged input lists, and existing explicit/invalid language options.
  • Ruff 0.15.10 check/format and git diff --check pass. The full project test suite was not run.

Checklist

  • Added regression tests and documented empty-list behavior.
  • Preserved explicit language selection and invalid-input checks.
  • Apply the bug label.

View guided diff

@cragwolfe cragwolfe left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the language-selection change and its partition and metadata callers. I found no actionable correctness or compatibility issues. The existing code treats an empty selection as a default, while [''] is the explicit skip value in the language metadata helper. The change makes direct empty-list calls follow the existing automatic path.

This was a static review; I did not run tests. The CI workflows for this commit currently require approval.

(authored by codex)

@cragwolfe
cragwolfe enabled auto-merge November 12, 2026 20:12
@cragwolfe
cragwolfe added this pull request to the merge queue Nov 12, 2026
Merged via the queue into Unstructured-IO:main with commit 5df7ea2 Nov 12, 2026
51 of 52 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants