fix(chunker): only split Markdown tables by columns by CyberRik · Pull Request #932 · docling-project/docling-core

CyberRik · 2026-10-28T10:23:04Z

Fixes #931.

segment() sends any oversized table to _split_table_by_columns (from #690), which rebuilds Markdown sub-tables. But the table text may not be Markdown. With the default (triplet) serializer or HTMLTableSerializer, get_header_and_body_lines finds no Markdown header. The column split still runs, puts a made-up | - | separator in every chunk, wraps the text in | ... | rows, and can leave a chunk that is only pipes.

The change: take the column split only when header_lines has a MARKDOWN_SEPARATOR_ROW_RE row. Other tables go back to the LineBasedTokenChunker path they used before #690. Markdown tables are unchanged.

Before and after, for a 4-row table where one row is too wide for the header (max_tokens=50, HTMLTableSerializer):

# main 5582916
'| <table><tbody> |\n| <tr><th>Region</th><th>Notes</th></tr> |\n| - |\n|'
'| <table><tbody> |\n| <tr><th>Region</th><th>Notes</th></tr> |\n| - |\n <tr><td>North</td><'
...                                                    # 13 chunks, all with the fake "| - |" header
# this PR
'<table><tbody><tr><th>Region</th><th>Notes</th></tr><tr><td>North</td><td>stable</td>'
'<table><tbody><tr><th>Region</th><th>Notes</th></tr></tr><tr><td>South</td><td>growth'
...                                                    # 7 chunks, HTML header repeated, as before #690

The triplet output from #931 now has no | - | rows and no pipes-only chunk.

Not in this PR: with the default serializer, the base get_header_and_body_lines uses split("\n") and drops line endings, so a caption runs into the first row (…by regionregion 1, …). That behavior predates #690 (it dates from #527), so I left it alone. I can send a separate PR if you'd like it changed.

$ python -m pytest tests/test_hybrid_chunker.py tests/test_hierarchical_chunker.py tests/test_line_chunker.py -q   # CPython 3.11
106 passed
# hybrid_chunker.py from main 5582916, tests kept
FAILED tests/test_hybrid_chunker.py::test_column_split_is_skipped_for_non_markdown_tables[triplets]
FAILED tests/test_hybrid_chunker.py::test_column_split_is_skipped_for_non_markdown_tables[html]
$ python -m pytest tests -q --ignore=tests/test_code_chunker.py
# same set of passing tests (808) with and without this change; the 312 failures are local doclang schematron validation, same on main
$ ruff format --check && ruff check   # the two changed files
2 files already formatted
All checks passed!
$ mypy docling_core/transforms/chunker/hybrid_chunker.py
# no errors in hybrid_chunker.py (94 pre-existing errors in other modules, same on main)

# not run: tests/test_code_chunker.py (needs a git clone of external fixtures)

_split_table_by_columns rebuilds Markdown sub-tables, but segment() sent any oversized table to it, whatever the table serializer. For triplet text from the default serializer, or HTML, get_header_and_body_lines finds no Markdown header, so every chunk got a made-up "| - |" separator, the text was wrapped in "| ... |" rows, and the last chunk could be just pipes. The column split now runs only when the header includes a Markdown separator row. Other tables use the line-based split again, as they did before docling-project#690. Fixes docling-project#931 Signed-off-by: Ritankar Mondal <[email protected]>

github-actions · 2026-10-28T10:23:15Z

✅ DCO Check Passed

Thanks @CyberRik, all your commits are properly signed off. 🎉

mergify · 2026-10-28T10:23:42Z

Merge Protections

🔴 1 of 2 protections blocking · waiting on 👀 reviews

	Protection	Waiting on
🔴	Require two reviewer for test updates	👀 reviews
🟢	Enforce conventional commit	—

🔴 Require two reviewer for test updates

Waiting for

#approved-reviews-by >= 2

This rule is failing.

When test data is updated, we require two reviewers

#approved-reviews-by >= 2

Show 1 satisfied protection

🟢 Enforce conventional commit

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert)(?:\(.+\))?(!)?:

github-actions · 2026-11-03T16:22:24Z

AI triage (advisory)

Checked commit 6f6aa9d338. The analysis read the diff only. It did not run code from this pull request.

The PR limits the column split in HybridChunker.segment() to Markdown tables. A table counts as Markdown only when its header lines include a separator row. Other tables, such as triplet or HTML text, use the line-based chunker again. The PR adds a regression test.

Topic labels added: bug, chunking.

CI safety: no concern found. A maintainer must still approve CI.

2 related pull request(s) in the same code

PR	State	Signal	Assessment
#690	merged	referenced as #690	related: PR 690 is merged and added the column split in `_split_table_by_columns`. This PR does not redo that work. It adds a guard so the split runs only for Markdown tables.
#527	merged	referenced as #527	related: PR 527 is merged and added table-aware chunking in `segment()`. This PR changes a different part: the column-split condition added later.

_{The ai:* labels are advisory. They do not approve CI and they do not add tests:full. A new push runs this check again.}

github-actions

AI first review (advisory): no blocker found

This review covers the full diff at 6f6aa9d338. It read the code and did not run it.

I found no defect. The column split now runs only when header_lines has a Markdown separator row. Markdown tables keep the old path, and triplet and HTML tables use LineBasedTokenChunker as they did before #690.

_{A maintainer makes the merge decision. This review never approves the pull request.}

codecov · 2026-11-06T16:12:35Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

github-actions Bot added ai:ci-safe AI triage found no CI-safety concern (advisory) bug Something isn't working chunking Document chunking labels Nov 3, 2026

github-actions Bot reviewed Nov 3, 2026

View reviewed changes

github-actions Bot added the ai:review-lgtm AI first review found no blocker or major issue (advisory) label Nov 3, 2026

flysheep-ai mentioned this pull request Nov 6, 2026

LineBasedTokenChunker does not terminate when a mandatory prefix leaves no room for content #987

Open

Navigation Menu

Repository navigation

fix(chunker): only split Markdown tables by columns - #932

fix(chunker): only split Markdown tables by columns#932
CyberRik wants to merge 1 commit into
docling-project:mainfrom
CyberRik:fix/chunker-column-split-non-markdown

CyberRik commented Oct 28, 2026

Uh oh!

github-actions Bot commented Oct 28, 2026

Uh oh!

mergify Bot commented Oct 28, 2026

🟢 Enforce conventional commit

Uh oh!

github-actions Bot commented Nov 3, 2026

Uh oh!

github-actions Bot left a comment

Uh oh!

codecov Bot commented Nov 6, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

CyberRik commented Oct 28, 2026

Uh oh!

github-actions Bot commented Oct 28, 2026

Uh oh!

mergify Bot commented Oct 28, 2026

Merge Protections

🔴 Require two reviewer for test updates

🟢 Enforce conventional commit

Uh oh!

github-actions Bot commented Nov 3, 2026

AI triage (advisory)

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

AI first review (advisory): no blocker found

Uh oh!

codecov Bot commented Nov 6, 2026

Codecov Report

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant