fix(chunking): cap combine_text_under_n_chars at new_after_n_chars - #4487

…4501)

No open issue covers this. None. Found while reading the three sibling
validators in `_validate()` against the parameters they guard.

`ChunkingOptions._validate()` at `unstructured/chunking/base.py` rejects
a negative `new_after_n_chars` and a negative `new_after_n_tokens`. It
never checks `overlap`. A negative overlap then corrupts output two
ways, with no error and no warning.

`base.py:2731` computes the remainder as `s[maxlen - self._opts.overlap
:]`. With `overlap = -5` that remainder starts five characters past the
end of the fragment, so five characters of the document disappear at
every split boundary.

`base.py:755` computes `self._text[-overlap:]`. With `overlap = -5` that
is a forward slice, so the tail carried into the next chunk is most of
the current chunk.

Before, on a 60-character input of unique characters at
`max_characters=20`:

```
$ python repro.py
overlap=   0: ['ABCDEFGHIJKLMNOPQRST', 'UVWXYZabcdefghijklmn', 'opqrstuvwxyz01234567']
           dropped from output entirely: ''
overlap=  -5: ['ABCDEFGHIJKLMNOPQRST', 'Zabcdefghijklmnopqrs', 'yz01234567']
           dropped from output entirely: 'UVWXYtuvwx'
```

The `overlap=0` line is the control: same input, same commit, nothing
lost.

With `overlap_all=True`, three 10-character elements at
`max_characters=12`:

```
overlap=  0 -> ['AAAAAAAAAA', 'BBBBBBBBBB', 'CCCCCCCCCC']
overlap= -4 -> ['AAAAAAAAAA', 'AAAAAA\n\nBBBB', 'BB', 'AA\n\nBBBBBBBB', 'CCCCCCCCCC']
```

Text duplicated and out of order.

Reachable from all three public entry points: `chunk_elements(...)`,
`chunk_by_title(...)`, and `partition(filename=...,
chunking_strategy="basic", overlap=-5)`. All three drop the same
characters.

After, the four added lines mirror the existing `new_after_n_chars`
check and raise `ValueError`.

```
$ python -m pytest test_unstructured/chunking -q
484 passed, 24 skipped in 15.04s
```

Baseline on `1bedf7b` is `482 passed, 24 skipped`, so the delta is the
two new parametrised cases. Reverting `unstructured/chunking/base.py`
and keeping the test gives `2 failed, 482 passed, 24 skipped`, both of
them `Failed: DID NOT RAISE ValueError`. `ruff check` is clean on both
touched files.

I did not run `make test` or `make check` in full, and no PDF, image or
OCR test ran here: the install was `-e ".[csv,docx,md,pptx,xlsx]"`
without the inference extras. `make check-version` passes, with
`CHANGELOG.md` and `__version__.py` both at `0.27.10`.

Duplicate check: all 130 open pull requests enumerated, 10 touching
`unstructured/chunking/`. I grepped each diff for added lines matching
`_validate|overlap.*must be|overlap_arg`. The only hit is #4487, which
caps `combine_text_under_n_chars` in `title.py::_validate`: different
file, different function, different parameter. #4392 is the nearest
neighbour and changes `_split_by_tokens` rather than validation, so the
two do not collide.

@cragwolfe you merged the last chunking change here. Would you rather
this raised, or clamped a negative overlap to zero with a warning?


<!-- This is an auto-generated description by cubic. -->
<a
href="https://cubic.dev/pr/Unstructured-IO/unstructured/pull/4501?utm_source=github"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img
alt="Review in cubic"
src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a>
<!-- End of auto-generated description by cubic. -->

…s set (#4528)

`PreChunk.can_combine()` sizes its text with `len()`, so the
`combine_text_under_n_chars` threshold and the hard-max check count
characters even when `max_tokens` has selected token counting. Every
sibling computation in the class — `will_fit`, `_text_length` — goes
through `ChunkingOptions.measure()`, which branches on the counting
mode; `can_combine` is the one that doesn't. In token mode a section of
a few tokens is many more characters than the token budget, so every
section looks too large to combine and becomes its own chunk. With
`max_tokens=60`, `chunk_elements` fills the window while
`chunk_by_title` produces twelve ten-token chunks — about 17%
utilization — and the obvious workaround,
`combine_text_under_n_chars=240`, is rejected because it exceeds the
hard max.

The fix is the two `measure()` substitutions. Character-mode behavior is
unchanged (`measure()` is `len()` there): a 3,000-chunk-list corpus
across both chunkers and five option sets hashes identically before and
after, and the existing character-mode `can_combine` cases pass
untouched.

Tests: a deterministic offline token counter fixture (no tiktoken
download), a token-mode `can_combine` parametrization mirroring the
character-mode sibling, a `chunk_by_title` case asserting the token
budget is filled and the chunk texts equal `chunk_elements()` on the
same input, and a discriminating character-mode case that would flip if
the measure were swapped. The first two fail on `main`.
`test_unstructured/chunking/test_base.py` + `test_title.py`: 391 passed,
16 skipped; ruff clean. CHANGELOG entry under a new `0.27.18` section
with the matching `__version__` bump, following the one-patch-per-PR
pattern of the recent entries.

Related but different: #4487 changes what the
`combine_text_under_n_chars` threshold *is* (caps it at
`new_after_n_chars`); this changes which *unit* it's compared in. No
textual overlap.

<!-- This is an auto-generated description by cubic. -->
<a
href="https://cubic.dev/pr/Unstructured-IO/unstructured/pull/4528?utm_source=github"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img
alt="Review in cubic"
src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a>
<!-- End of auto-generated description by cubic. -->