Skip to content

refactor title.py - #2657

Merged
scanny merged 7 commits into
mainfrom
jj/rfctr_chunk_title
Mar 19, 2024
Merged

scanny merged 7 commits into
mainfrom
jj/rfctr_chunk_title

Conversation

@Coniferish

@Coniferish Coniferish commented Mar 17, 2024 •

Copy link
Copy Markdown
Contributor

Minor refactor after conversation with @scanny

Updates docstring and how chunking options are accessed. self._kwargs.get() should only be used in the lazyproperty definition of an instance's attribute. Other calls should use self.<attribute>

@scanny scanny left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, LGTM :)

@scanny scanny left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, LGTM. Some small polishing items to fix up before merge but approving in advance.

Comment thread unstructured/chunking/title.py Resolved
@Coniferish
Coniferish requested a review from scanny March 19, 2024 17:45
@scanny
scanny added this pull request to the merge queue Mar 19, 2024
Merged via the queue into main with commit 9ac4445 Mar 19, 2024
@scanny
scanny deleted the jj/rfctr_chunk_title branch March 19, 2024 18:23
kaaloo pushed a commit to inclusif/unstructured that referenced this pull request Apr 8, 2024
Minor refactor after conversation with @scanny

Updates docstring and how chunking options are accessed.
`self._kwargs.get()` should only be used in the `lazyproperty`
definition of an instance's attribute. Other calls should use
`self.<attribute>`
kaaloo pushed a commit to inclusif/unstructured that referenced this pull request Apr 8, 2024
Minor refactor after conversation with @scanny

Updates docstring and how chunking options are accessed.
`self._kwargs.get()` should only be used in the `lazyproperty`
definition of an instance's attribute. Other calls should use
`self.<attribute>`
abo3losh1 added a commit to abo3losh1/unstructured that referenced this pull request Sep 22, 2026
`combine_text_under_n_chars` is documented as capped at `new_after_n_chars`:
a combine-threshold above the soft-max recombines pre-chunks the soft-max
has already declared full, so the caller gets chunks well past the size
they asked for.

The cap was implemented until the `title.py` refactor in 9ac4445 (Unstructured-IO#2657),
which replaced

    return soft_max if combine_text_under_n_chars > soft_max else combine_text_under_n_chars

with a plain read of the argument, while leaving both the property's own
docstring ("Is reduced to `new_after_n_chars` when it exceeds that value")
and the public option documentation in `base.py` ("Note this value is
'capped' at the `new_after_n_chars` value") describing the old behavior.

    chunk_by_title(
        elements, max_characters=500, new_after_n_chars=100,
        combine_text_under_n_chars=400,
    )

returned one 367-character chunk where three 121-character chunks were
asked for.

Restore the cap. Validation now reads the argument as the caller gave it
rather than the capped property, so a negative value and a value greater
than `max_characters` are both still rejected with the same messages; the
cap would otherwise have hidden the second case.

This branch was previously deployed

1 inactive deployment
ci — 71f09035 Deployed Mar 19, 2024 by Coniferish
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants