Description

Document.tokenized_text cached its first tokenization, but text is a plain public field, so assigning new text kept the old tokens. Extraction then prompted with the old text and aligned against it, producing MATCH_EXACT intervals that point at the wrong span of the returned text.

The getter now compares the cached tokenization's text with the current text and tokenizes again when they differ. The same check is applied to AnnotatedDocument. A tokenization set explicitly for the current text is still kept.

Fixes #580

Bug fix

How Has This Been Tested?

Added three tests to tests/data_lib_test.py: a Document and an AnnotatedDocument re-tokenize after their text changes, and a custom tokenization for unchanged text is kept. The first two fail without the fix. The issue's offline reproduction now extracts Bob with a correct interval.

$ pytest tests
984 passed, 5 failed (create_provider_plugin_test.py, also failing on main in my environment: they need pip network access)

Checklist:

• I have read and acknowledged Google's Open Source
  Code of conduct.
• I have read the
  Contributing
  page, and I either signed the Google
  Individual CLA
  or am covered by my company's
  Corporate CLA.
• I have discussed my proposed solution with code owners in the linked
  issue(s) and we have agreed upon the general approach.
• I have made any needed documentation changes, or noted in the linked
  issue(s) that documentation elsewhere needs updating.
• I have added tests, or I have ensured existing tests cover the changes
• I have followed
  Google's Python Style Guide
  and ran pylint over the affected code.