Skip to content

One invalid document blocks every collection read, with no recovery path #127

Open

Description

@cdwilson

One invalid document blocks every collection read, with no recovery path.

What happens

A document written by another tool (or by an older schema) that fails validation makes every collection read raise, while single-document and count paths keep working:

db.table("b").insert({"n": "not-an-int"})  # written by another tool

B.all()                   # ValidationError
B.find().limit(1).all()   # ValidationError
B.count()                 # 6      -- works, no validation
B.get_by_id(1)            # works  -- that document is fine

Verified at HEAD: all() and the chain terminals raise ValidationError; count() returns 6; get_by_id(1) returns the good document.

Why it matters

Validation-on-read is a genuine feature and the docs sell it well. The gap is that there is no supported way to read around a bad document — no skip_invalid=, no "give me the errors and the good rows", no diagnostic that says which document ids fail and why. The schema-evolution page's advice is to drop to the raw TinyDB table, which works, but means leaving the library at exactly the moment the library detected the problem.

find().limit(1).all() failing is the sharpest form: the caller asked for one document and got an error about a different one.

Options

  • Model.all(on_error="raise" | "skip" | "collect") — raise stays the default; skip yields the good rows; collect returns the good rows plus the failures.
  • Model.validate_table() — a diagnostic that reports which document ids fail and why, without changing any read path. Cheaper, and it is the thing a user actually needs first: what is broken?

The two compose; the diagnostic is the smaller starting point.

Reproduction

from tinydb import TinyDB
from tinydb.storages import MemoryStorage

from tinydantic import TinydanticModel

db = TinyDB(storage=MemoryStorage)


class Item(TinydanticModel, database=db):
    n: int


Item(n=1).insert()
db.table("item").insert({"n": "not-an-int"})  # written by another tool

print("count():   ", Item.count())
print("get_by_id():", Item.get_by_id(1))
Item.all()  # raises

Run with python repro.py.

Actual output:

count():    2
get_by_id(): id=1 n=1
Traceback (most recent call last):
  File "repro.py", line 18, in <module>
    Item.all()  # raises
  File ".../tinydantic/_model.py", line 1162, in all
    return [cls.from_tinydb_document(doc) for doc in iter(cls.get_table())]
  ...
pydantic_core._pydantic_core.ValidationError: 1 validation error for Item
n
  Input should be a valid integer, unable to parse string as an integer [type=int_parsing, input_value='not-an-int', input_type=str]

find().limit(1).all() fails the same way — the caller asks for one document and gets an error about a different one.

Expected: raising stays the default, but there is a supported way to read around the bad document and a diagnostic that names which ids fail and why — Model.validate_table() being the smaller starting point.

Environment: tinydantic 0.4.0 at 4d373d1, Python 3.10.20, TinyDB 4.9.0, pydantic 2.13.4, macOS. Run 2026-08-13.


Reproduced at e7d0ed0 on 2026-08-12. Originally finding L3 of the 2026-08-06 internal review.

Assisted-by: Claude:claude-opus-5

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    ai-assistedCreated or investigated with help from an AI agentapi-designShape of the public surfaceenhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions