Skip to content

Commit 02365b5

Browse files
committed
docs(query): correct matches() semantics and query docstrings
- Q1 GuardedCondition.__and__/__or__ said "Compose with and/or". They implement & and |, and this file's own error message warns that and/or silently discard half the query — the reference was teaching the exact mistake the guard exists to catch. - Q2 matches() was documented as a whole-value regex test. TinyDB implements it with re.match, which anchors at the start only, so `.*@example\.com` matches '[email protected]'. The error is inherited from TinyDB's own docstring and was republished on docs/usage/queries.md (D-QRY-3), where a runnable example now shows both the trap and the `$` fix. - Q3 the module docstring listed q() among the constructors of guarded queries. q() is an identity cast — handed a raw tinydb.where() it returns it unguarded. - Q4 "an "key is present"" -> "a". Per the two-artifact rule, matches() gets an upstream_limitations section and tracking issue #144. Nothing was filed upstream. The same page also had two TinyDB API names wrong: - D-CONTRIB-2 the batched-write section called the method update_many(). TinyDB has no update_many() — it is Table.update_multiple(); update_many() is tinydantic's own wrapper. The page is written to be read standalone by an upstream maintainer, who would find no such method. Issue #134's title and body carried the same conflation and were corrected. - D-CONTRIB-3 "Only Table.__iter__ yields Document instances" — search(), all(), and get() all return Documents in TinyDB 4.9.0. The load-bearing claim is that conditions are never *evaluated* against one, which the paragraph already states correctly. Assisted-by: Claude:claude-opus-5
1 parent d5bf5a1 commit 02365b5

5 files changed

Lines changed: 59 additions & 16 deletions

File tree

‎docs/contributing/upstream_limitations.md‎

Lines changed: 20 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -22,30 +22,42 @@ All observations in this section were verified against TinyDB 4.9.0 (the minimum
2222

2323
**Tracking issue:** [#133](https://github.com/tinydantic/tinydantic/issues/133)
2424

25-
**Limitation.** TinyDB evaluates every query condition against the raw document _body_ mapping — the value in the `{doc_id: body}` table dict — never against a [`Document`](https://tinydb.readthedocs.io/en/latest/api.html#tinydb.table.Document) carrying `doc_id`. All five evaluation sites behave this way: `Table.search()`, `Table.get(cond=...)`, and the updater loops inside `Table.update(cond=...)`, `Table.update_multiple()`, and `Table.remove(cond=...)`. (`Table._update_table()`'s docstring notes that skipping the `Document` wrap is a deliberate optimization.) Only `Table.__iter__` yields `Document` instances. Consequently, no query object — however it is constructed — can express "document id equals 1": the id is structurally invisible to the condition, even though it sits right there as the dict key in every evaluation loop.
25+
**Limitation.** TinyDB evaluates every query condition against the raw document _body_ mapping — the value in the `{doc_id: body}` table dict — never against a [`Document`](https://tinydb.readthedocs.io/en/latest/api.html#tinydb.table.Document) carrying `doc_id`. All five evaluation sites behave this way: `Table.search()`, `Table.get(cond=...)`, and the updater loops inside `Table.update(cond=...)`, `Table.update_multiple()`, and `Table.remove(cond=...)`. (`Table._update_table()`'s docstring notes that skipping the `Document` wrap is a deliberate optimization.) `Table.__iter__`, `search()`, `all()`, and `get()` all _return_ `Document` instances — but none of them evaluate the condition against one. Consequently, no query object — however it is constructed — can express "document id equals 1": the id is structurally invisible to the condition, even though it sits right there as the dict key in every evaluation loop.
2626

27-
**Why it matters to an ODM.** Every mainstream document ODM lets users query by the model's id attribute (Beanie and ODMantic translate `Model.id` to MongoDB's `_id`; Firestore special-cases `FieldPath.documentId()` because — exactly like TinyDB — its document key is not a body field). `tinydantic` maps its `id` field to `doc_id`, so supporting `User.get(User.id == 1)` required a translation layer: id-bearing conditions are detected in every model method and executed via `doc_id=`/`doc_ids=` operations or by iterating the table (the one API that yields `Document`s). See `src/tinydantic/_query.py` and the id-condition branches in `src/tinydantic/_model.py`.
27+
**Why it matters to an ODM.** Every mainstream document ODM lets users query by the model's id attribute (Beanie and ODMantic translate `Model.id` to MongoDB's `_id`; Firestore special-cases `FieldPath.documentId()` because — exactly like TinyDB — its document key is not a body field). `tinydantic` maps its `id` field to `doc_id`, so supporting `User.get(User.id == 1)` required a translation layer: id-bearing conditions are detected in every model method and executed via `Table.update(doc_ids=…)`/`Table.remove(doc_ids=…)` operations or by iterating the table (the one API that hands the condition a `Document`). See `src/tinydantic/_query.py` and the id-condition branches in `src/tinydantic/_model.py`.
2828

2929
**Suggested improvement.** An opt-in protocol that keeps the existing fast path free:
3030

3131
1. Give `QueryInstance` a `needs_doc_id: bool = False` attribute, propagated through `__and__`/`__or__`/`__invert__` (`self.needs_doc_id or other.needs_doc_id`).
3232
2. Add a `DocId` query type to `tinydb.queries` whose comparisons build `QueryInstance`s with `needs_doc_id=True` and test `doc.doc_id`.
3333
3. At each of the five evaluation sites: `if getattr(cond, "needs_doc_id", False)`, wrap the body in `self.document_class(doc, self.document_id_class(doc_id))` before calling the condition; otherwise call it with the raw body exactly as today.
3434

35-
Existing queries and third-party `QueryLike` objects pay nothing (the `getattr` default preserves current behavior); id queries work in every API including `update_many()`; the storage format is untouched. This would let `tinydantic` delete its entire translation layer.
35+
Existing queries and third-party `QueryLike` objects pay nothing (the `getattr` default preserves current behavior); id queries work in every API including `update_multiple()`; the storage format is untouched. This would let `tinydantic` delete its entire translation layer.
3636

37-
### `update_many()` cannot select documents by id
37+
### `Table.update_multiple()` cannot select documents by id
3838

3939
**Tracking issue:** [#134](https://github.com/tinydantic/tinydantic/issues/134)
4040

4141
**Limitation.** `Table.update(fields, cond, doc_ids)` and `Table.remove(cond, doc_ids)` both accept a `doc_ids` selector, but `Table.update_multiple(updates)` accepts only `(fields, cond)` pairs. Combined with the evaluation limitation above, there is no way to run a batched update that targets documents by id — neither via a condition (conditions cannot see ids) nor via an explicit selector (the parameter does not exist).
4242

43-
**Why it matters to an ODM.** `update_many()` is TinyDB's only batched write — the whole batch runs in one atomic read-modify-write cycle (`Table._update_table()`), which is the reason to use it over looping `update()`. `tinydantic` translates id conditions to `doc_ids=` operations where a public landing spot exists, but `update_many()` offers none, so supporting id conditions there (without sacrificing the batch's single-write atomicity) required the private-API use recorded in the [registry below](#tinydb-private-api-usage-registry).
43+
**Why it matters to an ODM.** `Table.update_multiple()` is TinyDB's only batched write — the whole batch runs in one atomic read-modify-write cycle (`Table._update_table()`), which is the reason to use it over looping `Table.update()`. `tinydantic` translates id conditions to `doc_ids=` operations where a public landing spot exists, but `Table.update_multiple()` offers none, so supporting id conditions in `tinydantic`'s own [`update_many()`][tinydantic.TinydanticModel.update_many] wrapper (without sacrificing the batch's single-write atomicity) required the private-API use recorded in the [registry below](#tinydb-private-api-usage-registry).
4444

4545
**Suggested improvement.** Either of:
4646

47-
- Accept an optional per-pair id selector, e.g. `update_many(updates: Iterable[tuple[fields, cond | None, doc_ids | None]])` (or a parallel `update_multiple_by_ids()`), mirroring the `update()`/`remove()` signatures.
48-
- Ship the `needs_doc_id`/`DocId` improvement above, which subsumes this: id conditions would then work in `update_many()`'s existing signature.
47+
- Accept an optional per-pair id selector, e.g. `update_multiple(updates: Iterable[tuple[fields, cond | None, doc_ids | None]])` (or a parallel `update_multiple_by_ids()`), mirroring the `update()`/`remove()` signatures.
48+
- Ship the `needs_doc_id`/`DocId` improvement above, which subsumes this: id conditions would then work in `update_multiple()`'s existing signature.
49+
50+
### `Query.matches()` is documented as whole-value and anchors only at the start
51+
52+
**Tracking issue:** [#144](https://github.com/tinydantic/tinydantic/issues/144)
53+
54+
**Limitation.** [`Query.matches()`](https://tinydb.readthedocs.io/en/latest/api.html#tinydb.queries.Query.matches) documents itself as "Run a regex test against a dict value (whole string has to match)" and implements the test with `re.match`, which anchors at the start of the value only. `matches(r".*@example\.com")` therefore matches `"[email protected]"`. `re.fullmatch` is the function the docstring describes; `Query.search()` (the deliberately unanchored spelling) uses `re.search`.
55+
56+
**Why it matters to an ODM.** The pattern users reach for is an allow-list — "this field must look like _this_" — and the docstring tells them it is one. It is not, and nothing about the result says so: the query returns rows, just more of them than intended. `tinydantic` presents these builders as its primary query API and had copied the upstream wording verbatim into `GuardedQuery.matches()` and the [Queries](../usage/queries.md) page, so the same wrong contract was published twice more.
57+
58+
There is no code workaround, and none is wanted: changing the behavior would silently narrow every existing caller's results. `tinydantic` carries the correction in prose instead — `GuardedQuery.matches()` (`src/tinydantic/_query.py`) and the Queries page both name the anchoring and point at `$`.
59+
60+
**Suggested improvement.** Correct the docstring — "the pattern is matched against the start of the value (`re.match`); end it with `$` to require a whole-value match" — and, optionally, add a `fullmatch=` flag or a `Query.fullmatch()` builder for the behavior the current wording promises. A behavior change to `re.fullmatch` is explicitly _not_ proposed.
4961

5062
### `Table.update(doc_ids=…)` and `Table.remove(doc_ids=…)` skip missing ids silently
5163

@@ -88,7 +100,7 @@ Every approved use of a TinyDB internal/private API in `tinydantic` is recorded
88100
| TinyDB internal | Used by | Status | Reason | Upstream change that would remove it |
89101
| --- | --- | --- | --- | --- |
90102
| `QueryInstance._hash` (read-only attribute access) | `tinydantic._query.has_id_condition()` — detecting id conditions inside composed queries | **In use** (approved 2026-08-02; shipped with the original id-query work). Tracking: [#133](https://github.com/tinydantic/tinydantic/issues/133) | Composing queries (`&`/`\|`/`~`) produces plain `QueryInstance` objects, so a custom condition type cannot survive composition — the hashval tree is the only place a marker does. The access is read-only via `getattr(cond, "_hash", None)` and degrades loudly, never silently: if a future TinyDB renames the attribute, bare id conditions are still detected by `isinstance`, and undetected compositions raise `DocumentIDConditionError` when TinyDB's evaluator runs them. | The `needs_doc_id`/`DocId` evaluator improvement above (composition would propagate a public flag); alternatively, a public accessor for a query's hash tree. |
91-
| `Table._update_table(updater)` | `TinydanticModel._run_write_cycle()` — all `update()`/`update_many()` writes (which validate each matched document's merged result unless `validate_writes=False`), plus the id-condition paths of `remove()` and `upsert()` | **In use** (approved 2026-07-13 for id-condition writes; scope extended 2026-08-02 to all validated update writes). Tracking: [#134](https://github.com/tinydantic/tinydantic/issues/134) | The only way to select write targets by id inside one atomic read-modify-write cycle, and the only way to validate-then-write atomically: `update_many()` has no `doc_ids` parameter, conditions cannot see ids (the two limitations above), and a public two-pass validate-then-update has a read-modify-write race between passes. The custom updater evaluates every condition against `Document(body, doc_id)` wrappers inside upstream's own read → mutate → write → cache-clear lifecycle, validates merged bodies before the write, applies mutations copy-on-write (so an aborted or validation-failed cycle leaks nothing, even on `MemoryStorage`, whose `read()` shares body dicts by reference), and skips the storage write entirely when nothing matched. Benchmarked ~23% faster than the two-pass public-API alternative on a 5,000-document JSONStorage table (one full file read saved per write). | Either `update_many()` improvement above plus an atomic validate-hook, or the `needs_doc_id`/`DocId` evaluator change combined with a public batched read-modify-write API. |
103+
| `Table._update_table(updater)` | `TinydanticModel._run_write_cycle()` — all `update()`/`update_by_ids()`/`update_all()`/`update_many()` writes (which validate each matched document's merged result unless `validate_writes=False`), plus the id-condition paths of `remove()` and `upsert()` | **In use** (approved 2026-07-13 for id-condition writes; scope extended 2026-08-02 to all validated update writes). Tracking: [#134](https://github.com/tinydantic/tinydantic/issues/134) | The only way to select write targets by id inside one atomic read-modify-write cycle, and the only way to validate-then-write atomically: `Table.update_multiple()` has no `doc_ids` parameter, conditions cannot see ids (the two limitations above), and a public two-pass validate-then-update has a read-modify-write race between passes. The custom updater evaluates every condition against `Document(body, doc_id)` wrappers inside upstream's own read → mutate → write → cache-clear lifecycle, validates merged bodies before the write, applies mutations copy-on-write (so an aborted or validation-failed cycle leaks nothing, even on `MemoryStorage`, whose `read()` shares body dicts by reference), and skips the storage write entirely when nothing matched. Benchmarked ~23% faster than the two-pass public-API alternative on a 5,000-document JSONStorage table (one full file read saved per write). | Either the `update_multiple()` improvement above plus an atomic validate-hook, or the `needs_doc_id`/`DocId` evaluator change combined with a public batched read-modify-write API. |
92104

93105
### Resolved upstream in TinyDB
94106

‎docs/usage/queries.md‎

Lines changed: 21 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -60,7 +60,7 @@ User(id=1, name='Alice', age=30, email='[email protected]', address=Address(city
6060

6161
```
6262

63-
The query object also exposes TinyDB's own methods. `.matches` tests the _whole_ value against a regular expression, `.search` looks for the pattern _anywhere_ in the value, and `.test` runs an arbitrary predicate:
63+
The query object also exposes TinyDB's own methods. `.matches` anchors a regular expression at the _start_ of the value, `.search` looks for the pattern _anywhere_ in the value, and `.test` runs an arbitrary predicate:
6464

6565
```pycon
6666
>>> User.search(User.email.matches(r".*@example\.com"))
@@ -74,6 +74,26 @@ The query object also exposes TinyDB's own methods. `.matches` tests the _whole_
7474

7575
```
7676

77+
> [!WARNING]
78+
>
79+
> `.matches()` is anchored at the start only — it runs `re.match`, not `re.fullmatch`, so the pattern above also matches `[email protected]`. End the pattern with `$` when you mean the whole value. (TinyDB's own docstring says "whole string has to match"; that is the wording, not the behavior — tracked in [#144](https://github.com/tinydantic/tinydantic/issues/144).)
80+
81+
```pycon
82+
>>> mallory = User(
83+
... name="Mallory",
84+
... age=40,
85+
... email="[email protected]",
86+
... address=Address(city="Berlin", country="DE"),
87+
... )
88+
>>> _ = mallory.insert()
89+
>>> [u.name for u in User.search(User.email.matches(r".*@example\.com"))]
90+
['Alice', 'Carol', 'Mallory']
91+
>>> [u.name for u in User.search(User.email.matches(r".*@example\.com$"))]
92+
['Alice', 'Carol']
93+
>>> mallory.delete()
94+
95+
```
96+
7797
> [!WARNING]
7898
>
7999
> `.matches()` and `.search()` compile their pattern with Python's `re` module. Never compile untrusted input as a pattern — attacker-chosen regexes can trigger catastrophic backtracking (ReDoS). See [Security considerations](security.md#untrusted-input-in-queries).

‎project-words.txt‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,7 @@ followee
4040
fontawesome
4141
frozenset
4242
fspath
43+
fullmatch
4344
functools
4445
getattr
4546
getitem

‎src/tinydantic/_query.py‎

Lines changed: 16 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -15,8 +15,10 @@
1515
└── DocIdQuery ── DocIdCondition
1616
```
1717
18-
**Guarded queries** are what ``Model.field``, ``q()``, and
19-
``field()`` return. They behave exactly like TinyDB's own types —
18+
**Guarded queries** are what ``Model.field`` and ``field()``
19+
return. (``q()`` is an identity cast for the type checkers: it
20+
returns whatever it was handed, so a raw ``tinydb.where(...)``
21+
comes back unguarded.) They behave exactly like TinyDB's own types —
2022
same tests, same hashvals, same equality, so they stay
2123
interchangeable with raw TinyDB conditions — except that the three
2224
protocols TinyDB leaves silently wrong raise
@@ -185,11 +187,11 @@ def __bool__(self) -> bool:
185187
raise QueryTypeError(_BOOL_COND_MSG)
186188

187189
def __and__(self, other: QueryInstance) -> QueryInstance:
188-
"""Compose with ``and``, keeping the guard."""
190+
"""Compose with ``&``, keeping the guard."""
189191
return _guarded(super().__and__(other))
190192

191193
def __or__(self, other: QueryInstance) -> QueryInstance:
192-
"""Compose with ``or``, keeping the guard."""
194+
"""Compose with ``|``, keeping the guard."""
193195
return _guarded(super().__or__(other))
194196

195197
def __invert__(self) -> QueryInstance:
@@ -288,11 +290,19 @@ def __ge__(self, rhs: Any) -> Any:
288290
__hash__ = Query.__hash__
289291

290292
def exists(self) -> Any:
291-
"""Build an "key is present" condition."""
293+
"""Build a "key is present" condition."""
292294
return _guarded(super().exists())
293295

294296
def matches(self, regex: str, flags: int = 0) -> Any:
295-
"""Build a whole-value regex condition."""
297+
r"""Build a prefix-anchored regex condition.
298+
299+
TinyDB implements this with ``re.match``, which anchors at
300+
the start of the value only — so a pattern like
301+
``.*@example\.com`` also matches
302+
``'[email protected]'``. End the pattern with ``$``
303+
for a whole-value match. TinyDB's own docstring says
304+
"whole string"; see the Upstream Limitations docs page.
305+
"""
296306
return _guarded(super().matches(regex, flags))
297307

298308
def search(self, regex: str, flags: int = 0) -> Any:

‎tests/model/test_query_guards.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -131,7 +131,7 @@ def test_existence_advice_is_runnable(self) -> None:
131131
exactly the input it is offered for. Running all three
132132
spellings against a miss pins that they return a bool.
133133
"""
134-
cond = User.name == "nobody by this name"
134+
cond = q(User.name) == "nobody by this name"
135135
with pytest.raises(QueryTypeError) as excinfo:
136136
bool(cond)
137137
message = str(excinfo.value)

0 commit comments

Comments
 (0)