You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 02365b5
Browse filesBrowse the repository at this point in the historyBrowse files
docs(query): correct matches() semantics and query docstrings
- Q1 GuardedCondition.__and__/__or__ said "Compose with and/or".
They implement & and |, and this file's own error message warns
that and/or silently discard half the query — the reference was
teaching the exact mistake the guard exists to catch.
- Q2 matches() was documented as a whole-value regex test. TinyDB
implements it with re.match, which anchors at the start only, so
`.*@example\.com` matches '[email protected]'. The error is
inherited from TinyDB's own docstring and was republished on
docs/usage/queries.md (D-QRY-3), where a runnable example now
shows both the trap and the `$` fix.
- Q3 the module docstring listed q() among the constructors of
guarded queries. q() is an identity cast — handed a raw
tinydb.where() it returns it unguarded.
- Q4 "an "key is present"" -> "a".
Per the two-artifact rule, matches() gets an upstream_limitations
section and tracking issue #144. Nothing was filed upstream.
The same page also had two TinyDB API names wrong:
- D-CONTRIB-2 the batched-write section called the method
update_many(). TinyDB has no update_many() — it is
Table.update_multiple(); update_many() is tinydantic's own
wrapper. The page is written to be read standalone by an
upstream maintainer, who would find no such method. Issue #134's
title and body carried the same conflation and were corrected.
- D-CONTRIB-3 "Only Table.__iter__ yields Document instances" —
search(), all(), and get() all return Documents in TinyDB 4.9.0.
The load-bearing claim is that conditions are never *evaluated*
against one, which the paragraph already states correctly.
Assisted-by: Claude:claude-opus-5
**Limitation.** TinyDB evaluates every query condition against the raw document _body_ mapping — the value in the `{doc_id: body}` table dict — never against a [`Document`](https://tinydb.readthedocs.io/en/latest/api.html#tinydb.table.Document) carrying `doc_id`. All five evaluation sites behave this way: `Table.search()`, `Table.get(cond=...)`, and the updater loops inside `Table.update(cond=...)`, `Table.update_multiple()`, and `Table.remove(cond=...)`. (`Table._update_table()`'s docstring notes that skipping the `Document` wrap is a deliberate optimization.) Only `Table.__iter__` yields `Document` instances. Consequently, no query object — however it is constructed — can express "document id equals 1": the id is structurally invisible to the condition, even though it sits right there as the dict key in every evaluation loop.
25
+
**Limitation.** TinyDB evaluates every query condition against the raw document _body_ mapping — the value in the `{doc_id: body}` table dict — never against a [`Document`](https://tinydb.readthedocs.io/en/latest/api.html#tinydb.table.Document) carrying `doc_id`. All five evaluation sites behave this way: `Table.search()`, `Table.get(cond=...)`, and the updater loops inside `Table.update(cond=...)`, `Table.update_multiple()`, and `Table.remove(cond=...)`. (`Table._update_table()`'s docstring notes that skipping the `Document` wrap is a deliberate optimization.) `Table.__iter__`, `search()`, `all()`, and `get()` all _return_`Document` instances — but none of them evaluate the condition against one. Consequently, no query object — however it is constructed — can express "document id equals 1": the id is structurally invisible to the condition, even though it sits right there as the dict key in every evaluation loop.
26
26
27
-
**Why it matters to an ODM.** Every mainstream document ODM lets users query by the model's id attribute (Beanie and ODMantic translate `Model.id` to MongoDB's `_id`; Firestore special-cases `FieldPath.documentId()` because — exactly like TinyDB — its document key is not a body field). `tinydantic` maps its `id` field to `doc_id`, so supporting `User.get(User.id == 1)` required a translation layer: id-bearing conditions are detected in every model method and executed via `doc_id=`/`doc_ids=` operations or by iterating the table (the one API that yields `Document`s). See `src/tinydantic/_query.py` and the id-condition branches in `src/tinydantic/_model.py`.
27
+
**Why it matters to an ODM.** Every mainstream document ODM lets users query by the model's id attribute (Beanie and ODMantic translate `Model.id` to MongoDB's `_id`; Firestore special-cases `FieldPath.documentId()` because — exactly like TinyDB — its document key is not a body field). `tinydantic` maps its `id` field to `doc_id`, so supporting `User.get(User.id == 1)` required a translation layer: id-bearing conditions are detected in every model method and executed via `Table.update(doc_ids=…)`/`Table.remove(doc_ids=…)` operations or by iterating the table (the one API that hands the condition a `Document`). See `src/tinydantic/_query.py` and the id-condition branches in `src/tinydantic/_model.py`.
28
28
29
29
**Suggested improvement.** An opt-in protocol that keeps the existing fast path free:
30
30
31
31
1. Give `QueryInstance` a `needs_doc_id: bool = False` attribute, propagated through `__and__`/`__or__`/`__invert__` (`self.needs_doc_id or other.needs_doc_id`).
32
32
2. Add a `DocId` query type to `tinydb.queries` whose comparisons build `QueryInstance`s with `needs_doc_id=True` and test `doc.doc_id`.
33
33
3. At each of the five evaluation sites: `if getattr(cond, "needs_doc_id", False)`, wrap the body in `self.document_class(doc, self.document_id_class(doc_id))` before calling the condition; otherwise call it with the raw body exactly as today.
34
34
35
-
Existing queries and third-party `QueryLike` objects pay nothing (the `getattr` default preserves current behavior); id queries work in every API including `update_many()`; the storage format is untouched. This would let `tinydantic` delete its entire translation layer.
35
+
Existing queries and third-party `QueryLike` objects pay nothing (the `getattr` default preserves current behavior); id queries work in every API including `update_multiple()`; the storage format is untouched. This would let `tinydantic` delete its entire translation layer.
36
36
37
-
### `update_many()` cannot select documents by id
37
+
### `Table.update_multiple()` cannot select documents by id
**Limitation.**`Table.update(fields, cond, doc_ids)` and `Table.remove(cond, doc_ids)` both accept a `doc_ids` selector, but `Table.update_multiple(updates)` accepts only `(fields, cond)` pairs. Combined with the evaluation limitation above, there is no way to run a batched update that targets documents by id — neither via a condition (conditions cannot see ids) nor via an explicit selector (the parameter does not exist).
42
42
43
-
**Why it matters to an ODM.**`update_many()` is TinyDB's only batched write — the whole batch runs in one atomic read-modify-write cycle (`Table._update_table()`), which is the reason to use it over looping `update()`. `tinydantic` translates id conditions to `doc_ids=` operations where a public landing spot exists, but `update_many()` offers none, so supporting id conditions there (without sacrificing the batch's single-write atomicity) required the private-API use recorded in the [registry below](#tinydb-private-api-usage-registry).
43
+
**Why it matters to an ODM.**`Table.update_multiple()` is TinyDB's only batched write — the whole batch runs in one atomic read-modify-write cycle (`Table._update_table()`), which is the reason to use it over looping `Table.update()`. `tinydantic` translates id conditions to `doc_ids=` operations where a public landing spot exists, but `Table.update_multiple()` offers none, so supporting id conditions in `tinydantic`'s own [`update_many()`][tinydantic.TinydanticModel.update_many] wrapper (without sacrificing the batch's single-write atomicity) required the private-API use recorded in the [registry below](#tinydb-private-api-usage-registry).
44
44
45
45
**Suggested improvement.** Either of:
46
46
47
-
- Accept an optional per-pair id selector, e.g. `update_many(updates: Iterable[tuple[fields, cond | None, doc_ids | None]])` (or a parallel `update_multiple_by_ids()`), mirroring the `update()`/`remove()` signatures.
48
-
- Ship the `needs_doc_id`/`DocId` improvement above, which subsumes this: id conditions would then work in `update_many()`'s existing signature.
47
+
- Accept an optional per-pair id selector, e.g. `update_multiple(updates: Iterable[tuple[fields, cond | None, doc_ids | None]])` (or a parallel `update_multiple_by_ids()`), mirroring the `update()`/`remove()` signatures.
48
+
- Ship the `needs_doc_id`/`DocId` improvement above, which subsumes this: id conditions would then work in `update_multiple()`'s existing signature.
49
+
50
+
### `Query.matches()` is documented as whole-value and anchors only at the start
**Limitation.**[`Query.matches()`](https://tinydb.readthedocs.io/en/latest/api.html#tinydb.queries.Query.matches) documents itself as "Run a regex test against a dict value (whole string has to match)" and implements the test with `re.match`, which anchors at the start of the value only. `matches(r".*@example\.com")` therefore matches `"[email protected]"`. `re.fullmatch` is the function the docstring describes; `Query.search()` (the deliberately unanchored spelling) uses `re.search`.
55
+
56
+
**Why it matters to an ODM.** The pattern users reach for is an allow-list — "this field must look like _this_" — and the docstring tells them it is one. It is not, and nothing about the result says so: the query returns rows, just more of them than intended. `tinydantic` presents these builders as its primary query API and had copied the upstream wording verbatim into `GuardedQuery.matches()` and the [Queries](../usage/queries.md) page, so the same wrong contract was published twice more.
57
+
58
+
There is no code workaround, and none is wanted: changing the behavior would silently narrow every existing caller's results. `tinydantic` carries the correction in prose instead — `GuardedQuery.matches()` (`src/tinydantic/_query.py`) and the Queries page both name the anchoring and point at `$`.
59
+
60
+
**Suggested improvement.** Correct the docstring — "the pattern is matched against the start of the value (`re.match`); end it with `$` to require a whole-value match" — and, optionally, add a `fullmatch=` flag or a `Query.fullmatch()` builder for the behavior the current wording promises. A behavior change to `re.fullmatch` is explicitly _not_ proposed.
49
61
50
62
### `Table.update(doc_ids=…)` and `Table.remove(doc_ids=…)` skip missing ids silently
51
63
@@ -88,7 +100,7 @@ Every approved use of a TinyDB internal/private API in `tinydantic` is recorded
88
100
| TinyDB internal | Used by | Status | Reason | Upstream change that would remove it |
89
101
| --- | --- | --- | --- | --- |
90
102
|`QueryInstance._hash` (read-only attribute access) |`tinydantic._query.has_id_condition()` — detecting id conditions inside composed queries |**In use** (approved 2026-08-02; shipped with the original id-query work). Tracking: [#133](https://github.com/tinydantic/tinydantic/issues/133)| Composing queries (`&`/`\|`/`~`) produces plain `QueryInstance` objects, so a custom condition type cannot survive composition — the hashval tree is the only place a marker does. The access is read-only via `getattr(cond, "_hash", None)` and degrades loudly, never silently: if a future TinyDB renames the attribute, bare id conditions are still detected by `isinstance`, and undetected compositions raise `DocumentIDConditionError` when TinyDB's evaluator runs them. | The `needs_doc_id`/`DocId` evaluator improvement above (composition would propagate a public flag); alternatively, a public accessor for a query's hash tree. |
91
-
| `Table._update_table(updater)` | `TinydanticModel._run_write_cycle()` — all `update()`/`update_many()` writes (which validate each matched document's merged result unless `validate_writes=False`), plus the id-condition paths of `remove()` and `upsert()` | **In use** (approved 2026-07-13 for id-condition writes; scope extended 2026-08-02 to all validated update writes). Tracking: [#134](https://github.com/tinydantic/tinydantic/issues/134) | The only way to select write targets by id inside one atomic read-modify-write cycle, and the only way to validate-then-write atomically: `update_many()` has no `doc_ids` parameter, conditions cannot see ids (the two limitations above), and a public two-pass validate-then-update has a read-modify-write race between passes. The custom updater evaluates every condition against `Document(body, doc_id)` wrappers inside upstream's own read → mutate → write → cache-clear lifecycle, validates merged bodies before the write, applies mutations copy-on-write (so an aborted or validation-failed cycle leaks nothing, even on `MemoryStorage`, whose `read()` shares body dicts by reference), and skips the storage write entirely when nothing matched. Benchmarked ~23% faster than the two-pass public-API alternative on a 5,000-document JSONStorage table (one full file read saved per write). | Either `update_many()` improvement above plus an atomic validate-hook, or the `needs_doc_id`/`DocId` evaluator change combined with a public batched read-modify-write API. |
103
+
| `Table._update_table(updater)` | `TinydanticModel._run_write_cycle()` — all `update()`/`update_by_ids()`/`update_all()`/`update_many()` writes (which validate each matched document's merged result unless `validate_writes=False`), plus the id-condition paths of `remove()` and `upsert()` | **In use** (approved 2026-07-13 for id-condition writes; scope extended 2026-08-02 to all validated update writes). Tracking: [#134](https://github.com/tinydantic/tinydantic/issues/134) | The only way to select write targets by id inside one atomic read-modify-write cycle, and the only way to validate-then-write atomically: `Table.update_multiple()` has no `doc_ids` parameter, conditions cannot see ids (the two limitations above), and a public two-pass validate-then-update has a read-modify-write race between passes. The custom updater evaluates every condition against `Document(body, doc_id)` wrappers inside upstream's own read → mutate → write → cache-clear lifecycle, validates merged bodies before the write, applies mutations copy-on-write (so an aborted or validation-failed cycle leaks nothing, even on `MemoryStorage`, whose `read()` shares body dicts by reference), and skips the storage write entirely when nothing matched. Benchmarked ~23% faster than the two-pass public-API alternative on a 5,000-document JSONStorage table (one full file read saved per write). | Either the `update_multiple()` improvement above plus an atomic validate-hook, or the `needs_doc_id`/`DocId` evaluator change combined with a public batched read-modify-write API. |
The query object also exposes TinyDB's own methods. `.matches`tests the _whole_ value against a regular expression, `.search` looks for the pattern _anywhere_ in the value, and `.test` runs an arbitrary predicate:
63
+
The query object also exposes TinyDB's own methods. `.matches`anchors a regular expression at the _start_ of the value, `.search` looks for the pattern _anywhere_ in the value, and `.test` runs an arbitrary predicate:
@@ -74,6 +74,26 @@ The query object also exposes TinyDB's own methods. `.matches` tests the _whole_
74
74
75
75
```
76
76
77
+
> [!WARNING]
78
+
>
79
+
> `.matches()` is anchored at the start only — it runs `re.match`, not `re.fullmatch`, so the pattern above also matches `[email protected]`. End the pattern with `$` when you mean the whole value. (TinyDB's own docstring says "whole string has to match"; that is the wording, not the behavior — tracked in [#144](https://github.com/tinydantic/tinydantic/issues/144).)
>>> [u.name for u in User.search(User.email.matches(r".*@example\.com"))]
90
+
['Alice', 'Carol', 'Mallory']
91
+
>>> [u.name for u in User.search(User.email.matches(r".*@example\.com$"))]
92
+
['Alice', 'Carol']
93
+
>>> mallory.delete()
94
+
95
+
```
96
+
77
97
> [!WARNING]
78
98
>
79
99
> `.matches()` and `.search()` compile their pattern with Python's `re` module. Never compile untrusted input as a pattern — attacker-chosen regexes can trigger catastrophic backtracking (ReDoS). See [Security considerations](security.md#untrusted-input-in-queries).
0 commit comments