Skip to content

Commit 4f23d9a

Browse files
committed
fix(model): honor document.id in upsert()'s no-match insert
upsert() with a condition serialized the model with force_dict=True, so a set document.id was dropped on the no-match insert: the document landed at a fresh id and document.id was then overwritten with it, destroying even the evidence that the caller asked for a specific id (#120). The same instance produced two different outcomes depending on whether a condition was passed. The no-match insert now honors document.id exactly as insert() does: a set id says "put it here", so the document is inserted at that id, and a taken id raises DocumentAlreadyExistsError instead of silently duplicating the document elsewhere. Only id=None still gets a fresh id, and the condition's own id value is still never adopted. A matching condition still wins over a preset id (pinned by a test). Delegating to Table.upsert() cannot express this: handed a Document and a cond, TinyDB extracts the doc_id and passes it to Table.update(document, cond, doc_ids=[doc_id]), which checks doc_ids before cond — the condition is never evaluated. The non-id-condition branch therefore inlines upstream's update-then-insert through the public Table.update()/Table.insert(); the id-condition branch keeps its atomic write cycle and now passes the id-bearing Document through to the fallback insert. Both branches translate the taken-id ValueError via the new _upsert_insert() helper. Friction recorded on the Upstream Limitations page and tracked as #148. Assisted-by: Claude:claude-fable-5
1 parent 3d7d09a commit 4f23d9a

8 files changed

Lines changed: 173 additions & 15 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
88

99
### Fixed
1010

11+
- `upsert()` no longer silently discards a set `document.id` when a condition is passed ([#120](https://github.com/tinydantic/tinydantic/issues/120)). The same instance used to produce two different outcomes: with no condition, `upsert(Book(id=7, ...))` honored the id; with a condition that matched nothing, the insert landed at a fresh id and `document.id` was then overwritten with it — destroying even the evidence that the caller asked for 7. The no-match insert now honors `document.id` exactly as `insert()` does: a set id says "put it here", so the document is inserted at that id, and an id that is already taken raises `DocumentAlreadyExistsError` instead of silently duplicating the document elsewhere; only a document with `id=None` still gets a fresh id. The condition's own id value is still never adopted (`upsert(book, Book.id == 7)` matching nothing does not insert at 7), matching TinyDB's own upsert semantics, and a matching condition still wins over a preset id, which is now pinned by a test. Behind the fix, the non-id-condition branch no longer delegates to `Table.upsert()`: handed a `Document` _and_ a condition, TinyDB silently prefers the `doc_id` and never evaluates the condition, so tinydantic inlines upstream's update-then-insert through the public `Table.update()`/`Table.insert()` instead — friction recorded on [Upstream Limitations](https://tinydantic.github.io/tinydantic/latest/contributing/upstream_limitations/) and tracked as [#148](https://github.com/tinydantic/tinydantic/issues/148).
12+
1113
- Three error messages recommended APIs that do not exist or do not work, all of them survivors of this cycle's renames. The "condition has no truth value" guard offered `Model.get(cond) is not None` as an existence check — but `get()` raises `DocumentNotFoundError` on a miss, so the advice fails on exactly the input it is offered for; the runnable spelling is `get_or_none(cond) is not None`. `DocumentIDUpdateError` said to "use `doc_ids=[...]`", a selector parameter removed this cycle in favor of `update_by_ids()`. And `DocumentIDConditionError` pointed at `doc_id=`/`doc_ids=` for the same reason; it now names `get_by_ids()`, `update_by_ids()`, and `remove_by_ids()`. Each is pinned by a test that runs the advice rather than only matching the text.
1214

1315
- Docs: the published reference and guides carried 76 verified inaccuracies, found by a full audit of every Markdown page, every `src/` and `scripts/` docstring, and every `tests/` docstring. The load-bearing ones: `get()` was documented as returning `None` on a miss in the README quickstart and demonstrated that way in the Concurrency page's ETag handlers (where it made the 404 branch dead code and turned a real miss into a 500); the Testing page's fixture never bound a database, so both example tests below it would have raised `DatabaseNotBoundError`; `ProfilingMiddleware`'s stacking advice was exactly inverted in both its docstring and the Storage page, so a reader instrumented the opposite layer from the one they wanted; `matches()` was described as a whole-value regex test when TinyDB anchors it at the start only, which turns the obvious allow-list pattern into one that accepts `[email protected]`; the Queries page said bitwise operators bind more loosely than comparisons (they bind more tightly, which is why it recommends parentheses) and carried a type-checker transcript that was wrong in four ways; twelve `Raises:` entries were missing across four exception types; and the enforcement, no-hook, and unique-bypass verb rosters each named two or three verbs where five apply. `docs/contributing/upstream_limitations.md` also called TinyDB's batched write `update_many()` — TinyDB has no such method, and the page is written to be read standalone by an upstream maintainer.

‎docs/contributing/upstream_limitations.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -47,6 +47,16 @@ Existing queries and third-party `QueryLike` objects pay nothing (the `getattr`
4747
- Accept an optional per-pair id selector, e.g. `update_multiple(updates: Iterable[tuple[fields, cond | None, doc_ids | None]])` (or a parallel `update_multiple_by_ids()`), mirroring the `update()`/`remove()` signatures.
4848
- Ship the `needs_doc_id`/`DocId` improvement above, which subsumes this: id conditions would then work in `update_multiple()`'s existing signature.
4949

50+
### `Table.upsert()` silently ignores the condition when the document carries a `doc_id`
51+
52+
**Tracking issue:** [#148](https://github.com/tinydantic/tinydantic/issues/148)
53+
54+
**Limitation.** `Table.upsert(document, cond)` treats its two selectors as either/or: when `document` is a [`Document`](https://tinydb.readthedocs.io/en/latest/api.html#tinydb.table.Document) carrying a `doc_id`, the implementation extracts that id and calls `Table.update(document, cond, doc_ids=[doc_id])` — and `update()` checks `doc_ids` before `cond`, so a condition passed alongside a `Document` is **never evaluated**. Nothing raises or warns; the call simply performs a different operation (upsert by id) than the one written (upsert by condition). Consequently the public API cannot express "update the documents matching this condition; if none match, insert at this specific id": passing both selects by id and silently drops the condition, while a plain dict body cannot carry an id into the fall-through insert.
55+
56+
**Why it matters to an ODM.** `tinydantic` maps its `id` field to `doc_id`, and its [`upsert()`][tinydantic.TinydanticModel.upsert] honors a set `document.id` on the no-match insert exactly as [`insert()`][tinydantic.TinydanticModel.insert] does — a set id says "put it here" (tinydantic [#120](https://github.com/tinydantic/tinydantic/issues/120) fixed the earlier behavior of silently discarding it). Expressing that means serializing the model as a `Document` so its id survives to the insert — and handing that `Document` plus the user's condition to `Table.upsert()` would silently turn every condition-upsert into an id-upsert. `TinydanticModel.upsert()` (`src/tinydantic/_model.py`) therefore inlines upstream's own update-then-insert instead of delegating: `Table.update(document, cond)` first, and when nothing matched, `Table.insert(document)` with the id-bearing `Document` — public API throughout, the same two storage cycles `Table.upsert()` itself spends, with the insert's "id already taken" `ValueError` translated to [`DocumentAlreadyExistsError`][tinydantic.DocumentAlreadyExistsError]. (Id-bearing _conditions_ take the atomic private-API path recorded in the [registry below](#tinydb-private-api-usage-registry) for other reasons.)
57+
58+
**Suggested improvement.** When both selectors are supplied, either honor the condition — evaluate `cond`, with the `doc_id` seeding only the fall-through insert, which is strictly more expressive and is what the call reads as — or refuse the combination with a `ValueError`, as the doc_id-less-and-cond-less call already does. Either is better than the silent precedence; at minimum, the docstring ("optional if you've passed a Document with a doc_id") should state that a `Document`'s id does not merely make `cond` optional but overrides it entirely.
59+
5060
### `Query.matches()` is documented as whole-value and anchors only at the start
5161

5262
**Tracking issue:** [#144](https://github.com/tinydantic/tinydantic/issues/144)

‎docs/usage/crud.md‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -63,6 +63,19 @@ Book(id=1, title='Dune', author='Herbert', year=1965, in_stock=True)
6363

6464
```
6565

66+
When nothing matches, the insert honors the instance's own `id` exactly as `insert()` does: a set `id` says "put it here", so the document lands at that id — and an id that is already taken is refused with [DocumentAlreadyExistsError][tinydantic.DocumentAlreadyExistsError] rather than silently duplicated at a fresh one. (The _condition's_ id value is never adopted: `upsert(book, Book.id == 7)` matching nothing inserts at `book.id` — or a fresh id if `book.id` is `None` — not at 7.)
67+
68+
```pycon
69+
>>> Book.upsert(
70+
... Book(id=1, title="Hyperion", author="Simmons", year=1989),
71+
... Book.title == "Nonesuch",
72+
... )
73+
Traceback (most recent call last):
74+
...
75+
tinydantic._errors.DocumentAlreadyExistsError: Document with id 1 already exists in table 'books' (model 'Book'). Omit id to let TinyDB assign one, or use save()/upsert() to update the existing document.
76+
77+
```
78+
6679
## Read
6780

6881
The table now holds four books. Read methods return validated model instances with `id` set from the stored document id.

‎project-words.txt‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -53,6 +53,7 @@ htmlcov
5353
httpx
5454
hynek
5555
inlinehilite
56+
inlines
5657
ints
5758
ipython
5859
isinstance

‎src/tinydantic/_model.py‎

Lines changed: 60 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -2444,8 +2444,12 @@ def upsert(
24442444
24452445
Conditions on ``Model.id`` are translated to document-id
24462446
operations. As with TinyDB's own upsert, when nothing
2447-
matches, the insert does not adopt the condition's id value —
2448-
the document is inserted with a fresh id.
2447+
matches, the insert does not adopt the condition's id
2448+
value. It does honor ``document.id``, exactly like
2449+
[insert][tinydantic.TinydanticModel.insert]: a set id
2450+
says "put it here", so the document is inserted at that id
2451+
— or refused when the id is already taken — and only a
2452+
document with ``id=None`` is inserted with a fresh id.
24492453
24502454
When exactly one document is affected — an insert, or an
24512455
update that matched a single document — ``document.id`` is
@@ -2469,6 +2473,9 @@ def upsert(
24692473
is nothing to select the document to update by.
24702474
QueryTypeError: If ``cond`` is given but is not
24712475
something TinyDB can evaluate as a condition.
2476+
DocumentAlreadyExistsError: If nothing matched
2477+
``cond`` and ``document.id`` is set to an id that
2478+
already exists in the table; nothing is written.
24722479
UniqueConstraintError: If the write would duplicate a
24732480
[Unique][tinydantic.Unique] field value or a
24742481
[UniqueConstraint][tinydantic.UniqueConstraint]
@@ -2499,18 +2506,27 @@ def upsert(
24992506
# held tokens elsewhere correctly go stale. One token for
25002507
# the batch: equality is the only comparison tokens face.
25012508
token = uuid4() if cls._uses_revision() else None
2502-
if selector is not None and has_id_condition(selector):
2503-
document_dict = document.to_tinydb_document(force_dict=True)
2504-
if token is not None:
2505-
document_dict["revision_id"] = str(token)
2506-
ids = cls._upsert_id_condition(document_dict, selector)
2509+
# No force_dict: a set id rides along as the Document's
2510+
# doc_id, so the no-match insert lands at the id the
2511+
# caller asked for. Matched updates read the payload as a
2512+
# plain mapping (doc_id is an attribute, not a body key),
2513+
# so they never adopt it.
2514+
serialized = document.to_tinydb_document()
2515+
if token is not None:
2516+
serialized["revision_id"] = str(token)
2517+
if selector is None:
2518+
ids = cls.get_table().upsert(serialized)
2519+
elif has_id_condition(selector):
2520+
ids = cls._upsert_id_condition(serialized, selector)
25072521
else:
2508-
serialized = document.to_tinydb_document(
2509-
force_dict=selector is not None,
2510-
)
2511-
if token is not None:
2512-
serialized["revision_id"] = str(token)
2513-
ids = cls.get_table().upsert(serialized, selector)
2522+
# Not Table.upsert(serialized, selector): handed a
2523+
# Document *and* a cond, TinyDB silently prefers the
2524+
# doc_id and never evaluates the cond (see the
2525+
# Upstream Limitations page), so its update-then-
2526+
# insert is inlined here.
2527+
ids = cls.get_table().update(serialized, selector)
2528+
if not ids:
2529+
ids = [cls._upsert_insert(serialized)]
25142530
if len(ids) == 1:
25152531
document.id = ids[0]
25162532
if token is not None:
@@ -2528,7 +2544,9 @@ def _upsert_id_condition(
25282544
Merges ``document_dict`` into every document matching
25292545
``cond`` in one read-modify-write cycle, inserting it as a
25302546
new document when nothing matched (TinyDB upsert
2531-
semantics — the insert does not adopt the condition's id).
2547+
semantics — the insert does not adopt the condition's id,
2548+
though it does honor a ``doc_id`` carried by
2549+
``document_dict`` itself).
25322550
"""
25332551
table = cls.get_table()
25342552

@@ -2550,9 +2568,36 @@ def apply_upsert(data: dict[int, Any]) -> list[int]:
25502568

25512569
ids = cls._run_write_cycle(apply_upsert)
25522570
if not ids:
2553-
ids = [table.insert(document_dict)]
2571+
ids = [cls._upsert_insert(document_dict)]
25542572
return ids
25552573

2574+
@classmethod
2575+
def _upsert_insert(cls, document_dict: dict[str, Any]) -> int:
2576+
"""Insert upsert()'s payload after nothing matched.
2577+
2578+
When ``document_dict`` is a [Document][tinydb.table.Document]
2579+
(the caller's model had ``id`` set), the insert targets its
2580+
``doc_id``, and TinyDB refuses a taken id with a bare
2581+
``ValueError`` — translated here to the same
2582+
[DocumentAlreadyExistsError][tinydantic.DocumentAlreadyExistsError]
2583+
that [insert][tinydantic.TinydanticModel.insert] raises. A
2584+
plain dict payload gets a fresh id and cannot collide.
2585+
2586+
Raises:
2587+
DocumentAlreadyExistsError: If the payload's ``doc_id``
2588+
is already present in the table.
2589+
"""
2590+
try:
2591+
return cls.get_table().insert(document_dict)
2592+
except ValueError as exc:
2593+
if not isinstance(document_dict, Document):
2594+
raise
2595+
raise DocumentAlreadyExistsError(
2596+
model_name=cls.__name__,
2597+
table_name=cls.get_table().name,
2598+
doc_ids=[document_dict.doc_id],
2599+
) from exc
2600+
25562601
@classmethod
25572602
def remove(
25582603
cls,

‎tests/model/test_id_queries.py‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@
1717

1818
from tests.model.models import UserBase
1919
from tinydantic import (
20+
DocumentAlreadyExistsError,
2021
DocumentIDConditionError,
2122
DocumentIDUpdateError,
2223
DocumentNotFoundError,
@@ -403,6 +404,28 @@ def test_upsert_inserts_when_no_match(
403404
assert document.id == 4
404405
assert users.count() == 4
405406

407+
def test_upsert_no_match_inserts_at_documents_id(
408+
self,
409+
users: type[UserBase],
410+
) -> None:
411+
"""The no-match insert honors the document's own free id."""
412+
document = users(id=7, name="Dave", age=40)
413+
ids = users.upsert(document, q(users.id) == 999)
414+
assert ids == [7]
415+
assert document.id == 7
416+
assert users.count() == 4
417+
418+
def test_upsert_no_match_taken_id_raises(
419+
self,
420+
users: type[UserBase],
421+
) -> None:
422+
"""A no-match insert at a taken id is refused whole."""
423+
document = users(id=2, name="Dave", age=40)
424+
with pytest.raises(DocumentAlreadyExistsError, match="id 2"):
425+
users.upsert(document, q(users.id) == 999)
426+
assert users.count() == 3
427+
assert users.get_by_id(2).name == "Bob"
428+
406429
def test_update_multiple_by_id_condition(
407430
self,
408431
users: type[UserBase],

‎tests/model/test_revision.py‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -671,3 +671,20 @@ class Book(TinydanticModel, database=db, use_revision=True):
671671
ids = Book.upsert(book, q(Book.title) == "Dune")
672672
assert book.id == ids[0]
673673
assert isinstance(book.revision_id, UUID)
674+
675+
def test_upsert_insert_branch_honors_preset_id(self, db: TinyDB):
676+
"""An upsert that inserts at a preset id still mints a token."""
677+
678+
class Book(TinydanticModel, database=db, use_revision=True):
679+
"""Test model."""
680+
681+
title: str
682+
683+
book = Book(id=7, title="Dune")
684+
ids = Book.upsert(book, q(Book.title) == "Dune")
685+
assert ids == [7]
686+
assert book.id == 7
687+
assert isinstance(book.revision_id, UUID)
688+
loaded = Book.get_by_id(7)
689+
assert loaded is not None
690+
assert loaded.revision_id == book.revision_id

‎tests/model/test_table_methods.py‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@
1313
from pydantic import ValidationError, model_validator
1414

1515
from tinydantic import (
16+
DocumentAlreadyExistsError,
1617
DocumentNotFoundError,
1718
SelectorError,
1819
TinydanticModel,
@@ -260,6 +261,52 @@ def test_upsert_without_cond_keeps_id(
260261
assert ids == [user.id]
261262
assert user.id == ids[0]
262263

264+
def test_upsert_no_match_inserts_at_documents_free_id(
265+
self,
266+
user_class: type[UserBase],
267+
):
268+
"""When nothing matches, the insert honors a preset free id."""
269+
document = user_class(id=7, name="Alice", age=37)
270+
ids = user_class.upsert(
271+
document,
272+
user_class.name == "Nobody", # type: ignore[arg-type]
273+
)
274+
assert ids == [7]
275+
assert document.id == 7
276+
fetched = user_class.get_by_id(7)
277+
assert fetched.name == "Alice"
278+
279+
def test_upsert_no_match_taken_id_raises(
280+
self,
281+
user_class: type[UserBase],
282+
):
283+
"""When nothing matches and the preset id is taken, refuse."""
284+
user_class(name="Alice", age=37).insert()
285+
document = user_class(id=1, name="Bob", age=24)
286+
with pytest.raises(DocumentAlreadyExistsError, match="id 1"):
287+
user_class.upsert(
288+
document,
289+
user_class.name == "Nobody", # type: ignore[arg-type]
290+
)
291+
assert document.id == 1
292+
assert user_class.count() == 1
293+
assert user_class.get_by_id(1).name == "Alice"
294+
295+
def test_upsert_match_wins_over_preset_id(
296+
self,
297+
user_class: type[UserBase],
298+
):
299+
"""A matching condition updates the match, not the preset id."""
300+
user_class(name="Alice", age=37).insert()
301+
document = user_class(id=7, name="Alice", age=99)
302+
ids = user_class.upsert(
303+
document,
304+
user_class.name == "Alice", # type: ignore[arg-type]
305+
)
306+
assert ids == [1]
307+
assert document.id == 1
308+
assert user_class.count() == 1
309+
263310

264311
class TestRemove:
265312
"""remove() mirrors TinyDB."""

0 commit comments

Comments
 (0)