Skip to content

Commit 049f261

Browse files
committed
feat(tinydb): add ProfilingMiddleware for storage operation counts
TinyDB has no indexes, so costs are paid in whole-table reads and writes -- and the recent performance issues (#122, #123, #130) each had to hand-roll a counting Storage subclass to show how many. Ship that tool instead: ProfilingMiddleware wraps any storage, counts each read() and write() that reaches it, and resets between measurements. Counters live on the middleware instance, which is the same object TinyDB uses, so a kept reference is all a test needs. Documented with the other middlewares on the storage page, with a doctested example asserting the exact per-insert counts -- so the docs' own numbers break loudly if the write path's cost changes. Assisted-by: Claude:claude-fable-5
1 parent 37b8523 commit 049f261

4 files changed

Lines changed: 174 additions & 1 deletion

File tree

‎docs/usage/storage.md‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -149,6 +149,32 @@ tickets:
149149

150150
Without it, that file would open with `1`, `10`, `2`. The trade-off is in the name: the middleware hands integer keys to the storage below it, which is fine for JSON and YAML but may break a storage that can only serialize string keys.
151151

152+
### Counting storage operations
153+
154+
TinyDB has no indexes, so costs are paid in whole-table reads and writes — and when something is slow, the first question is how many of those an operation triggered. [ProfilingMiddleware][tinydantic.tinydb.middlewares.ProfilingMiddleware] counts every `read()` and `write()` that reaches the storage it wraps. Keep a reference to the middleware — it is the same object TinyDB uses:
155+
156+
```pycon
157+
>>> from tinydantic.tinydb.middlewares import ProfilingMiddleware
158+
>>> storage = ProfilingMiddleware(MemoryStorage)
159+
>>> db = TinyDB(storage=storage)
160+
>>> class Event(TinydanticModel, database=db, table_name="events"):
161+
... name: str
162+
>>> Event(name="boot").insert()
163+
Event(id=1, name='boot')
164+
>>> storage.read_count, storage.write_count
165+
(2, 1)
166+
>>> Event.all()
167+
[Event(id=1, name='boot')]
168+
>>> storage.read_count
169+
3
170+
>>> storage.reset()
171+
>>> storage.read_count, storage.write_count
172+
(0, 0)
173+
174+
```
175+
176+
The counters are exact where timings are noisy, which makes them the right evidence for a performance bug report — an issue's measurement harness can state "3 storage reads" and be reproducible on any machine. Middlewares stack, and the position in the stack is what gets measured: `ProfilingMiddleware(CachingMiddleware(JSONStorage))` counts only what the cache lets through to disk, while `CachingMiddleware(ProfilingMiddleware(JSONStorage))` counts what the database asks for before caching.
177+
152178
## Beyond the built-ins
153179

154180
TinyDB's [storage documentation](https://tinydb.readthedocs.io/en/latest/usage.html#storage-types) lists more options, and third-party packages add backends such as SQLite. Any storage that presents the TinyDB `Storage` interface works with `tinydantic` unchanged, because `tinydantic` only ever hands the storage layer JSON-safe primitives.

‎docs/usage/testing.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -172,5 +172,5 @@ def db(db_session):
172172
## Where next
173173

174174
- [Configuration](configuration.md) — how `bind()` and config inheritance resolve across a class hierarchy.
175-
- [Storage](storage.md) — in-memory versus file-backed backends and their lifecycle.
175+
- [Storage](storage.md) — in-memory versus file-backed backends and their lifecycle, including [counting storage operations](storage.md#counting-storage-operations) in a test with `ProfilingMiddleware`.
176176
- [FastAPI](fastapi.md) — the application-factory and `bind()` patterns applied to a real API, tested with `TestClient`.

‎src/tinydantic/tinydb/middlewares.py‎

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -101,6 +101,69 @@ def write(self, data: dict[str, dict[str, Any]]) -> None:
101101
self.storage.write(data=int_keys_data) # type: ignore[arg-type]
102102

103103

104+
class ProfilingMiddleware(Middleware):
105+
"""Middleware that counts the storage operations under it.
106+
107+
Wraps any [Storage][tinydb.storages.Storage] and counts each
108+
``read()`` and ``write()`` call that reaches it, so the storage
109+
cost of an operation can be measured instead of guessed. TinyDB
110+
has no indexes — costs are paid in whole-table reads and
111+
writes — so the two counters are the numbers that matter when
112+
asking why something is slow, and they are exact where timings
113+
are noisy. Keep a reference to the middleware; it is the same
114+
object TinyDB uses:
115+
116+
```python
117+
storage = ProfilingMiddleware(JSONStorage)
118+
db = TinyDB("db.json", storage=storage)
119+
...
120+
storage.read_count, storage.write_count
121+
```
122+
123+
Counters start at zero, live on the instance, and only ever
124+
count operations on the storage this instance wraps. Call
125+
[reset()][tinydantic.tinydb.middlewares.ProfilingMiddleware.reset]
126+
to zero them between measurements.
127+
128+
Middlewares stack, and the position in the stack is what gets
129+
measured: ``ProfilingMiddleware(CachingMiddleware(JSONStorage))``
130+
counts only the operations the cache lets through, while
131+
``CachingMiddleware(ProfilingMiddleware(JSONStorage))`` counts
132+
what the database asks for before caching.
133+
134+
Attributes:
135+
read_count: Number of ``read()`` calls seen so far.
136+
write_count: Number of ``write()`` calls seen so far.
137+
"""
138+
139+
def __init__(self, storage_cls: type[Storage] | Middleware) -> None:
140+
"""Wrap ``storage_cls`` with operation counting.
141+
142+
Args:
143+
storage_cls: The storage class — or another middleware,
144+
such as ``CachingMiddleware(JSONStorage)`` — to
145+
wrap.
146+
"""
147+
super().__init__(storage_cls)
148+
self.read_count: int = 0
149+
self.write_count: int = 0
150+
151+
def read(self) -> Any:
152+
"""Count the read, then forward it to the wrapped storage."""
153+
self.read_count += 1
154+
return self.storage.read()
155+
156+
def write(self, data: Any) -> None:
157+
"""Count the write, then forward it to the wrapped storage."""
158+
self.write_count += 1
159+
self.storage.write(data)
160+
161+
def reset(self) -> None:
162+
"""Zero both counters."""
163+
self.read_count = 0
164+
self.write_count = 0
165+
166+
104167
def _acquire_lock(fd: int) -> None:
105168
"""Take a non-blocking exclusive OS lock on ``fd``.
106169
Lines changed: 84 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,84 @@
1+
# SPDX-FileCopyrightText: Chris Wilson <[email protected]>
2+
#
3+
# SPDX-License-Identifier: MIT
4+
5+
"""ProfilingMiddleware: storage operation counters."""
6+
7+
from __future__ import annotations
8+
9+
from typing import TYPE_CHECKING
10+
11+
from tinydb import TinyDB
12+
from tinydb.storages import JSONStorage, MemoryStorage
13+
14+
from tinydantic.tinydb.middlewares import ProfilingMiddleware
15+
16+
if TYPE_CHECKING:
17+
from pathlib import Path
18+
19+
20+
class TestProfilingMiddleware:
21+
"""Read/write counting and counter lifecycle."""
22+
23+
def test_counters_start_at_zero(self):
24+
"""A fresh middleware has counted nothing."""
25+
storage = ProfilingMiddleware(MemoryStorage)
26+
assert storage.read_count == 0
27+
assert storage.write_count == 0
28+
29+
def test_read_and_write_calls_are_counted(self):
30+
"""Each read()/write() call increments its counter by one."""
31+
storage = ProfilingMiddleware(MemoryStorage)
32+
storage() # bind the underlying storage, as TinyDB would
33+
storage.write({"t": {"1": {"n": 1}}})
34+
storage.read()
35+
storage.read()
36+
assert storage.write_count == 1
37+
assert storage.read_count == 2
38+
39+
def test_operations_are_forwarded(self):
40+
"""Counting does not alter what the storage stores."""
41+
storage = ProfilingMiddleware(MemoryStorage)
42+
storage()
43+
storage.write({"t": {"1": {"n": 1}}})
44+
assert storage.read() == {"t": {"1": {"n": 1}}}
45+
46+
def test_counts_operations_through_tinydb(self):
47+
"""Table operations are counted on the kept reference."""
48+
storage = ProfilingMiddleware(MemoryStorage)
49+
with TinyDB(storage=storage) as db:
50+
table = db.table("t")
51+
table.insert({"n": 1})
52+
assert storage.write_count == 1
53+
reads_after_insert = storage.read_count
54+
assert reads_after_insert > 0
55+
table.all()
56+
assert storage.read_count == reads_after_insert + 1
57+
58+
def test_wraps_file_storage(self, tmp_path: Path):
59+
"""A file-backed storage is counted and still hits disk."""
60+
path = tmp_path / "db.json"
61+
storage = ProfilingMiddleware(JSONStorage)
62+
with TinyDB(path, storage=storage) as db:
63+
db.insert({"n": 1})
64+
assert storage.write_count == 1
65+
assert path.read_text() == '{"_default": {"1": {"n": 1}}}'
66+
67+
def test_reset_zeroes_both_counters(self):
68+
"""reset() returns the middleware to its initial state."""
69+
storage = ProfilingMiddleware(MemoryStorage)
70+
storage()
71+
storage.write({})
72+
storage.read()
73+
storage.reset()
74+
assert storage.read_count == 0
75+
assert storage.write_count == 0
76+
77+
def test_instances_count_independently(self):
78+
"""Counters live on the instance, not the class."""
79+
counted = ProfilingMiddleware(MemoryStorage)
80+
idle = ProfilingMiddleware(MemoryStorage)
81+
counted()
82+
counted.write({})
83+
assert counted.write_count == 1
84+
assert idle.write_count == 0

0 commit comments

Comments
 (0)