Skip to content

Commit fc2ce67

Browse files
wpfleger96Duncan
andauthored
feat(archive): add observer-frame retention schema and gated DB adapter (#5719)
The local-save archive (`~/.buzz/archive/archive.db`) grows without bound — observer frames (kind 24200) are 99.95% of it by bytes (~1.3M rows, 75–375 MB/day). This lays the schema and process-wide foundations for bounding that growth: a single global retention window for observer frames, kept in `archive_meta`, plus the gated DB adapter every later phase builds on. The prune worker that actually deletes lands in a follow-up; nothing here removes data. ## What this adds - **Migration M4** (`add_archive_meta`, crash-safe): creates the `archive_meta` k/v table and the `archived_at`-covering scope-age index used by the future prune scan, and seeds `observer_retention_days=30`. Runs under one `BEGIN IMMEDIATE` with an in-lock marker recheck and the marker written last, so a crash before COMMIT rolls back every object and the next open re-runs from scratch. Fails closed on an externally-created `archive_meta` present without the marker. - **Process init barrier + gated `ArchiveDb` adapter** owning every production open. All subscription-mutation commands and the archive sync task route their DB work through `ArchiveDb::with_conn`, which awaits the init barrier once. This also fixes a pre-existing bug where `create_save_subscription` ran blocking DB work on the async runtime. - **Commands**: `get_observer_retention_days` / `set_observer_retention_days` (fail-closed — rejects days `< 1` or `> 36500` before writing) and `archive_size_stats` (physical file bytes for the main DB + `-wal` sidecar, plus `page_size` / `page_count` / `freelist_count` — PRAGMAs and file metadata only, no payload scans). ## Scope Observer frames (24200) are the only kind with a retention setting. NIP-AM metrics (44200) and every other archived kind are kept indefinitely with no retention state — no policy table, no per-kind or per-subscription configurability. This is the simplified design ruled in over the earlier per-subscription × per-kind approach: bounding observer frames alone captures essentially all the value. --------- Signed-off-by: Will Pfleger <[email protected]> Signed-off-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz> Co-authored-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz>
1 parent 6039fed commit fc2ce67

10 files changed

Lines changed: 1440 additions & 140 deletions

File tree

‎desktop/src-tauri/src/app_state.rs‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -129,6 +129,7 @@ pub struct AppState {
129129
/// bounded and letting a later leave correctly flip the channel back to
130130
/// `is_member=false`.
131131
pub pending_owned_channels: Mutex<std::collections::HashSet<(String, String)>>,
132+
pub archive_db: crate::archive::ArchiveDb,
132133
}
133134

134135
/// Parse the `BUZZ_PRIVATE_KEY` env var into identity keys. `Some` means the
@@ -219,9 +220,7 @@ pub fn build_app_state() -> AppState {
219220
huddle_audio: Default::default(),
220221
app_handle: Mutex::new(None),
221222
media_proxy_port: AtomicU16::new(0),
222-
prevent_sleep: Arc::new(Mutex::new(
223-
crate::prevent_sleep::PreventSleepState::default(),
224-
)),
223+
prevent_sleep: Default::default(),
225224
keyring_locked: AtomicBool::new(false),
226225
identity_lost: AtomicBool::new(false),
227226
reset_failed: AtomicBool::new(false),
@@ -232,6 +231,7 @@ pub fn build_app_state() -> AppState {
232231
#[cfg(feature = "mesh-llm")]
233232
mesh_coordinator: AsyncMutex::new(None),
234233
pending_owned_channels: Mutex::new(std::collections::HashSet::new()),
234+
archive_db: crate::archive::ArchiveDb::default(),
235235
}
236236
}
237237

Lines changed: 199 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,199 @@
1+
//! Process-wide gated adapter for the local archive SQLite database.
2+
//!
3+
//! Two coupled guarantees, both required by plan v3 (decisions 1–2):
4+
//!
5+
//! 1. **Init barrier.** Exactly one blocking task opens the DB the first time
6+
//! and completes every schema migration (including M4, whose index build
7+
//! over Will's 1.3M-row archive is not free). Every production open — every
8+
//! Tauri archive command and all future startup/prune work — `await`s that
9+
//! result before touching its own connection. The barrier is independent of
10+
//! identity/relay resolution: the archive DB is a single per-nest file
11+
//! (identity is a row column, not part of the path), and the path resolves
12+
//! from `nest_dir()`, which is fixed early in `setup()` before the async
13+
//! workspace relay override settles. This satisfies Thufir's binding
14+
//! condition that the barrier cover workspace/identity timing and the
15+
//! globally-mounted observer archive producer.
16+
//!
17+
//! 2. **Maintenance lock.** A shared `RwLock` whose read guard is held for the
18+
//! full lifetime of every ordinary connection (acquired before the blocking
19+
//! dispatch, released only after the closure returns and its connection is
20+
//! dropped). The Phase-4 "reclaim space" conversion will take the *write*
21+
//! guard for its sole-connection `VACUUM` sequence; nothing else may hold a
22+
//! live connection while that runs. Phase 1 only ever takes the read guard,
23+
//! but the lock lives here so the write path has a home.
24+
//!
25+
//! M4 additionally keeps its own `BEGIN IMMEDIATE` + in-lock recheck for crash
26+
//! and cross-process safety — the in-process barrier serializes this process's
27+
//! opens, but a second OS process (or a direct `open_archive_db` in tests) can
28+
//! still race the first open. The two mechanisms are complementary, not
29+
//! redundant: the barrier is startup orchestration, the immediate-lock is
30+
//! durability.
31+
32+
use std::path::PathBuf;
33+
34+
use rusqlite::Connection;
35+
use tokio::sync::{OnceCell, RwLock};
36+
37+
use super::store;
38+
use crate::managed_agents::nest_dir;
39+
40+
/// A hook run once on the blocking pool at the start of the single init task.
41+
/// Test-only: lets a test count initializations and hold the winner long
42+
/// enough to prove concurrent callers await it. Never set in production.
43+
#[cfg(test)]
44+
type InitHook = std::sync::Arc<dyn Fn() + Send + Sync>;
45+
46+
/// Test-only overrides so the barrier and guard-lifetime contracts can be
47+
/// exercised without a real nest: a fixed DB path in place of `nest_dir()` and
48+
/// an optional init hook. `Default` leaves this `None`, so production always
49+
/// resolves the path from the nest and runs no hook.
50+
#[cfg(test)]
51+
struct TestSeam {
52+
path: PathBuf,
53+
on_init: Option<InitHook>,
54+
}
55+
56+
/// Gated owner of every production archive DB connection. Lives in
57+
/// [`crate::app_state::AppState`]; commands call [`ArchiveDb::with_conn`].
58+
#[derive(Default)]
59+
pub struct ArchiveDb {
60+
/// Set to `()` once the first open (which runs all migrations incl. M4)
61+
/// succeeds. A failed init is NOT cached — the next caller retries — so a
62+
/// transient error (e.g. a briefly unavailable external volume) does not
63+
/// wedge the archive for the process lifetime.
64+
init: OnceCell<()>,
65+
/// Maintenance lock. Ordinary connections hold the read guard for their
66+
/// whole lifetime; the Phase-4 conversion holds the write guard.
67+
maintenance: RwLock<()>,
68+
/// Test-only path/hook overrides; always `None` in production.
69+
#[cfg(test)]
70+
test_seam: Option<TestSeam>,
71+
}
72+
73+
impl ArchiveDb {
74+
/// Resolve the archive DB path. Production resolves from the nest
75+
/// directory; a test seam (when present) supplies a fixed path so the
76+
/// barrier can be exercised without a real nest. Errors only when the nest
77+
/// cannot be resolved (fatal for archive access, same as the former
78+
/// `open_db`).
79+
fn db_path(&self) -> Result<PathBuf, String> {
80+
#[cfg(test)]
81+
if let Some(seam) = &self.test_seam {
82+
return Ok(seam.path.clone());
83+
}
84+
let nest = nest_dir().ok_or("cannot resolve nest directory for archive")?;
85+
Ok(nest.join("archive").join("archive.db"))
86+
}
87+
88+
/// The init hook, if a test installed one; always `None` in production.
89+
#[cfg(test)]
90+
fn init_hook(&self) -> Option<InitHook> {
91+
self.test_seam.as_ref().and_then(|s| s.on_init.clone())
92+
}
93+
94+
/// Complete the one-time init: open the DB once on the blocking pool,
95+
/// running `SCHEMA` + all migrations (incl. M4), then drop the connection.
96+
/// Concurrent callers await the same single execution. Idempotent and
97+
/// cheap after the first success (the cached `()` short-circuits).
98+
async fn ensure_initialized(&self) -> Result<(), String> {
99+
let path = self.db_path()?;
100+
#[cfg(test)]
101+
let hook = self.init_hook();
102+
self.init
103+
.get_or_try_init(|| async {
104+
tokio::task::spawn_blocking(move || {
105+
// Test hook runs at the very start of the single init task,
106+
// before the migration opens the DB — this is where a test
107+
// holds the winner past the busy timeout to prove ordinary
108+
// callers await it. No-op in production.
109+
#[cfg(test)]
110+
if let Some(hook) = hook {
111+
hook();
112+
}
113+
// Opening runs every migration; the connection exists only
114+
// to complete them behind the barrier, so drop it here.
115+
let conn = store::open_archive_db(&path)?;
116+
drop(conn);
117+
Ok::<(), String>(())
118+
})
119+
.await
120+
.map_err(|e| format!("archive init task failed: {e}"))?
121+
})
122+
.await
123+
.map(|_| ())
124+
}
125+
126+
/// Warm the init barrier without running a query. Called once from
127+
/// `setup()` so the first-open migration cost (M4's index build over a
128+
/// large archive) is paid at startup rather than blocking a user's first
129+
/// archive command. A failure here is non-fatal — the first real
130+
/// [`with_conn`](Self::with_conn) caller retries and surfaces the error.
131+
pub async fn warm_init(&self) -> Result<(), String> {
132+
self.ensure_initialized().await
133+
}
134+
135+
/// Run `task` against a fresh archive connection on the blocking pool.
136+
///
137+
/// Ordering: await the init barrier → acquire the maintenance read guard →
138+
/// dispatch the blocking closure with its own connection. The read guard is
139+
/// held across the `.await` on the blocking join, so it is released only
140+
/// after `task` returns and the connection it borrowed has dropped — the
141+
/// guard-lifetime contract the Phase-4 write path depends on.
142+
pub async fn with_conn<T, F>(&self, task: F) -> Result<T, String>
143+
where
144+
T: Send + 'static,
145+
F: FnOnce(&Connection) -> Result<T, String> + Send + 'static,
146+
{
147+
self.ensure_initialized().await?;
148+
let path = self.db_path()?;
149+
let _guard = self.maintenance.read().await;
150+
tokio::task::spawn_blocking(move || {
151+
let conn = store::open_archive_db(&path)?;
152+
task(&conn)
153+
})
154+
.await
155+
.map_err(|e| format!("archive db task failed: {e}"))?
156+
}
157+
}
158+
159+
#[cfg(test)]
160+
impl ArchiveDb {
161+
/// Build an adapter bound to a fixed DB path (no nest required), so the
162+
/// barrier and guard-lifetime contracts can be exercised in isolation.
163+
fn with_test_path(path: PathBuf) -> Self {
164+
Self {
165+
init: OnceCell::new(),
166+
maintenance: RwLock::new(()),
167+
test_seam: Some(TestSeam {
168+
path,
169+
on_init: None,
170+
}),
171+
}
172+
}
173+
174+
/// Build an adapter bound to a fixed path whose single initialization runs
175+
/// `hook` first — used to count initializations and to hold the init task
176+
/// open across the concurrent-caller window.
177+
fn with_test_hook(path: PathBuf, hook: InitHook) -> Self {
178+
Self {
179+
init: OnceCell::new(),
180+
maintenance: RwLock::new(()),
181+
test_seam: Some(TestSeam {
182+
path,
183+
on_init: Some(hook),
184+
}),
185+
}
186+
}
187+
188+
/// Whether the maintenance WRITE guard can be taken right now. A live
189+
/// `with_conn` connection holds the read guard, so this returns `false`
190+
/// while any ordinary connection is open and `true` once all have dropped —
191+
/// exactly the signal the Phase-4 sole-connection VACUUM will gate on.
192+
fn maintenance_write_available(&self) -> bool {
193+
self.maintenance.try_write().is_ok()
194+
}
195+
}
196+
197+
#[cfg(test)]
198+
#[path = "archive_db_tests.rs"]
199+
mod archive_db_tests;

0 commit comments

Comments
 (0)