Skip to content

snapshot_confidence fabricated: stale inputs default to fresh, valid_until hardcoded +6h regardless of data quality #3728

Description

@tg12

Summary

scripts/regional-snapshot/snapshot-meta.mjs produces a snapshot_confidence score that is presented to users as a trust signal for regional intelligence snapshots. The score is structurally broken in two ways that compound: stale inputs default to "fresh" when they lack a timestamp, and valid_until is unconditionally set to +6 hours regardless of input freshness. A snapshot built entirely from stale data will display high confidence and remain marked valid for 6 hours.

Defect 1 — Undated inputs treated as fresh (freshness.mjs:73-77)

// freshness.mjs lines 73-77 (classifyInputs):
// 3. If neither yields a timestamp, fall back to "fresh" (cannot prove
//    staleness). This fallback remains so we don't regress existing
//    keys that have never needed a meta key.

When an upstream seeder crashes and leaves a stale Redis payload that has no timestamp field, classifyInputs() returns "fresh" rather than "stale". The snapshot is built from the old data and snapshot_confidence is computed as if all inputs are current.

This is acknowledged in the codebase itself (todos/181-pending-p2-undated-inputs-treated-as-fresh-confidence-bug.md) but unfixed.

Defect 2 — valid_until hardcoded to +6 hours regardless of input staleness (snapshot-meta.mjs:53)

// snapshot-meta.mjs:53
valid_until: Date.now() + 6 * 60 * 60 * 1000, // 6h

valid_until is computed at snapshot write time, always set 6 hours ahead, and never reduced based on whether any inputs are stale. A snapshot that classifies 80% of inputs as stale still claims full 6-hour validity.

Combined impact

The two defects compound:

  1. Seeder crashes → stale payload with no timestamp
  2. classifyInputs returns "fresh" for that payload (Defect 1)
  3. snapshot_confidence formula: 0.6 * completeness + 0.4 * freshness computes a high score
  4. valid_until is set to +6h (Defect 2)
  5. Users see a high-confidence, valid-for-6-hours snapshot built from stale data

There is no observable signal to the user that any of this happened.

Confidence formula

// snapshot-meta.mjs:43
const snapshot_confidence = round(0.6 * cCompleteness + 0.4 * cFreshness);

The weights 0.6 and 0.4 are also undocumented, but the more critical issue is the freshness default that makes cFreshness artificially high.

Suggested remediation

  1. In classifyInputs(): flip the default for present-but-undated inputs from "fresh" to "stale". The existing metaKey pattern already handles keys that legitimately lack timestamps — they should be using the metaKey path.
  2. In buildPreMeta(): derive valid_until from the oldest fresh input's expected age rather than a fixed offset.
  3. Expose stale_inputs and missing_inputs in the UI alongside the confidence score so users can assess the basis for the score.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions