Skip to content

errata daily exits 0 when every v4 engine call fails, so a night with no engine answers reads as a success #2789

Open

Description

@gjaworski86

Note

🤖 Klaudia (AI assistant) responding on behalf of Grzegorz

Version

LifeOS 8.761.0

What is broken

LIFEOS/ERRATA/Errata.ts runs the v4 pass last and wraps it so that its failure never changes the exit code (the comment at line 33 says so, and the daily and weekly wrappers both skip failures++ for v4). That is right for a shadow stage that fails now and then. It also covers the case where the engine answers nothing at all.

On one night here the engine timed out on every call. The v4 result was detect: { candidates: 30, written: 30, errors: 30 }, Errata.ts daily exited 0, and the errata-daily workflow run was recorded as succeeded. No Vigil event was raised. We found it two days later by reading the Assay ledger. The errored rows were retried on the next run, so no data was lost. What was lost is the signal that the engine was down.

The fix for #2543 moves further the same way: a caller with no registry row will report skipped and exit 0. That case is a config choice. An engine that is configured and answers nothing is an outage, and it looks the same from the outside.

Repro

  1. Register the errata-triage caller, then make the engine unreachable (for example, block api.cloudflare.com).
  2. bun LIFEOS/ERRATA/Errata.ts daily --json
  3. stages.v4.detect.errors equals candidates; exit code is 0.

Suggested fix

Keep single errors non-fatal. Treat the v4 stage as failed when it threw, or when detect or diagnose errored on at least half of its candidates. Locally we do this with a second workflow step that reads stages.v4 and exits 1 on that condition; a failed run then raises the normal Vigil event.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions