Skip to content

check: treat an all-NOT_FOUND live test as a stale sample, not a broken channel - #909

Open
0xABCD01 wants to merge 1 commit into
Panniantong:mainfrom
0xABCD01:fix/bilibili-stale-sample
Open

0xABCD01 wants to merge 1 commit into
Panniantong:mainfrom
0xABCD01:fix/bilibili-stale-sample

Conversation

@0xABCD01

@0xABCD01 0xABCD01 commented Nov 3, 2026

Copy link
Copy Markdown

Fixes #889

agent-reach check bilibili reported read and subtitle as broken because
their live_test fixture (BV1GJ411x7h7) has been deleted upstream: every
backend answered NOT_FOUND, run_checks treated that like a capability
failure, and doctor flipped the channel to "broken".

A live test whose every backend answers NOT_FOUND only proves the sample
itself is gone; it says nothing about whether the capability works. This
change teaches run_checks that distinction, in the dispatcher rather
than in one channel, because the same failure mode applies to every
channel whose live_test names a specific resource (a video, a topic, a
repo, an account — see youtube, v2ex, github, twitter, instagram,
facebook, xueqiu).

What changes:

  • In run_checks, an action that failed on all backends with NOT_FOUND
    as the only error code is marked stale: its result rows keep
    passed=false and gain "stale": true, the action does not fail the
    check, and it is persisted as unverified rather than passed or failed.
    Doctor therefore no longer reports the channel broken, the stale
    actions stay out of verified_actions, and a deleted sample no longer
    counts as a regression when the action passed before.
  • check --text prints STALE for those rows and appends
    "; N skipped (live sample no longer exists)" to the summary. The exit
    code follows the remaining real results.
  • Genuine failures are unchanged: any other error code, an empty result,
    or a mix of NOT_FOUND with a real error still fails the action and
    marks the channel broken, and a NEED_LOGIN failure on a sibling action
    still reports needs_login.

The bilibili fixture itself is deliberately left in place: with stale
handling, a dead sample no longer poisons the verdict for any channel,
and swapping in a different video would only restart the clock until
that one is deleted too. The STALE rows keep the dead fixture visible so
it can be refreshed deliberately. The dynamic-sample alternative from
#889 was not chosen: it needs a new per-channel resolution hook in the
check path, adds a hot-list call (with its own failure modes) to every
bilibili check, and a random trending video is not guaranteed to have
subtitles, which would trade one false verdict for another.

Tests: new tests/test_check_stale_sample.py covers the stale case —
including end-to-end on the real BilibiliChannel with mocked transports
(bili-cli not_found envelope + API -404) — the regression interaction,
genuine and mixed failures, the NEED_LOGIN interaction, and the --text
rendering. On the unpatched code the five stale tests fail and the two
preservation tests pass; with this change all seven pass. Full suite:
530 passed. ruff check agent_reach tests clean.

Live check note, honestly: I ran agent-reach add bilibili and
agent-reach check bilibili --text on this machine before and after the
change. This network cannot complete a healthy bilibili check —
bili-cli's requests fail TLS negotiation (SSL WRONG_VERSION_NUMBER) and
the view endpoint answers HTTP 412 — so both runs report the same
environment failures (UPSTREAM_BROKEN, still FAIL, exit 1; attached
below). That also confirms the fix does not mask real connectivity
failures. The API itself is reachable from here (popular returns
code 0, and the deleted fixture returns Bilibili's 404 page). The
NOT_FOUND path is covered by the mocked-transport test above; on an
unrestricted network the expected output for bilibili today is
STALE rows for read/subtitle and "2/4 actions working; 2 skipped
(live sample no longer exists)" with exit 0.

…n channel

run_checks treated a live test whose sample was deleted upstream (every backend NOT_FOUND) like a capability failure, so check/doctor reported the channel broken. Such an action is now marked stale: it stays unverified instead of failing the check, while genuine and mixed failures still report broken.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

check bilibili falsely reports read/subtitle as broken: live_test fixture BV1GJ411x7h7 is gone (404)

1 participant