Repository navigation
fix(youtube): log ScrapeCreators transcript rescue instead of masking it - #851
Conversation
Greptile SummaryThis PR makes ScrapeCreators transcript rescues visible in YouTube fetch logs and summaries. The main changes are:
Confidence Score: 5/5This looks safe to merge.
|
| Filename | Overview |
|---|---|
| skills/last30days/scripts/lib/youtube_yt.py | Adds accurate rescue logging, per-video rescue tracking, and rescue counts in batch summaries. |
| tests/test_youtube_yt.py | Adds tests for rescue logging, status tracking, batch reporting, and the unchanged normal summary. |
| changelog.d/851.fixed.md | Documents the newly visible ScrapeCreators transcript rescue behavior. |
Reviews (4): Last reviewed commit: "docs(changelog): add towncrier fragment ..." | Re-trigger Greptile
|
Good catch on the attribution — the fallback can indeed fire when yt-dlp was never attempted (not installed) and only the direct HTTP path failed. Switched both messages to neutral wording in the latest commit: the per-video log now says the rescue happened "after the keyless fetch cascade failed", and the batch summary drops the yt-dlp clause entirely ( |
… it (mvanhorn#831) _sc_fetch_transcript() had no logging on its success path. When yt-dlp failed for every video in a batch (rate-limited or bot-gated) and the ScrapeCreators fallback silently rescued them, fetch_transcripts_parallel() printed "Got transcripts for N/N videos (0 failed)" with zero trace of the fallback ever firing — making a fully rate-limited run read as a clean success. Log the rescue in fetch_transcript() and flag it via status['sc_rescued'], then fold that count into the batch summary so a masked failure can't be misread as "0 failed": the line now reads "N/N videos (M failed, K rescued via ScrapeCreators fallback after yt-dlp failure)" whenever the fallback was needed, unchanged otherwise.
The SC fallback can fire when yt-dlp was never attempted (not installed) and only the direct HTTP path failed, so attributing the rescue to a yt-dlp failure could mislead operators about the cause. Attribute it to the keyless fetch cascade instead, and drop the yt-dlp clause from the batch summary. Addresses the review note.
|
Maintainer update: rebased onto |
910aa3b to
f60d477
Compare
Summary
_sc_fetch_transcript()had no logging on its success path. When yt-dlp fails for every video in a batch (rate-limited or bot-gated) and the ScrapeCreators fallback silently rescues them,fetch_transcripts_parallel()printedGot transcripts for N/N videos (0 failed)with zero trace of the fallback ever firing — making a fully rate-limited run read as a clean success. Reproduced the exact log pattern from the issue (six yt-dlp timeout/failure lines followed by a "0 failed" summary, no ScrapeCreators-related log line) with a scripted repro.Change
lib/youtube_yt.py— log the rescue infetch_transcript()and flag it viastatus['sc_rescued']; fold that count into the batch summary infetch_transcripts_parallel()so a masked failure can't be misread as a clean success. The line now readsN/N videos (M failed, K rescued via ScrapeCreators fallback after yt-dlp failure)when the fallback fired, unchanged otherwise.Tests
Added 4 tests to
TestScTranscriptFallbackcovering the rescue log/flag, the summary's rescue reporting, and the unchanged-format regression when no fallback is used. Full suite: 3168 passed, 5 skipped, coverage 86.84% (gate 84%).Fixes #831.