Skip to content

fix(youtube): log ScrapeCreators transcript rescue instead of masking it - #851

Merged
tmchow merged 3 commits into
mvanhorn:mainfrom
amitvijapur:fix/youtube-failure-count
Jul 25, 2026
Merged

tmchow merged 3 commits into
mvanhorn:mainfrom
amitvijapur:fix/youtube-failure-count

Conversation

@amitvijapur

Copy link
Copy Markdown
Contributor

Summary

_sc_fetch_transcript() had no logging on its success path. When yt-dlp fails for every video in a batch (rate-limited or bot-gated) and the ScrapeCreators fallback silently rescues them, fetch_transcripts_parallel() printed Got transcripts for N/N videos (0 failed) with zero trace of the fallback ever firing — making a fully rate-limited run read as a clean success. Reproduced the exact log pattern from the issue (six yt-dlp timeout/failure lines followed by a "0 failed" summary, no ScrapeCreators-related log line) with a scripted repro.

Change

lib/youtube_yt.py — log the rescue in fetch_transcript() and flag it via status['sc_rescued']; fold that count into the batch summary in fetch_transcripts_parallel() so a masked failure can't be misread as a clean success. The line now reads N/N videos (M failed, K rescued via ScrapeCreators fallback after yt-dlp failure) when the fallback fired, unchanged otherwise.

Tests

Added 4 tests to TestScTranscriptFallback covering the rescue log/flag, the summary's rescue reporting, and the unchanged-format regression when no fallback is used. Full suite: 3168 passed, 5 skipped, coverage 86.84% (gate 84%).

Fixes #831.

@greptile-apps

greptile-apps Bot commented Jul 21, 2026 •

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR makes ScrapeCreators transcript rescues visible in YouTube fetch logs and summaries. The main changes are:

  • Logs each successful ScrapeCreators rescue with neutral keyless-cascade wording.
  • Tracks rescued videos in per-video status data.
  • Reports the rescue count in batch summaries.
  • Adds tests for rescued and ordinary success paths.
  • Adds a changelog fragment for the fix.

Confidence Score: 5/5

This looks safe to merge.

  • The rescue wording now covers both yt-dlp and direct HTTP failures accurately.
  • Fresh status dictionaries prevent rescue flags from leaking between videos or batch calls.
  • Tests cover fallback rescues and the unchanged no-fallback output.

Important Files Changed

Filename Overview
skills/last30days/scripts/lib/youtube_yt.py Adds accurate rescue logging, per-video rescue tracking, and rescue counts in batch summaries.
tests/test_youtube_yt.py Adds tests for rescue logging, status tracking, batch reporting, and the unchanged normal summary.
changelog.d/851.fixed.md Documents the newly visible ScrapeCreators transcript rescue behavior.

Reviews (4): Last reviewed commit: "docs(changelog): add towncrier fragment ..." | Re-trigger Greptile

@amitvijapur

Copy link
Copy Markdown
Contributor Author

Good catch on the attribution — the fallback can indeed fire when yt-dlp was never attempted (not installed) and only the direct HTTP path failed. Switched both messages to neutral wording in the latest commit: the per-video log now says the rescue happened "after the keyless fetch cascade failed", and the batch summary drops the yt-dlp clause entirely (N/N videos (M failed, K rescued via ScrapeCreators fallback)). Tests updated where needed and passing.

amitvijapur and others added 3 commits July 24, 2026 18:07
… it (mvanhorn#831)

_sc_fetch_transcript() had no logging on its success path. When yt-dlp
failed for every video in a batch (rate-limited or bot-gated) and the
ScrapeCreators fallback silently rescued them, fetch_transcripts_parallel()
printed "Got transcripts for N/N videos (0 failed)" with zero trace of the
fallback ever firing — making a fully rate-limited run read as a clean
success.

Log the rescue in fetch_transcript() and flag it via status['sc_rescued'],
then fold that count into the batch summary so a masked failure can't be
misread as "0 failed": the line now reads "N/N videos (M failed, K rescued
via ScrapeCreators fallback after yt-dlp failure)" whenever the fallback
was needed, unchanged otherwise.
The SC fallback can fire when yt-dlp was never attempted (not
installed) and only the direct HTTP path failed, so attributing the
rescue to a yt-dlp failure could mislead operators about the cause.
Attribute it to the keyless fetch cascade instead, and drop the
yt-dlp clause from the batch summary. Addresses the review note.
@tmchow

tmchow commented Jul 25, 2026

Copy link
Copy Markdown
Owner

Maintainer update: rebased onto origin/main so Changelog guard (guard) re-runs after #878.

@tmchow
tmchow force-pushed the fix/youtube-failure-count branch from 910aa3b to f60d477 Compare July 25, 2026 01:07
@tmchow
tmchow merged commit 3d33d97 into mvanhorn:main Jul 25, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

YouTube: transcript fetch logs "N/N videos (0 failed)" despite every individual fetch failing; ScrapeCreators backstop never fires

2 participants