fix(youtube): date a video by its upload date by Kikobeats · Pull Request #869 · microlinkhq/metascraper

Kikobeats · 2026-08-10T16:33:30Z

The bug

$jsonld('dateCreated') searches the JSON-LD graph recursively, so on a schema.org VideoObject it descends into comment[] and returns a comment's timestamp. YouTube emits no top-level datePublished, so every YouTube URL was dated by whichever comment the page happened to embed.

https://www.youtube.com/watch?v=arj7oStGLkU

date: 2020-05-11T19:29:04.000Z   <- comment[0].dateCreated
date: 2016-04-06T16:59:35.000Z   <- uploadDate  (correct)

Reproducible on the live page:

$jsonld('dateCreated')($)  // => 2020-05-11T12:29:04-07:00
$jsonld('uploadDate')($)   // => 2016-04-06T09:59:35-07:00

Found while auditing metadata quality for a citation-generator customer, where a wrong publication date is worse than no date at all.

The fix

uploadDate is the publication date of a VideoObject, so read it before falling back to dateCreated, and accept the itemprop form YouTube also emits. Order is otherwise untouched: dateModified and datePublished still win, and a top-level dateCreated is still used when nothing better exists — all four covered by tests.

Why it went unnoticed

Two gaps, both closed here:

metascraper-date shipped with "test": "exit 0" and no test suite. Added one.
metascraper-youtube's test destructured date out of the snapshot and asserted only typeof date === 'string', so any wrong date passed. Pinned it.

Reverting src/index.js makes the new test fail with exactly the production value:

- '2020-05-11T19:29:04.000Z'
+ '2016-04-06T16:59:35.000Z'

Verification

5 new tests in metascraper-date, all passing
metascraper-youtube 8/8 passing with the date now pinned in the snapshot
Full monorepo suite run: only pre-existing network-dependent failures remain (metascraper-logo-favicon DNS, metascraper audiense favicon fallback). Both verified to fail identically on master without this change.
End to end on the live 1.3 MB YouTube page: date: '2016-04-06T16:59:35.000Z'

Known follow-up (not in this PR)

The underlying $jsonld recursion into comment[] / review[] is still there. With uploadDate ahead of it YouTube is fixed, but a VideoObject that omits uploadDate and carries comments could still be mis-dated. Containing that means changing searchSchemaResults traversal in @metascraper/helpers, which is a wider blast radius and deserves its own PR.

🤖 Generated with Claude Code

https://claude.ai/code/session_01DiDJXMPg2e98kgL3drjg6f

Note

Low Risk
Scoped change to YouTube date extraction with regression tests; no auth, security, or broad behavioral changes beyond correct dates on YouTube URLs.

Overview
Fixes YouTube watch pages being dated by embedded comment timestamps instead of the video’s upload time when generic JSON-LD dateCreated walks into comment[].

metascraper-youtube now defines a date rule that reads itemprop="uploadDate" (via the shared date helper) so the YouTube plugin supplies the correct publication date for video URLs.

Adds a regression fixture (VideoObject JSON-LD plus nested comments) and a test that expects 2016-04-06T16:59:35.000Z. The image-size snapshot includes date instead of only checking typeof date === 'string'.

^{Reviewed by Cursor Bugbot for commit 8013fd9. Bugbot is set up for automated code reviews on this repo. Configure here.}

Summary by CodeRabbit

Bug Fixes
- Improved metadata extraction to use the primary content’s publication date instead of dates from comments.
- Prevented comment-level authors and timestamps from being incorrectly promoted to top-level metadata.
- Preserved the correct article or video author information.
Tests
- Added regression coverage for YouTube video metadata and nested comment data.

coderabbitai · 2026-08-10T16:34:07Z

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

@coderabbitai resume to resume automatic reviews.
@coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

▶️ Resume reviews
🔍 Trigger review

📝 Walkthrough

Walkthrough

The JSON-LD helper now ignores comment properties during fuzzy traversal. YouTube tests add a comment-bearing fixture and verify that metadata uses the video upload date.

Changes

YouTube date extraction coverage

Layer / File(s)	Summary
Audience metadata filtering `packages/metascraper-helpers/src/index.js`, `packages/metascraper-helpers/test/index.js`	Fuzzy JSON-LD traversal skips `comment` properties. Regression tests exclude comment-level `dateCreated` and `author` values while preserving the article author.
YouTube upload-date regression coverage `packages/metascraper-youtube/test/fixtures/youtube-video-comments.html`, `packages/metascraper-youtube/test/index.js`, `packages/metascraper-youtube/test/snapshots/index.js.md`	A comment-bearing YouTube fixture verifies upload-date selection. The complete metadata snapshot now includes the extracted video date.

Estimated code review effort: 2 (Simple) | ~10 minutes

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the main fix: using a video's upload date instead of its newest comment date.

✨ Finishing Touches

📝 Generate docstrings

Create stacked PR
Commit on current branch

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch fix/video-upload-date

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

coderabbitai

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@packages/metascraper-date/src/index.js`:
- Around line 25-29: Reorder the date extraction rules so the HTML
itemprop="uploaddate" content and title checks run before the dateCreated
fallback, while keeping the JSON-LD uploadDate rule adjacent to them. Add a
regression test covering nested JSON-LD Comment.dateCreated alongside an HTML
itemprop="uploadDate", asserting the HTML upload date is selected.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 8dfec389-7cce-4856-ac20-696b9c5b486e

📥 Commits

Reviewing files that changed from the base of the PR and between c6fdffe and f9d18bf.

⛔ Files ignored due to path filters (1)

packages/metascraper-youtube/test/snapshots/index.js.snap is excluded by !**/*.snap

📒 Files selected for processing (6)

packages/metascraper-date/package.json
packages/metascraper-date/src/index.js
packages/metascraper-date/test/helpers.js
packages/metascraper-date/test/index.js
packages/metascraper-youtube/test/index.js
packages/metascraper-youtube/test/snapshots/index.js.md

Kikobeats · 2026-08-10T16:44:26Z

Pushed a second commit covering metascraper-youtube itself.

The package declared no date rule at all, so YouTube fell through to metascraper-date. Since vendor rules run before the generic ones (vendorRules → basicRules in the API's rule composition), metascraper-youtube is the package that should decide this — and the generic path is still poisonable by the same class of bug, just through a different property:

{
  '@type': 'VideoObject',
  uploadDate: '2016-04-06T09:59:35-07:00',
  comment: [{ '@type': 'Comment', dateModified: '2024-01-01T00:00:00.000Z' }]
}

// metascraper-date alone => 2024-01-01T00:00:00.000Z

date tries dateModified before datePublished, and $jsonld searches the graph recursively, so a comment's dateModified outranks the video's uploadDate. The first commit only closed the dateCreated door.

Two tests added, and both were verified to fail when the rule is removed:

the real watch-page JSON-LD (comment and all) as a fixture
the dateModified hardening case above, which drops to 2024-01-01 without the vendor rule

metascraper-youtube 10/10 passing. Full monorepo suite unchanged: only the two pre-existing network failures.

coderabbitai

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@packages/metascraper-youtube/test/fixtures/youtube-video-comments.html`:
- Around line 18-25: Sanitize the comment fixture’s user-generated data by
removing or replacing the YouTube handle, display name, profile URL, full
comment text, and upvoteCount while preserving the comment object and timestamp
fields required by the date regression. Confirm the replacement content complies
with the repository privacy policy.

In `@packages/metascraper-youtube/test/index.js`:
- Around line 24-32: Add a separate YouTube fixture containing only an
itemprop="uploadDate" value, with no JSON-LD or competing metadata, and add a
dedicated test alongside the existing test that loads it and asserts the
expected date. Keep the current JSON-LD/comments scenario unchanged so the
itemprop selector is verified independently.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 99c2e441-07d2-4568-9b41-9928394ac313

📥 Commits

Reviewing files that changed from the base of the PR and between f9d18bf and 2f92dee.

📒 Files selected for processing (3)

packages/metascraper-youtube/src/index.js
packages/metascraper-youtube/test/fixtures/youtube-video-comments.html
packages/metascraper-youtube/test/index.js

coderabbitai

🧹 Nitpick comments (1)

packages/metascraper-helpers/test/index.js (1)
453-483: 🗄️ Data Integrity & Integration | 🔵 Trivial | ⚡ Quick win

Cover the downstream author.name query.

packages/metascraper-author/src/index.js calls $jsonld('author.name'), but this test only checks $jsonld('author'). Add assertions for the property path used by the author scraper.
Suggested test additions
     t.is($jsonld('author')($), undefined)
+    t.is($jsonld('author.name')($), undefined)

     t.deepEqual($jsonld('author')($), { name: 'the writer' })
+    t.is($jsonld('author.name')($), 'the writer')
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/metascraper-helpers/test/index.js` around lines 453 - 483, Add
assertions in the existing `$jsonld does not answer with what an audience said`
test for `$jsonld('author.name')`, covering the reviewer/comment author cases
and the article’s writer author path used by
`packages/metascraper-author/src/index.js`. Preserve the current
`$jsonld('author')` assertions.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@packages/metascraper-helpers/test/index.js`:
- Around line 453-483: Add assertions in the existing `$jsonld does not answer
with what an audience said` test for `$jsonld('author.name')`, covering the
reviewer/comment author cases and the article’s writer author path used by
`packages/metascraper-author/src/index.js`. Preserve the current
`$jsonld('author')` assertions.

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 9f5b95fe-66b3-47ef-add3-fc7c539d2b58

📥 Commits

Reviewing files that changed from the base of the PR and between 961543a and cbc1fbc.

📒 Files selected for processing (7)

packages/metascraper-date/src/index.js
packages/metascraper-date/test/index.js
packages/metascraper-helpers/src/index.js
packages/metascraper-helpers/test/index.js
packages/metascraper-youtube/test/fixtures/youtube-video-comments.html
packages/metascraper-youtube/test/fixtures/youtube-video-itemprop-only.html
packages/metascraper-youtube/test/index.js

🚧 Files skipped from review as they are similar to previous changes (2)

packages/metascraper-youtube/test/fixtures/youtube-video-comments.html
packages/metascraper-date/src/index.js

cursor

Cursor Bugbot has reviewed your changes and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit e5c5335. Configure here.}

metascraper-youtube owned title, author, publisher and image but left date to the generic rules, where `$jsonld('dateCreated')` searches the JSON-LD tree recursively, walks into `comment[]` and answers with a viewer's timestamp: https://www.youtube.com/watch?v=arj7oStGLkU 2020-05-11T19:29:04.000Z <- a comment 2016-04-06T16:59:35.000Z <- the upload Declare `date` here like the other fields, read from the `uploadDate` the watch page carries. Vendor rules run before the generic ones, so YouTube no longer depends on how the shared date rules are ordered. The fixture is the watch page's own JSON-LD, comment included, with the commenter's details replaced. Removing the rule fails it with the exact production value. The snapshot test alongside it destructured `date` out and asserted only that it was a string, so it could not have caught this; pin it. The recursive search reaching into comments is a wider bug that also affects `review` nodes. It belongs in its own change, against @metascraper/helpers. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01DiDJXMPg2e98kgL3drjg6f

coveralls · 2026-08-10T18:55:58Z

coverage: 94.689%. first build — fix/video-upload-date into master

cursor Bot reviewed Aug 10, 2026

View reviewed changes

Comment thread packages/metascraper-date/src/index.js Resolved

coderabbitai Bot reviewed Aug 10, 2026

View reviewed changes

Comment thread packages/metascraper-date/src/index.js Resolved

cursor Bot reviewed Aug 10, 2026

View reviewed changes

Comment thread packages/metascraper-youtube/src/index.js Resolved

coderabbitai Bot reviewed Aug 10, 2026

View reviewed changes

Comment thread packages/metascraper-youtube/test/fixtures/youtube-video-comments.html Resolved

Comment thread packages/metascraper-youtube/test/index.js Resolved

coderabbitai Bot reviewed Aug 10, 2026

View reviewed changes

cursor Bot reviewed Aug 10, 2026

View reviewed changes

Comment thread packages/metascraper-helpers/src/index.js Resolved

Kikobeats force-pushed the fix/video-upload-date branch from 057fd4f to 74b0bae Compare August 10, 2026 18:36

Kikobeats force-pushed the fix/video-upload-date branch from 74b0bae to 8013fd9 Compare August 10, 2026 18:50

Kikobeats changed the title ~~fix(date): date a video by its upload date, not by its newest comment~~ fix(youtube): date a video by its upload date Aug 10, 2026

Kikobeats merged commit 7040d5c into master Aug 10, 2026
58 checks passed

Kikobeats deleted the fix/video-upload-date branch August 10, 2026 18:55

Navigation Menu

Repository navigation

fix(youtube): date a video by its upload date - #869

fix(youtube): date a video by its upload date#869
Kikobeats merged 1 commit into
masterfrom
fix/video-upload-date

Kikobeats commented Aug 10, 2026 •

edited by cursor Bot

Loading

Uh oh!

coderabbitai Bot commented Aug 10, 2026 •

edited

Loading

Reviews paused

Walkthrough

Changes

Uh oh!

Uh oh!

coderabbitai Bot left a comment

Uh oh!

Uh oh!

Kikobeats commented Aug 10, 2026

Uh oh!

Uh oh!

coderabbitai Bot left a comment

Uh oh!

Uh oh!

Uh oh!

coderabbitai Bot left a comment

Uh oh!

cursor Bot left a comment

Uh oh!

Uh oh!

Uh oh!

coveralls commented Aug 10, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Kikobeats commented Aug 10, 2026 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

The bug

The fix

Why it went unnoticed

Verification

Known follow-up (not in this PR)

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Aug 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reviews paused

Walkthrough

Changes

Uh oh!

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Kikobeats commented Aug 10, 2026

Uh oh!

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

coveralls commented Aug 10, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Kikobeats commented Aug 10, 2026 •

edited by cursor Bot

Loading

coderabbitai Bot commented Aug 10, 2026 •

edited

Loading