Skip to content

Commit d40856b

Browse files
Kikobeatscursoragentclaude
authored
feat(helpers): add rule-level validate (like test) (#866)
* feat(helpers): add per-call validate hook to findRule Allow callers to reject a rule result and continue to the next rule without throwing, so logo XSS fallthrough can skip hostile candidates. Co-authored-by: Cursor <[email protected]> * fix(helpers): scope validate hook and forward it through nested rules `validate` applied to every property, so a validator written for one field silently nulled unrelated ones. It now accepts an object keyed by property name; the function form still runs for every property. A throwing validator aborted the whole property through get-data's catch. It now rejects only the candidate that triggered it and the search resumes at the next rule, with the error logged under DEBUG=metascraper:find-rule. metascraper-video and metascraper-audio rebuilt the args object for their nested findRule calls, so validate never reached rules behind an iframe and per-candidate fallthrough did not happen there. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01VSEyFT3dw16tyvhRGVAWrx * refactor(video,audio): keep param destructuring in iframe rules Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01VSEyFT3dw16tyvhRGVAWrx * refactor(helpers): share withIframe for audio/video Deduplicate nested iframe probing so validate keeps one code path, and tighten docs/tests from review feedback. Co-authored-by: Cursor <[email protected]> * refactor(helpers): move validate onto rules like test Scrape-time validate mixed a second control plane into the call. Bundle/rule validate matches test and needs no args forwarding. Co-authored-by: Cursor <[email protected]> * fix(metascraper): keep pkgName on inline mergeRules loadRules already copied bundle pkgName; mergeRules dropped it so inline bundles lost debug names and pkgName could become a prop key. Co-authored-by: Cursor <[email protected]> * feat(helpers): pass find-rule debug logger to validate Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01VSEyFT3dw16tyvhRGVAWrx * refactor: simplify validate plumbing and its tests Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01VSEyFT3dw16tyvhRGVAWrx * refactor: drop redundant meta guard and duplicated iframe tests Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01VSEyFT3dw16tyvhRGVAWrx * test(helpers): require package entry like sibling tests Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01VSEyFT3dw16tyvhRGVAWrx --------- Co-authored-by: Cursor <[email protected]> Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
1 parent 9c5e77b commit d40856b

13 files changed

Lines changed: 682 additions & 129 deletions

File tree

‎CONTRIBUTING.md‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -58,6 +58,37 @@ const rules = []
5858
rules.test = ({ url }) => test(url)
5959
```
6060

61+
### Defining `validate` function
62+
63+
You can associate a `validate` function with a single rule or with the whole rules bundle (copied onto every rule, like `test`):
64+
65+
```js
66+
rules.validate = async (value, { url }) =>
67+
typeof value !== 'string' || !value.startsWith('data:')
68+
```
69+
70+
Or per rule:
71+
72+
```js
73+
Object.assign(() => 'https://example.com/logo.png', {
74+
validate: async value => !(await isHostile(value))
75+
})
76+
```
77+
78+
`validate` runs after a rule produces a value. Returning a falsy value (or throwing) discards that candidate and `findRule` continues with the next rule for the property — exactly as if the rule had returned nothing. If every rule is rejected, the property is `null`.
79+
80+
The first argument is the candidate value; the second is the same args object passed to the rule / `test`; the third is the `metascraper:find-rule` logger, so a rejection can explain itself:
81+
82+
```js
83+
rules.validate = (value, { url }, debug) => {
84+
if (!value.startsWith('data:')) return true
85+
debug('logo:rejected', { url, reason: 'data-uri' })
86+
return false
87+
}
88+
```
89+
90+
Set `DEBUG=metascraper:find-rule` to see those lines, plus the rejections metascraper logs on its own.
91+
6192
### Defining `pkgName` property
6293

6394
Additionally you can define `pkgName` property associated with your rules:

‎packages/metascraper-audio/src/index.js‎

Lines changed: 3 additions & 34 deletions
Original file line numberDiff line numberDiff line change
@@ -6,10 +6,8 @@ const {
66
audio,
77
createGetIframeCached,
88
defaultGetIframe,
9-
findRule,
10-
has,
11-
normalizeUrl,
12-
toRule
9+
toRule,
10+
withIframe
1311
} = require('@metascraper/helpers')
1412

1513
const toAudio = toRule(audio)
@@ -71,36 +69,7 @@ const audioRules = [
7169
module.exports = ({ getIframe = defaultGetIframe } = {}) => {
7270
const getIframeCached = createGetIframeCached(getIframe)
7371
const rules = {
74-
audio: audioRules.concat(
75-
async ({ htmlDom: $, url }) => {
76-
const srcs = $('iframe[src^="http"], iframe[src^="/"]')
77-
.map((_, element) => $(element).attr('src'))
78-
.get()
79-
if (srcs.length === 0) return
80-
const seenSrcs = new Set()
81-
for (const src of srcs) {
82-
try {
83-
const normalizedSrc = normalizeUrl(url, src)
84-
if (!normalizedSrc) continue
85-
if (seenSrcs.has(normalizedSrc)) continue
86-
seenSrcs.add(normalizedSrc)
87-
88-
const htmlDom = await getIframeCached(url, $, normalizedSrc)
89-
const result = await findRule(audioRules, { htmlDom, url })
90-
if (has(result)) return result
91-
} catch (_) {}
92-
}
93-
},
94-
async ({ htmlDom: $, url }) => {
95-
const src = $('meta[name="twitter:player"]').attr('content')
96-
return src
97-
? findRule(audioRules, {
98-
htmlDom: await getIframeCached(url, $, src),
99-
url
100-
})
101-
: undefined
102-
}
103-
)
72+
audio: withIframe(audioRules, getIframeCached, 'audio')
10473
}
10574

10675
rules.pkgName = 'metascraper-audio'

‎packages/metascraper-audio/test/iframe.js‎

Lines changed: 24 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -5,8 +5,8 @@ const test = require('ava').default
55

66
const { runServer } = require('./helpers')
77

8-
const createMetascraper = (...args) =>
9-
require('metascraper')([require('../src')(...args)])
8+
const createMetascraper = (opts, meta) =>
9+
require('metascraper')([Object.assign(require('../src')(opts), meta)])
1010

1111
test('absolute http', async t => {
1212
const url = await runServer(t, ({ res }) => {
@@ -68,6 +68,28 @@ test('stop iframe probing after first audio match', async t => {
6868
t.deepEqual(calls, ['https://example.com/ok'])
6969
})
7070

71+
test('validate reaches the rules nested behind an iframe', async t => {
72+
const metascraper = createMetascraper(
73+
{
74+
getIframe: async (url, $, { src }) =>
75+
cheerio.load(
76+
`<meta property="og:audio" content="https://cdn.microlink.io${src.replace(
77+
'https://example.com',
78+
''
79+
)}.mp3">`
80+
)
81+
},
82+
{ validate: value => !value.endsWith('/hostile.mp3') }
83+
)
84+
85+
const metadata = await metascraper({
86+
url: 'https://example.com',
87+
html: '<iframe src="/hostile"></iframe><iframe src="/safe"></iframe>'
88+
})
89+
90+
t.is(metadata.audio, 'https://cdn.microlink.io/safe.mp3')
91+
})
92+
7193
test('dedupe normalized iframe urls while probing', async t => {
7294
let calls = 0
7395
const metascraper = createMetascraper({

‎packages/metascraper-helpers/src/index.js‎

Lines changed: 63 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -548,19 +548,44 @@ const validator = {
548548

549549
const truthyTest = () => true
550550

551-
const findRule = async (rules, args, propName) => {
551+
const isValidValue = async (rule, value, args) => {
552+
try {
553+
return await rule.validate(value, args, debug)
554+
} catch (error) {
555+
debug('validate:error', {
556+
pkgName: rule.pkgName,
557+
errorName: error?.name,
558+
errorMessage: error?.message
559+
})
560+
return false
561+
}
562+
}
563+
564+
const findRule = async (rules, args = {}, propName) => {
552565
let index = 0
553566
let value
567+
let hasValue = false
554568

555569
do {
556570
const rule = rules[index++]
557571
const test = rule.test || truthyTest
558572
if (test(args)) {
559573
const duration = debug.duration()
560574
value = await rule(args)
561-
duration(`${rule.pkgName}:${propName}:${index - 1}:${has(value)}`)
575+
hasValue = has(value)
576+
const isRejected =
577+
hasValue && rule.validate && !(await isValidValue(rule, value, args))
578+
if (isRejected) {
579+
value = undefined
580+
hasValue = false
581+
}
582+
duration(
583+
`${rule.pkgName}:${propName}:${index - 1}:${hasValue}${
584+
isRejected ? ':rejected' : ''
585+
}`
586+
)
562587
}
563-
} while (!has(value) && index < rules.length)
588+
} while (!hasValue && index < rules.length)
564589

565590
return value
566591
}
@@ -624,6 +649,39 @@ const createGetIframeCached = getIframe => {
624649
}
625650
}
626651

652+
const withIframe = (rules, getIframe, propName) => {
653+
const probe = async (src, args) => {
654+
try {
655+
return await findRule(
656+
rules,
657+
{ ...args, htmlDom: await getIframe(args.url, args.htmlDom, src) },
658+
propName
659+
)
660+
} catch (_) {}
661+
}
662+
663+
return rules.concat(async args => {
664+
const { htmlDom: $, url } = args
665+
const srcs = $('iframe[src^="http"], iframe[src^="/"]')
666+
.map((_, el) => $(el).attr('src'))
667+
.get()
668+
669+
if (srcs.length > 0) {
670+
const seen = new Set()
671+
for (const src of srcs) {
672+
const normalized = normalizeUrl(url, src)
673+
if (!normalized || seen.has(normalized)) continue
674+
seen.add(normalized)
675+
const value = await probe(normalized, args)
676+
if (has(value)) return value
677+
}
678+
}
679+
680+
const twitter = $('meta[name="twitter:player"]').attr('content')
681+
if (twitter) return probe(twitter, args)
682+
})
683+
}
684+
627685
module.exports = {
628686
$filter,
629687
$jsonld,
@@ -676,5 +734,6 @@ module.exports = {
676734
url,
677735
validator,
678736
video,
679-
videoExtensions
737+
videoExtensions,
738+
withIframe
680739
}
Lines changed: 96 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,96 @@
1+
'use strict'
2+
3+
const test = require('ava').default
4+
5+
const { findRule } = require('..')
6+
7+
const withValidate = (fn, validate) => Object.assign(fn, { validate })
8+
9+
test('args are optional', async t => {
10+
t.is(await findRule([() => 'value']), 'value')
11+
})
12+
13+
test('rules are evaluated until one produces a value', async t => {
14+
const calls = []
15+
const rule = (name, value) => () => {
16+
calls.push(name)
17+
return value
18+
}
19+
20+
t.is(
21+
await findRule([rule('first'), rule('second', 'value'), rule('third')]),
22+
'value'
23+
)
24+
t.deepEqual(calls, ['first', 'second'])
25+
})
26+
27+
test('a rule test skips the rule entirely', async t => {
28+
const skipped = Object.assign(() => 'skipped', { test: () => false })
29+
t.is(await findRule([skipped, () => 'value']), 'value')
30+
})
31+
32+
test('validate rejects a candidate and resumes at the next rule', async t => {
33+
t.is(
34+
await findRule([
35+
withValidate(
36+
() => 'bad',
37+
v => v === 'good'
38+
),
39+
withValidate(
40+
() => 'good',
41+
v => v === 'good'
42+
)
43+
]),
44+
'good'
45+
)
46+
})
47+
48+
test('validate rejecting every rule returns undefined', async t => {
49+
t.is(
50+
await findRule([
51+
withValidate(
52+
() => 'a',
53+
() => false
54+
),
55+
withValidate(
56+
() => 'b',
57+
() => false
58+
)
59+
]),
60+
undefined
61+
)
62+
})
63+
64+
test('a throwing validate rejects only that candidate', async t => {
65+
const value = await findRule([
66+
withValidate(
67+
() => 'a',
68+
v => {
69+
if (v === 'a') throw new Error('boom')
70+
}
71+
),
72+
withValidate(
73+
() => 'b',
74+
() => true
75+
)
76+
])
77+
78+
t.is(value, 'b')
79+
})
80+
81+
test('validate receives the find-rule debug logger', async t => {
82+
let debug
83+
await findRule([
84+
withValidate(
85+
() => 'value',
86+
(value, args, logger) => {
87+
debug = logger
88+
return true
89+
}
90+
)
91+
])
92+
93+
t.is(typeof debug, 'function')
94+
t.falsy(debug.enabled)
95+
t.notThrows(() => debug('validate:custom', { reason: 'test' }))
96+
})

0 commit comments

Comments
 (0)