Skip to content

Commit 2d72298

Browse files
Kikobeatsclaude
andauthored
feat(helpers): read the declared protocol instead of resolving it (#864)
* perf(helpers): read protocol without a thrown URL construction `protocol` resolved the scheme through `new URL` inside a try/catch, so every schemeless value paid for a constructed and thrown exception to arrive at ''. Relative hrefs are the common input for callers walking a document's links. A value with no scheme can never parse without a base, so it resolves to '' without touching the URL parser. Anything carrying a scheme still goes through `new URL`, which keeps the cases where parsing the whole URL is what decides the answer: `new URL('http://')` throws, so `protocol('http://')` stays ''. 100k calls, best of 3: relative hrefs 299.3ms -> 1.9ms 155x mixed 162.2ms -> 8.5ms 19x absolute http 14.2ms -> 15.4ms 0.9x Inputs that do carry a scheme pay one extra regex test, about 8% on a ~140ns call, since the URL construction happens either way. Output is unchanged. Verified by differential fuzz against the previous implementation over 200,074 inputs, including leading control characters, tabs and newlines inside the scheme, non-strings, and URLs the parser rejects outright: no mismatches. * feat(helpers): read the declared protocol instead of resolving it `protocol` resolved the scheme through `new URL` inside a try/catch, so a value without a scheme constructed a URL, threw, and got caught on the way to ''. Relative hrefs are the common input for callers walking a document's links, which put the exception on the hot path. Which scheme a value declares is a lexical question, so it is now read directly. 100k calls, best of 3: relative hrefs 299.4ms -> 2.6ms 116x mixed 157.1ms -> 3.6ms 44x absolute http 14.7ms -> 4.0ms 3.7x non-http schemes 13.5ms -> 4.3ms 3.2x The scheme is read after the cleanup the URL parser applies before reading it itself: a leading C0 control or space is skipped and every ASCII tab or newline is removed, so ' mai\tlto:a@b' still declares `mailto`. Output changes for one class of input: a URL whose authority the parser rejects. protocol('http://') '' -> 'http' protocol('http://[') '' -> 'http' protocol('foo://[') '' -> 'foo' '' described the whole URL rather than the scheme sitting in front of it. Whether a URL is usable is what `isUrl` answers. Verified by differential fuzz against the previous implementation over 300,079 inputs, including leading control characters, tabs and newlines inside the scheme, and non-strings: 300,070 identical, 9 divergences, every one of them the class above, and none outside it. * perf(helpers): read the protocol in a single scan The regex ran over a value that a trim loop and three `includes` scans had already walked. One charCode pass answers it, with the tab/newline removal demoted to the rare path that needs it. 100k calls per set, best of 5: | input | before | after | | |-------------------|--------|-------|-------| | relative hrefs | 2.52ms | 0.80ms| 3.2x | | mixed | 3.20ms | 1.25ms| 2.6x | | absolute http | 4.64ms | 1.89ms| 2.5x | | non-http schemes | 4.27ms | 1.80ms| 2.4x | | tabs / leading ws | 8.52ms | 6.58ms| 1.3x | Differential fuzz against the previous implementation over 500,044 inputs: 0 mismatches. `normalizeUrl` asked the same question through `new URL`, so it now reads the helper and drops a URL construction per call. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_014CoQ8BQTgwWhjmLpS8jqTE * perf(helpers): inline the scheme character classes V8 was not inlining `isSchemeHead`/`isSchemeChar`, and `toLowerCase` was called on schemes that were already lowercase. Both are now decided in the loop that is already reading the characters. 100k calls per set, best of 9, against a noise floor of +/-15% measured with a byte-identical clone of the previous body: | input | before | after | | |------------------|--------|--------|-------| | mixed | 1.56ms | 1.16ms | -25% | | absolute http | 1.88ms | 1.48ms | -22% | | non-http schemes | 1.79ms | 1.42ms | -21% | Differential fuzz against both earlier implementations over 500,044 inputs: 0 mismatches. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_014CoQ8BQTgwWhjmLpS8jqTE * refactor(helpers): drop the single-use temporaries in protocol `isLower` was an alias for the flag's initial value and `isDigit` named a class that every neighbouring line inlines. Measured against the previous body: within the +/-15% noise floor on every input class. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_014CoQ8BQTgwWhjmLpS8jqTE * test(helpers): give the strip comment a subject Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_014CoQ8BQTgwWhjmLpS8jqTE --------- Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
1 parent e66c71f commit 2d72298

2 files changed

Lines changed: 75 additions & 6 deletions

File tree

‎packages/metascraper-helpers/src/index.js‎

Lines changed: 34 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -123,6 +123,8 @@ const REGEX_LOCATION = /^[A-Z\s]+\s+[-—–]\s+/
123123

124124
const REGEX_TITLE_SEPARATOR = /^[^|\-/•—]+/
125125

126+
const REGEX_URL_TAB_OR_NEWLINE = /[\t\n\r]/g
127+
126128
const AUTHOR_MAX_LENGTH = 128
127129

128130
const removeLocation = value => replace(value, REGEX_LOCATION, '')
@@ -143,6 +145,37 @@ const absoluteUrl = (baseUrl, relativePath) => {
143145
return urlObject(relativePath, baseUrl).toString()
144146
}
145147

148+
// https://url.spec.whatwg.org/#scheme-state
149+
const protocol = url => {
150+
if (!isString(url)) return ''
151+
const { length } = url
152+
let start = 0
153+
while (start < length && url.charCodeAt(start) <= 0x20) start++
154+
const head = url.charCodeAt(start)
155+
let isLowerCased = head >= 0x61 && head <= 0x7a
156+
if (!isLowerCased && !(head >= 0x41 && head <= 0x5a)) return ''
157+
for (let index = start + 1; index < length; index++) {
158+
const code = url.charCodeAt(index)
159+
if (code === 0x3a) {
160+
const scheme = url.slice(start, index)
161+
return isLowerCased ? scheme : scheme.toLowerCase()
162+
}
163+
if (code >= 0x61 && code <= 0x7a) continue
164+
if (code >= 0x41 && code <= 0x5a) {
165+
isLowerCased = false
166+
continue
167+
}
168+
if (code >= 0x30 && code <= 0x39) continue
169+
if (code === 0x2b || code === 0x2d || code === 0x2e) continue
170+
// `mai\tlto:a@b` declares `mailto`
171+
if (code === 0x09 || code === 0x0a || code === 0x0d) {
172+
return protocol(url.replace(REGEX_URL_TAB_OR_NEWLINE, ''))
173+
}
174+
return ''
175+
}
176+
return ''
177+
}
178+
146179
const sanetizeUrl = (url, opts) =>
147180
_normalizeUrl(url, {
148181
stripWWW: false,
@@ -156,7 +189,7 @@ const normalizeUrl = (baseUrl, relativePath, opts) => {
156189
try {
157190
const absolute = absoluteUrl(baseUrl, relativePath)
158191
// normalize-url v9 no longer rejects `javascript:` URLs; keep them out
159-
if (urlObject(absolute).protocol === 'javascript:') return undefined
192+
if (protocol(absolute) === 'javascript') return undefined
160193
return sanetizeUrl(absolute, opts)
161194
} catch (_) {}
162195
}
@@ -205,11 +238,6 @@ const isAuthor = (str, opts = { relative: false }) =>
205238
const getAuthor = (str, { removeBy = true, ...opts } = {}) =>
206239
titleize(str, { removeBy, ...opts })
207240

208-
const protocol = url => {
209-
const { protocol = '' } = urlObject(url)
210-
return protocol.replace(':', '')
211-
}
212-
213241
const isExtension = (url, type, ext = extension(url)) =>
214242
type === EXTENSIONS[ext]
215243

‎packages/metascraper-helpers/test/index.js‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,7 @@ const {
2727
lang,
2828
normalizeUrl,
2929
parseUrl,
30+
protocol,
3031
url,
3132
video
3233
} = require('../src')
@@ -64,6 +65,46 @@ test('.parseUrl', t => {
6465
t.true(measure(fn) > measure(fn)) // eslint-disable-line
6566
})
6667

68+
test('.protocol', t => {
69+
t.is(protocol('http://example.com'), 'http')
70+
t.is(protocol('HTTPS://Example.com'), 'https')
71+
t.is(protocol('mailto:[email protected]'), 'mailto')
72+
t.is(protocol('MAILTO:[email protected]'), 'mailto')
73+
t.is(protocol('tel:+34123456789'), 'tel')
74+
t.is(protocol('javascript:alert(1)'), 'javascript')
75+
t.is(protocol('data:text/plain,hello'), 'data')
76+
t.is(protocol('ftp://example.com/file.txt'), 'ftp')
77+
t.is(protocol('file:///etc/passwd'), 'file')
78+
t.is(protocol('urn:isbn:0451450523'), 'urn')
79+
t.is(protocol('blob:https://example.com/uuid'), 'blob')
80+
81+
t.is(protocol('/path/to/page'), '')
82+
t.is(protocol('page.html'), '')
83+
t.is(protocol('../up'), '')
84+
t.is(protocol('//example.com/path'), '')
85+
t.is(protocol('#fragment'), '')
86+
t.is(protocol(''), '')
87+
88+
/* the declared scheme is read, even when the rest is not a URL */
89+
t.is(protocol('example.com:8080/path'), 'example.com')
90+
t.is(protocol('http://'), 'http')
91+
t.is(protocol('http://['), 'http')
92+
t.is(protocol('foo://['), 'foo')
93+
94+
/* whitespace and tabs are stripped before the scheme is read */
95+
t.is(protocol(' mailto:[email protected]'), 'mailto')
96+
t.is(protocol('\tmailto:[email protected]'), 'mailto')
97+
t.is(protocol('mai\tlto:[email protected]'), 'mailto')
98+
t.is(protocol('ht\ntp://example.com'), 'http')
99+
100+
t.is(protocol('1http://example.com'), '')
101+
t.is(protocol('+http://example.com'), '')
102+
103+
t.is(protocol(undefined), '')
104+
t.is(protocol(null), '')
105+
t.is(protocol(0), '')
106+
})
107+
67108
test('.normalizeUrl', t => {
68109
t.is(normalizeUrl('https://example.com', 'javascript:false'), undefined)
69110
t.is(

0 commit comments

Comments
 (0)