fix(sector-valuations): proxy Yahoo quoteSummary via Decodo curl egress#3134
Merged
Conversation
Yahoo's /v10/finance/quoteSummary returns HTTP 401 from Railway container IPs. Railway logs 2026-04-16 show all 12 sector ETFs failing every 5-min cron: [Sector] Yahoo quoteSummary XLK HTTP 401 (x12 per tick) [Market] Seeded 12/12 sectors, 0 valuations Add a curl-based proxy fallback that matches scripts/_yahoo-fetch.mjs: hit us.decodo.com (curl egress pool) NOT gate.decodo.com (CONNECT egress pool). Per the 2026-04-16 probe documented in _yahoo-fetch.mjs header, Yahoo blocks Decodo's CONNECT egress IPs but accepts the curl egress. Reusing ytFetchViaProxy here would keep failing silently. Shares the existing _yahooProxyFailCount / _yahooProxyCooldownUntil cooldown state with fetchYahooChartDirect so both Yahoo paths pause together if Decodo's curl pool also gets blocked. No change to direct-path behavior when Yahoo is healthy.
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
…iew) Direct 200 with data.quoteSummary.result[0] absent is an app-level "no data for this symbol" signal (e.g. delisted ETF). Proxy won't return different data for a symbol Yahoo itself doesn't carry — falling back would burn the 5-failure cooldown budget on structurally empty symbols and mask a genuine proxy outage. Resolve null on !result; keep JSON.parse catch going to proxy (garbage body IS a transport-level signal — captive portal, Cloudflare challenge). Review feedback from PR #3134.
Contributor
Greptile SummaryThis PR fixes the 401 failures on Yahoo's
Confidence Score: 3/5Safe to merge only after replacing execFileSync with async execFile to avoid blocking the AIS relay event loop during sector seeding. The proxy routing, curl egress selection, and cooldown sharing are all correct. However, execFileSync in a live I/O callback is a P1 issue: it can stall the WebSocket relay for up to 180 s per cron tick. The async execFile is already available and the refactor is straightforward. scripts/ais-relay.cjs — _yahooQuoteSummaryProxyFallback needs to be converted to return a Promise using async execFile. Important Files Changed
Sequence DiagramsequenceDiagram
participant Cron as seedSectorSummary (cron)
participant QS as fetchYahooQuoteSummary
participant YD as Yahoo v10/finance/quoteSummary (direct)
participant FB as _yahooQuoteSummaryProxyFallback
participant Curl as execFileSync curl
participant Decodo as us.decodo.com
participant YP as Yahoo v10 (via proxy)
loop for each of 12 SECTOR_SYMBOLS
Cron->>QS: await fetchYahooQuoteSummary(symbol)
QS->>YD: https.get (direct, timeout 12s)
alt Direct succeeds (HTTP 200 + valid result)
YD-->>QS: 200 OK + JSON
QS-->>Cron: valuation object
else Direct fails (401 / bad body / error / timeout)
YD-->>QS: 401 (or error)
QS->>FB: _yahooQuoteSummaryProxyFallback(symbol, url)
Note over FB,Curl: execFileSync BLOCKS event loop up to 15s
FB->>Curl: execFileSync curl via us.decodo.com
Curl->>Decodo: HTTP request via curl egress
Decodo->>YP: forward to Yahoo v10
YP-->>Decodo: 200 + JSON
Decodo-->>Curl: proxy response
Curl-->>FB: stdout with JSON + status code
FB-->>QS: parsed valuation (or null on failure)
QS-->>Cron: valuation object (or null)
end
Cron->>Cron: await sleep(150ms)
end
Reviews (1): Last reviewed commit: "fix(sector-valuations): don't proxy on e..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Comment on lines
+1705
to
+1751
| function _yahooQuoteSummaryProxyFallback(symbol, url) { | ||
| const proxyAuth = resolveProxyString(); | ||
| if (!proxyAuth) return null; | ||
| if (Date.now() < _yahooProxyCooldownUntil) return null; | ||
| try { | ||
| const { execFileSync } = require('child_process'); | ||
| const args = [ | ||
| '-sS', '--compressed', '--max-time', '15', '-L', | ||
| '-x', `http://${proxyAuth}`, | ||
| '-H', `User-Agent: ${CHROME_UA}`, | ||
| '-H', 'Accept: application/json', | ||
| '-w', '\n%{http_code}', | ||
| url, | ||
| ]; | ||
| const out = execFileSync('curl', args, { encoding: 'utf8', timeout: 20000, stdio: ['pipe', 'pipe', 'pipe'] }); | ||
| const nl = out.lastIndexOf('\n'); | ||
| const status = parseInt(out.slice(nl + 1).trim(), 10); | ||
| if (status < 200 || status >= 300) { | ||
| _yahooProxyFailCount++; | ||
| if (_yahooProxyFailCount >= 5) { | ||
| _yahooProxyCooldownUntil = Date.now() + _YAHOO_PROXY_COOLDOWN_MS; | ||
| _yahooProxyFailCount = 0; | ||
| logThrottled('warn', 'sector-yahoo-proxy-cooldown', '[Sector] Yahoo proxy cooldown 5min after 5 failures'); | ||
| } | ||
| logThrottled('warn', `sector-yahoo-proxy-${status}:${symbol}`, `[Sector] Yahoo quoteSummary ${symbol} proxy HTTP ${status}`); | ||
| return null; | ||
| } | ||
| _yahooProxyFailCount = 0; | ||
| const data = JSON.parse(out.slice(0, nl)); | ||
| const result = data?.quoteSummary?.result?.[0]; | ||
| if (!result) return null; | ||
| const sd = result.summaryDetail || {}; | ||
| const ks = result.defaultKeyStatistics || {}; | ||
| const raw = (obj) => typeof obj === 'object' && obj !== null ? (obj.raw ?? obj.fmt ?? null) : (typeof obj === 'number' ? obj : null); | ||
| return { | ||
| trailingPE: raw(sd.trailingPE), | ||
| forwardPE: raw(sd.forwardPE), | ||
| beta: raw(sd.beta) ?? raw(ks.beta3Year), | ||
| ytdReturn: raw(ks.ytdReturn), | ||
| threeYearReturn: raw(ks.threeYearAverageReturn), | ||
| fiveYearReturn: raw(ks.fiveYearAverageReturn), | ||
| }; | ||
| } catch (err) { | ||
| logThrottled('warn', `sector-yahoo-proxy-err:${symbol}`, `[Sector] Yahoo quoteSummary ${symbol} proxy error: ${err.message}`); | ||
| return null; | ||
| } | ||
| } |
Contributor
There was a problem hiding this comment.
execFileSync blocks the event loop for up to 15 s per symbol
_yahooQuoteSummaryProxyFallback is invoked synchronously inside the https I/O callback via resolve(_yahooQuoteSummaryProxyFallback(...)). Because execFileSync is blocking, the entire Node.js event loop — including all WebSocket message handling for the AIS relay — freezes for up to --max-time 15 seconds per call. With 12 sector symbols all failing the direct path and processed sequentially, the worst-case stall is 12 × 15 = 180 s.
The async execFile is already imported at the top of the file (const { execFile } = require('child_process')). The fix is to:
- Return a
Promisefrom_yahooQuoteSummaryProxyFallback(useexecFilewith a callback and wrap innew Promise(...)) - Update each call site from
resolve(_yahooQuoteSummaryProxyFallback(symbol, url))to_yahooQuoteSummaryProxyFallback(symbol, url).then(resolve)
This keeps the existing timeout/cooldown/parse logic identical but releases the event loop while curl runs.
| if (!proxyAuth) return null; | ||
| if (Date.now() < _yahooProxyCooldownUntil) return null; | ||
| try { | ||
| const { execFileSync } = require('child_process'); |
Contributor
There was a problem hiding this comment.
require for an already-imported module
child_process is already imported at the top of the file (const { execFile } = require('child_process')). Adding execFileSync there (or to the existing destructure) avoids a redundant require call inside the function body and makes the dependency explicit at module load time.
Suggested change
| const { execFileSync } = require('child_process'); | |
| const args = [ |
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
…ort failures (review) Review feedback on PR #3134, both P1. P1 #1 — transport failures bypassed cooldown execFileSync timeouts, proxy-connect refusals, and JSON.parse on garbage bodies all went through the catch block and returned null without ticking _yahooProxyFailCount. In the exact failure mode this PR hardens against, the relay would have thrashed through 12 × 20s curl attempts per tick with no backoff. Extract a bumpCooldown() helper and call it from both the non-2xx branch and the catch block. P1 #2 — two Decodo egress pools shared one cooldown budget fetchYahooChartDirect uses CONNECT via gate.decodo.com. _yahooQuoteSummaryProxyFallback uses curl via us.decodo.com. These are independent egress IP pools — per the 2026-04-16 probe, Yahoo blocks CONNECT but accepts curl. Sharing cooldown means 5 CONNECT failures suppress the healthy curl path (and vice versa). Split into _yahooConnectProxy* (chart) and _yahooCurlProxy* (sector valuations). Also: on proxy 200 with empty result, reset the curl counter. The route is healthy even if this specific symbol has no data — don't pretend it's a failure.
…d 3) Review feedback on PR #3134, both P1. P1 #1 - double proxy invocation on timeout/error race req.destroy() inside the timeout handler can still emit 'error', and both handlers eagerly called resolve(_yahooQuoteSummaryProxyFallback(...)). A single upstream timeout therefore launched two curl subprocesses, double-ticked the cooldown counter, and blocked twice. Add a settled flag; settle() exits early on the second handler before evaluating the fallback. P1 #2 - execFileSync blocks the relay event loop The relay serves HTTP/WS traffic on the same thread that awaits seedSectorSummary's per-symbol Yahoo fetch. execFileSync for up to 20s per failure x 5 failures before cooldown = ~100s of frozen event loop. Switch to promisify(execFile). resolve(promise) chains the Promise through fetchYahooQuoteSummary's outer Promise, so the main-loop await yields while curl runs. Other traffic continues during the fetch. tests/sector-valuations.test.mjs: bump the static-analysis window from 1500 to 2000 chars so the field-extraction markers (ytdReturn etc.) stay inside the window after the settle guard was added.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why this PR?
Railway logs 2026-04-16 show every 5-min cron tick failing all 12 sector ETFs:
Yahoo's
/v10/finance/quoteSummaryrejects Railway container egress IPs with 401 while/v8/finance/chart(the other path in the same relay) still gets through direct.fetchYahooQuoteSummaryinscripts/ais-relay.cjshad no proxy fallback, so valuations have been silently zero on every sector seed.Fix
Add a curl-based proxy fallback to
fetchYahooQuoteSummarythat matches the sharedscripts/_yahoo-fetch.mjspattern used by the other Yahoo-touching seeders (market-quotes, commodity-quotes, etf-flows, gulf-quotes).Critically: route through Decodo's curl egress (
us.decodo.com), NOT the CONNECT egress (gate.decodo.com). Per the 2026-04-16 probe documented in_yahoo-fetch.mjs:Reusing the existing CJS
ytFetchViaProxywould have silently kept failing because it CONNECT-tunnels viagate.decodo.com.Implementation
_yahooQuoteSummaryProxyFallback(symbol, url)hoisted function declaration (kept outsidefetchYahooQuoteSummaryso static-analysis tests still seeUser-Agent,timeout:,trailingPE,v10/finance/quoteSummaryinside the 1500-char window)resolveProxyString()from_proxy-utils.cjs— same helper all.mjsseeders already use; auto-rewritesgate.decodo.com→us.decodo.comcurlviaexecFileSync(curl binary present onDockerfile.relay)_yahooProxyFailCount/_yahooProxyCooldownUntil/_YAHOO_PROXY_COOLDOWN_MScooldown state withfetchYahooChartDirectso the whole service pauses together if Decodo's curl pool ever gets blockedTest plan
node --check scripts/ais-relay.cjspassesnode --test tests/sector-valuations.test.mjs— 20/20 passnode --test tests/yahoo-fetch.test.mjs tests/relay-helper.test.mjs tests/shared-relay.test.mjs— 114/114 pass[Market] Seeded 12/12 sectors, N>0 valuationsin Railway logs after next cron ticksectorValuationssurface in the Market panel UI no longer all zeroRollback
Straight
git revert— only touches one file, no cache key bumps, no schema changes. If Decodo's curl pool also starts getting blocked by Yahoo, the_yahooProxyFailCount >= 5cooldown kicks in after 5 failed symbols and the service quietly degrades to "0 valuations" (current behavior) rather than thrashing.