Repository navigation
Commit 5574924
fix(infra): pool-leak + Scrape.do 429 storm
Two production issues, one commit because the test surface overlaps.
Connection-pool leak
--------------------
Logs were showing "non-checked-in connection" GC warnings twice per long
job. Two compounding causes:
* pool_recycle=300 was killing connections every 5 min — right when a
~5-min Intel pipeline was mid-flight, racing with asyncpg cleanup.
Bumped to 1800 (30 min); added explicit pool_timeout=30 so pool
exhaustion fails fast instead of hanging.
* asyncio.create_task in every router was returning the Task to a local
variable that went out of scope as soon as the handler returned. The
event loop only weak-refs tasks, so the GC could collect a still-running
pipeline mid-await — and that cancellation racing with
AsyncSession.__aexit__ matches the SQLAlchemy issue exactly
(sqlalchemy/sqlalchemy#8145, #6652).
Added pin_background_task() and register_pipeline_task() helpers in
_job_errors.py that hold strong refs in a module-level set and self-clean
on done. Migrated all six routers (intel, people, g2, upload, funding,
maps) to use them. funding/maps keep their bespoke failure handlers but
now also pin the task.
Scrape.do 429 storm
-------------------
At job start we were firing ~100+ concurrent requests at api.scrape.do
and getting 90%+ 429s, which silently dropped scraped pages. Verified
against the live /info endpoint that the account is on Hobby tier with
ConcurrentRequest=10 — so scrape_concurrency=30 was 3x over the cap
before counting inner-page fan-out.
Three fixes:
* Added a module-level _SCRAPE_DO_SEM = Semaphore(scrape_do_concurrency)
in scraper.py that wraps every Scrape.do API call. Hard cap independent
of any caller's batch size. Default 8 leaves 2 slots of headroom on a
Hobby plan; bump scrape_do_concurrency for higher tiers.
* Lowered scrape_concurrency 30 -> 10 (outer batch size; real cap is the
semaphore above).
* Fixed a separate latent bug in scrape_page: response.raise_for_status()
was running *outside* the retry_async wrapper, so 429s bubbled straight
up and the configured retries did nothing. Moved raise_for_status()
inside the retry lambda. Also bumped Scrape.do retries 2 -> 3 and
max_delay 15 -> 30 since burst recovery sometimes needs >2 attempts.
retry.py now honors the Retry-After header (numeric seconds, clamped to
max_delay) before falling back to exponential backoff with jitter. Added
two regression tests pinning that behavior; fake response fixture got a
headers dict.
Verification: 182/182 backend tests pass; smoke import confirms
scrape_concurrency=10, scrape_do_concurrency=8, pool_recycle=1800,
semaphore initial value=8.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>1 parent 42a8340 commit 5574924
12 files changed
Lines changed: 150 additions & 25 deletions
File tree
- backend
- app
- routers
- services
- tests
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
35 | 35 | | |
36 | 36 | | |
37 | 37 | | |
38 | | - | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
39 | 45 | | |
40 | 46 | | |
41 | 47 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
18 | 18 | | |
19 | 19 | | |
20 | 20 | | |
21 | | - | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
22 | 26 | | |
23 | 27 | | |
24 | 28 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
24 | 24 | | |
25 | 25 | | |
26 | 26 | | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
27 | 55 | | |
28 | 56 | | |
29 | 57 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
19 | 19 | | |
20 | 20 | | |
21 | 21 | | |
| 22 | + | |
22 | 23 | | |
23 | 24 | | |
24 | 25 | | |
| |||
212 | 213 | | |
213 | 214 | | |
214 | 215 | | |
| 216 | + | |
215 | 217 | | |
216 | 218 | | |
217 | 219 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
18 | 18 | | |
19 | 19 | | |
20 | 20 | | |
21 | | - | |
| 21 | + | |
22 | 22 | | |
23 | 23 | | |
24 | 24 | | |
| |||
141 | 141 | | |
142 | 142 | | |
143 | 143 | | |
144 | | - | |
| 144 | + | |
145 | 145 | | |
146 | 146 | | |
147 | 147 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
19 | 19 | | |
20 | 20 | | |
21 | 21 | | |
22 | | - | |
| 22 | + | |
23 | 23 | | |
24 | 24 | | |
25 | 25 | | |
| |||
131 | 131 | | |
132 | 132 | | |
133 | 133 | | |
134 | | - | |
| 134 | + | |
135 | 135 | | |
136 | 136 | | |
137 | 137 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
18 | 18 | | |
19 | 19 | | |
20 | 20 | | |
| 21 | + | |
21 | 22 | | |
22 | 23 | | |
23 | 24 | | |
| |||
181 | 182 | | |
182 | 183 | | |
183 | 184 | | |
| 185 | + | |
184 | 186 | | |
185 | 187 | | |
186 | 188 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
19 | 19 | | |
20 | 20 | | |
21 | 21 | | |
22 | | - | |
| 22 | + | |
23 | 23 | | |
24 | 24 | | |
25 | 25 | | |
| |||
125 | 125 | | |
126 | 126 | | |
127 | 127 | | |
128 | | - | |
| 128 | + | |
129 | 129 | | |
130 | 130 | | |
131 | 131 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
12 | 12 | | |
13 | 13 | | |
14 | 14 | | |
15 | | - | |
| 15 | + | |
16 | 16 | | |
17 | 17 | | |
18 | 18 | | |
| |||
153 | 153 | | |
154 | 154 | | |
155 | 155 | | |
156 | | - | |
| 156 | + | |
157 | 157 | | |
158 | 158 | | |
159 | 159 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
15 | 15 | | |
16 | 16 | | |
17 | 17 | | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
18 | 34 | | |
19 | 35 | | |
20 | 36 | | |
| |||
36 | 52 | | |
37 | 53 | | |
38 | 54 | | |
39 | | - | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
40 | 58 | | |
41 | 59 | | |
42 | 60 | | |
| |||
0 commit comments