Skip to content

Commit 5574924

Browse files
mattvinallclaude
andcommitted
fix(infra): pool-leak + Scrape.do 429 storm
Two production issues, one commit because the test surface overlaps. Connection-pool leak -------------------- Logs were showing "non-checked-in connection" GC warnings twice per long job. Two compounding causes: * pool_recycle=300 was killing connections every 5 min — right when a ~5-min Intel pipeline was mid-flight, racing with asyncpg cleanup. Bumped to 1800 (30 min); added explicit pool_timeout=30 so pool exhaustion fails fast instead of hanging. * asyncio.create_task in every router was returning the Task to a local variable that went out of scope as soon as the handler returned. The event loop only weak-refs tasks, so the GC could collect a still-running pipeline mid-await — and that cancellation racing with AsyncSession.__aexit__ matches the SQLAlchemy issue exactly (sqlalchemy/sqlalchemy#8145, #6652). Added pin_background_task() and register_pipeline_task() helpers in _job_errors.py that hold strong refs in a module-level set and self-clean on done. Migrated all six routers (intel, people, g2, upload, funding, maps) to use them. funding/maps keep their bespoke failure handlers but now also pin the task. Scrape.do 429 storm ------------------- At job start we were firing ~100+ concurrent requests at api.scrape.do and getting 90%+ 429s, which silently dropped scraped pages. Verified against the live /info endpoint that the account is on Hobby tier with ConcurrentRequest=10 — so scrape_concurrency=30 was 3x over the cap before counting inner-page fan-out. Three fixes: * Added a module-level _SCRAPE_DO_SEM = Semaphore(scrape_do_concurrency) in scraper.py that wraps every Scrape.do API call. Hard cap independent of any caller's batch size. Default 8 leaves 2 slots of headroom on a Hobby plan; bump scrape_do_concurrency for higher tiers. * Lowered scrape_concurrency 30 -> 10 (outer batch size; real cap is the semaphore above). * Fixed a separate latent bug in scrape_page: response.raise_for_status() was running *outside* the retry_async wrapper, so 429s bubbled straight up and the configured retries did nothing. Moved raise_for_status() inside the retry lambda. Also bumped Scrape.do retries 2 -> 3 and max_delay 15 -> 30 since burst recovery sometimes needs >2 attempts. retry.py now honors the Retry-After header (numeric seconds, clamped to max_delay) before falling back to exponential backoff with jitter. Added two regression tests pinning that behavior; fake response fixture got a headers dict. Verification: 182/182 backend tests pass; smoke import confirms scrape_concurrency=10, scrape_do_concurrency=8, pool_recycle=1800, semaphore initial value=8. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
1 parent 42a8340 commit 5574924

12 files changed

Lines changed: 150 additions & 25 deletions

File tree

‎backend/app/config.py‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,13 @@ class Settings(BaseSettings):
3535
normalize_batch_size: int = 200
3636
enrich_batch_size: int = 50
3737
scrape_do_api_key: str = ""
38-
scrape_concurrency: int = 30
38+
# Outer batch-level fan-out across domains. Real per-process cap is enforced
39+
# by scrape_do_concurrency below; this just bounds asyncio task spawn rate.
40+
scrape_concurrency: int = 10
41+
# Hard global cap on simultaneous Scrape.do API requests. Must stay under
42+
# your plan's "ConcurrentRequest" limit (Hobby=10, Pro=50, Business=100).
43+
# Default 8 leaves 2 slots of headroom for retries on a Hobby plan.
44+
scrape_do_concurrency: int = 8
3945
max_pages_per_site: int = 6
4046
scrape_timeout: int = 20
4147
intel_extraction_concurrency: int = 4 # Gemini Flash free tier ~10 RPM; 10 concurrent calls burst over the limit and silently 429

‎backend/app/database.py‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,11 @@
1818
"prepared_statement_cache_size": 0,
1919
},
2020
pool_pre_ping=True,
21-
pool_recycle=300,
21+
# 30 min — long enough that a single ~5-min job won't trigger mid-flight
22+
# connection recycling, which was racing with asyncpg cleanup and surfacing
23+
# as "non-checked-in connection" GC warnings.
24+
pool_recycle=1800,
25+
pool_timeout=30,
2226
)
2327

2428
AsyncSessionLocal = async_sessionmaker(

‎backend/app/routers/_job_errors.py‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,34 @@
2424
# can't reap them mid-flight. Tasks self-remove on completion.
2525
_pending_failure_writes: set[asyncio.Task] = set()
2626

27+
# Same pattern for the long-running pipeline tasks themselves. asyncio's event
28+
# loop only keeps weak references to tasks; if no caller holds a strong ref,
29+
# the GC can collect a still-running task and abruptly cancel it mid-await.
30+
# That cancellation racing with SQLAlchemy's AsyncSession.__aexit__ is the
31+
# source of the "non-checked-in connection" warnings we were seeing in prod.
32+
_pending_pipelines: set[asyncio.Task] = set()
33+
34+
35+
def pin_background_task(task: asyncio.Task) -> None:
36+
"""Hold a strong reference to *task* so the GC can't reap it mid-flight.
37+
38+
Use this for any fire-and-forget asyncio task that lives longer than the
39+
function that spawned it. Tasks self-remove on completion.
40+
"""
41+
_pending_pipelines.add(task)
42+
task.add_done_callback(_pending_pipelines.discard)
43+
44+
45+
def register_pipeline_task(
46+
task: asyncio.Task,
47+
job_id: uuid.UUID,
48+
logger_: logging.Logger,
49+
label: str = "Pipeline",
50+
) -> None:
51+
"""Pin the task so GC can't reap it, and wire up failure persistence."""
52+
pin_background_task(task)
53+
task.add_done_callback(make_failure_callback(job_id, logger_, label))
54+
2755

2856
async def mark_job_failed(job_id: uuid.UUID, exc: BaseException) -> None:
2957
"""Set Job.status='failed' and record the error so the UI can surface it.

‎backend/app/routers/funding.py‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@
1919
from app.config import settings
2020
from app.database import AsyncSessionLocal, get_db
2121
from app.models import Job, JobResult
22+
from app.routers._job_errors import pin_background_task
2223
from app.services.funding_discovery import discover_funded_companies
2324

2425
router = APIRouter(prefix="/funding", tags=["funding"])
@@ -212,6 +213,7 @@ def _on_done(t: asyncio.Task) -> None:
212213
asyncio.create_task(_mark_job_failed_on_error(job.id, exc))
213214

214215
task = asyncio.create_task(run_funding_pipeline({}, str(job.id)))
216+
pin_background_task(task)
215217
task.add_done_callback(_on_done)
216218

217219
new_token = create_token(email, str(job.id))

‎backend/app/routers/g2.py‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@
1818
from app.config import settings
1919
from app.database import get_db
2020
from app.models import Job, JobResult
21-
from app.routers._job_errors import make_failure_callback
21+
from app.routers._job_errors import register_pipeline_task
2222
from app.services.g2_categories import (
2323
get_all_categories,
2424
get_all_parents,
@@ -141,7 +141,7 @@ async def submit_g2_extraction(
141141
from app.workers.g2_pipeline import run_g2_pipeline
142142

143143
task = asyncio.create_task(run_g2_pipeline({}, str(job.id)))
144-
task.add_done_callback(make_failure_callback(job.id, logger, "G2 pipeline"))
144+
register_pipeline_task(task, job.id, logger, "G2 pipeline")
145145

146146
new_token = create_token(email, str(job.id))
147147
return {

‎backend/app/routers/intel.py‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@
1919
from app.config import settings
2020
from app.database import get_db
2121
from app.models import Job, JobResult
22-
from app.routers._job_errors import make_failure_callback
22+
from app.routers._job_errors import register_pipeline_task
2323

2424
router = APIRouter(prefix="/intel", tags=["intel"])
2525
logger = logging.getLogger(__name__)
@@ -131,7 +131,7 @@ async def submit_extraction(
131131
from app.workers.intel_pipeline import run_intel_pipeline
132132

133133
task = asyncio.create_task(run_intel_pipeline({}, str(job.id)))
134-
task.add_done_callback(make_failure_callback(job.id, logger, "Intel pipeline"))
134+
register_pipeline_task(task, job.id, logger, "Intel pipeline")
135135

136136
new_token = create_token(email, str(job.id))
137137
return {

‎backend/app/routers/maps.py‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@
1818
from app.config import settings
1919
from app.database import AsyncSessionLocal, get_db
2020
from app.models import Job, JobResult
21+
from app.routers._job_errors import pin_background_task
2122

2223
router = APIRouter(prefix="/maps", tags=["maps"])
2324
logger = logging.getLogger(__name__)
@@ -181,6 +182,7 @@ def _on_done(t: asyncio.Task) -> None:
181182
asyncio.create_task(_mark_job_failed_on_error(job.id, exc))
182183

183184
task = asyncio.create_task(run_maps_pipeline({}, str(job.id)))
185+
pin_background_task(task)
184186
task.add_done_callback(_on_done)
185187

186188
new_token = create_token(email, str(job.id))

‎backend/app/routers/people.py‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@
1919
from app.config import settings
2020
from app.database import get_db
2121
from app.models import Job, JobResult
22-
from app.routers._job_errors import make_failure_callback
22+
from app.routers._job_errors import register_pipeline_task
2323

2424
logger = logging.getLogger(__name__)
2525

@@ -125,7 +125,7 @@ async def submit_people_extraction(
125125
from app.workers.people_pipeline import run_people_pipeline
126126

127127
task = asyncio.create_task(run_people_pipeline({}, str(job.id)))
128-
task.add_done_callback(make_failure_callback(job.id, logger, "People pipeline"))
128+
register_pipeline_task(task, job.id, logger, "People pipeline")
129129

130130
new_token = create_token(email, str(job.id))
131131
return {

‎backend/app/routers/upload.py‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@
1212
from app.config import settings
1313
from app.database import get_db
1414
from app.models import Job, JobResult
15-
from app.routers._job_errors import make_failure_callback
15+
from app.routers._job_errors import register_pipeline_task
1616
from app.services.rate_limiter import check_rate_limit
1717

1818
router = APIRouter(tags=["upload"])
@@ -153,7 +153,7 @@ async def upload_csv(
153153
logger = logging.getLogger(__name__)
154154

155155
task = asyncio.create_task(run_pipeline({}, str(job.id)))
156-
task.add_done_callback(make_failure_callback(job.id, logger, "Company Intel pipeline"))
156+
register_pipeline_task(task, job.id, logger, "Company Intel pipeline")
157157

158158
# --- return response ---
159159
new_token = create_token(email, str(job.id))

‎backend/app/services/retry.py‎

Lines changed: 19 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,22 @@
1515
RETRYABLE_STATUS_CODES = frozenset({429, 500, 502, 503, 504})
1616

1717

18+
def _retry_delay_from_response(
19+
response: httpx.Response,
20+
attempt: int,
21+
base_delay: float,
22+
max_delay: float,
23+
) -> float:
24+
"""Honor Retry-After if present, else exponential backoff with jitter."""
25+
retry_after = response.headers.get("retry-after")
26+
if retry_after:
27+
try:
28+
return min(max(float(retry_after), 0.0), max_delay)
29+
except ValueError:
30+
pass # Non-numeric (HTTP-date) — fall through to backoff.
31+
return min(base_delay * (2 ** attempt) + random.uniform(0, 0.5), max_delay)
32+
33+
1834
async def retry_async(
1935
fn: Callable[[], Awaitable[T]],
2036
*,
@@ -36,7 +52,9 @@ async def retry_async(
3652
return await fn()
3753
except httpx.HTTPStatusError as exc:
3854
if exc.response.status_code in RETRYABLE_STATUS_CODES and attempt < max_retries:
39-
delay = min(base_delay * (2 ** attempt) + random.uniform(0, 0.5), max_delay)
55+
delay = _retry_delay_from_response(
56+
exc.response, attempt, base_delay, max_delay,
57+
)
4058
logger.warning(
4159
"Retry %d/%d after HTTP %d, delay=%.1fs",
4260
attempt + 1,

0 commit comments

Comments
 (0)