Skip to content

fix: resolve QueuePool exhaustion by consolidating duplicate SQLAlchemy engines#1452

Merged
aliasaria merged 4 commits intomainfrom
fix/sqlalchemy-connection-pool-exhaustion
Mar 4, 2026
Merged

fix: resolve QueuePool exhaustion by consolidating duplicate SQLAlchemy engines#1452
aliasaria merged 4 commits intomainfrom
fix/sqlalchemy-connection-pool-exhaustion

Conversation

@aliasaria
Copy link
Copy Markdown
Member

Problem

QueuePool limit of size 5 overflow 10 reached, connection timed out, timeout 30.00

The app had two separate create_async_engine() calls against the same database — one in db/session.py and another in shared/models/user_model.py — each with SQLAlchemy's default pool settings (pool_size=5, max_overflow=10). Under concurrent load, both pools would exhaust their connections.

Fix

  1. Consolidated to a single engineuser_model.py now imports and reuses the engine and session factory from db/session.py instead of creating its own.
  2. Increased pool limitspool_size=20, max_overflow=40, pool_timeout=60 on the shared engine.

Testing

  • All 117 existing tests pass (1 pre-existing failure in test_create_team unrelated to this change).
  • Docker test environment verified healthy.

…my engines

- Eliminated duplicate create_async_engine() in user_model.py; now reuses
  the single engine from db/session.py
- Increased pool_size from 5 to 20, max_overflow from 10 to 40, and
  pool_timeout from 30s to 60s on the shared engine
- Fixes 'QueuePool limit of size 5 overflow 10 reached' timeout errors
SQLite serializes writes at the file level so connection pooling provides
no benefit and can cause QueuePool exhaustion. Use NullPool for SQLite
and retain higher pool limits (pool_size=20, max_overflow=40) for Postgres.
@paragon-review
Copy link
Copy Markdown

paragon-review bot commented Mar 4, 2026

Paragon Summary

This pull request review identified 1 issue across 1 category in 2 files. The review analyzed code changes, potential bugs, security vulnerabilities, performance issues, and code quality concerns using automated analysis tools.

This PR fixes a connection pool exhaustion issue by consolidating two separate SQLAlchemy engine instances into a single shared engine and increasing pool limits, eliminating the QueuePool limit of size 5 overflow 10 reached error that occurred under concurrent load.

Key changes:

  • Consolidated two duplicate SQLAlchemy async engines into a single shared engine
  • user_model.py now imports/reuses engine from db/session.py instead of creating its own
  • Increased pool limits: pool_size=20, max_overflow=40, pool_timeout=60
  • Fixes QueuePool limit reached connection exhaustion under concurrent load

Confidence score: 5/5

  • This PR has low risk with no critical or high-priority issues identified
  • Score reflects clean code review with only minor suggestions or no issues found
  • Code quality checks passed - safe to proceed with merge

2 files reviewed, 1 comment

Severity breakdown: Low: 1


Tip: @paragon-run <instructions> to chat with our agent or push fixes!

Dashboard

import uuid

from transformerlab.db.constants import DATABASE_URL
from transformerlab.db.session import async_engine, async_session as AsyncSessionLocal
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: async_engine is imported but never used in user_model

async_engine is imported but never used in user_model.py. This adds unnecessary coupling and dead code. Remove the unused import.

View Details

Location: api/transformerlab/shared/models/user_model.py (lines 11)

Analysis

async_engine is imported but never used in user_model.py

What fails async_engine import is unused, adding unnecessary dependency coupling
Result Code compiles with dead import that increases coupling between modules
Expected Import should only include what is used (async_session as AsyncSessionLocal)
Impact Low - adds unnecessary import coupling but no runtime impact
How to reproduce
Check user_model.py - async_engine is imported but not referenced anywhere in the visible code
Patch Details
-from transformerlab.db.session import async_engine, async_session as AsyncSessionLocal
+from transformerlab.db.session import async_session as AsyncSessionLocal
AI Fix Prompt
Fix this issue: async_engine is imported but never used in user_model.py. This adds unnecessary coupling and dead code. Remove the unused import.

Location: api/transformerlab/shared/models/user_model.py (lines 11)
Problem: async_engine import is unused, adding unnecessary dependency coupling
Current behavior: Code compiles with dead import that increases coupling between modules
Expected: Import should only include what is used (async_session as AsyncSessionLocal)
Steps to reproduce: Check user_model.py - async_engine is imported but not referenced anywhere in the visible code

Provide a code fix.


Tip: Reply with @paragon-run to automatically fix this issue

@sentry
Copy link
Copy Markdown

sentry bot commented Mar 4, 2026

Codecov Report

❌ Patch coverage is 66.66667% with 2 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
api/transformerlab/db/session.py 50.00% 1 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

@aliasaria aliasaria merged commit 792fc92 into main Mar 4, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants