Skip to content

feat(playground): basic in-memory rate limiting to prevent runaway loops (#181) - #229

Merged
Karan-Raj-KR merged 2 commits into
Karan-Raj-KR:mainfrom
Subodh-17:feat/playground-rate-limiting-181
Oct 24, 2026
Merged

Karan-Raj-KR merged 2 commits into
Karan-Raj-KR:mainfrom
Subodh-17:feat/playground-rate-limiting-181

Conversation

@Subodh-17

@Subodh-17 Subodh-17 commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

What does this PR do?

Implements configurable, in-memory sliding-window rate limiting for the CrewAI Playground to prevent runaway execution loops (#181).

Changes

  • Add a thread-safe, standard-library sliding-window rate limiter in playground/rate_limiter.py.
  • Apply a shared per-client-IP quota to /run, /run/stream, and /run/ws in playground/main.py.
  • Support PLAYGROUND_RATE_LIMIT, defaulting to 30/5min; setting it to 0 disables rate limiting.
  • Return HTTP 429 responses with a Retry-After header and a descriptive JSON error for rejected HTTP requests.
  • Reject over-limit WebSocket connections before accepting them (RFC 6455 close code 1008).
  • Keep recipe listing (/recipes) and static asset routes unaffected.
  • Add deterministic tests for limits, expiry, concurrency, client isolation, endpoint behavior, and rejected execution.

Related issue

Closes #181

Type of change

  • New recipe
  • Recipe improvement / bug fix
  • Documentation
  • CI / tooling
  • Other (describe above)

Checklist

  • I ran my recipe end-to-end from a fresh virtual environment
  • ruff check and ruff format --check pass locally
  • I added/updated the recipe's README.md (inputs, outputs, sample run)
  • .env.example lists all required variables with placeholder values (no real keys)
  • This PR touches one recipe (or is a docs/tooling-only change)

Verification & Sample output

Test Results

  • Rate-limiter test suite: 14 passed in 3.33s (pytest -v test_rate_limiter.py)
  • Full playground test suite: 31 passed in 5.51s (pytest -v)
  • Recipe contract test suite: 78 passed in 0.29s (pytest tests/test_recipe_contract.py -v)
  • Repository-wide Ruff lint check: All checks passed across recipes/, playground/, tools/, tests/ (ruff==0.15.10)
  • Repository-wide Ruff format check: 123 files already formatted (ruff==0.15.10)

Note: Verified with repository-pinned ruff==0.15.10 matching CI (.github/workflows/ci.yml:43 and requirements-dev.txt:2). All repository files pass cleanly.

Sample Test Output

============================= test session starts =============================
platform win32 -- Python 3.11.0, pytest-9.0.2, pluggy-1.6.0
collected 14 items

test_rate_limiter.py::test_sliding_window_unit_limits_and_expiry PASSED  [  7%]
test_rate_limiter.py::test_sliding_window_disabled_mode PASSED           [ 14%]
test_rate_limiter.py::test_sliding_window_client_isolation PASSED        [ 21%]
test_rate_limiter.py::test_sliding_window_time_units_and_fallback PASSED [ 28%]
test_rate_limiter.py::test_sliding_window_concurrency_atomic PASSED      [ 35%]
test_rate_limiter.py::test_sliding_window_memory_pruning PASSED          [ 42%]
test_rate_limiter.py::test_get_client_ip_fallbacks PASSED                [ 50%]
test_rate_limiter.py::test_run_endpoint_31st_request_returns_429 PASSED  [ 57%]
test_rate_limiter.py::test_run_endpoint_rejection_prevents_llm_execution PASSED [ 64%]
test_rate_limiter.py::test_cross_endpoint_quota_sharing PASSED           [ 71%]
test_rate_limiter.py::test_stream_endpoint_rate_limit_exceeded PASSED    [ 78%]
test_rate_limiter.py::test_websocket_endpoint_rejection_code_1008 PASSED [ 85%]
test_rate_limiter.py::test_disabled_mode_permits_burst_on_endpoints PASSED [ 92%]
test_rate_limiter.py::test_exempt_routes_unaffected_when_quota_exhausted PASSED [100%]

============================= 14 passed in 3.33s ==============================

@github-actions

Copy link
Copy Markdown

🎉 Thanks for your first pull request to crewai-recipes! A maintainer will review it soon (usually within 7 days). Make sure CI is green and the PR checklist is filled in — that speeds up review a lot.

@qodo-code-review

qodo-code-review Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

Code Review by Qodo

Grey Divider

Sorry, something went wrong

We weren't able to complete the code review on our side. Please try again manually by commenting /agentic_review on this PR.

Grey Divider

Qodo Logo

@qodo-code-review

Copy link
Copy Markdown

PR Summary by Qodo

Add per-client rate limiting to Playground execution endpoints

✨ Enhancement 🧪 Tests 🕐 20-40 Minutes

Grey Divider

AI Description

• Limit execution requests per client IP to prevent runaway Playground loops.
• Configure the shared quota with PLAYGROUND_RATE_LIMIT; reject excess HTTP and WebSocket requests
 before execution.
• Test window expiry, concurrency, client isolation, endpoint behavior, and exempt routes.
Diagram

graph TD
  Client["Playground client"] --> Routes["Run endpoints"] --> IP["Client IP"] --> Limiter["Sliding window"] --> Quota{"Quota available?"} -->|Yes| Execution["Recipe execution"]
  Quota -->|No, HTTP| HTTP["HTTP 429"]
  Quota -->|No, WebSocket| WS["WS policy close"]
Loading
High-Level Assessment

The following are alternative approaches to this PR:

1. Centralized rate-limit store
  • ➕ Enforces one quota across multiple server processes or instances.
  • ➖ Adds infrastructure and operational complexity to a local Playground.

Recommendation: Keep the in-memory approach for the Playground's local, single-process use case: it provides protection without a new dependency. Document that quotas are per process; use a centralized store if multi-worker deployment becomes a requirement.

Files changed (3) +614 / -1

Enhancement (2) +224 / -1
main.pyGate all three execution endpoints with a shared quota +34/-1

Gate all three execution endpoints with a shared quota

• Creates one limiter and checks the connection's client IP before execution or cache access. Excess HTTP and SSE requests receive 429 with Retry-After; excess WebSocket connections close with code 1008 before acceptance. Recipe listing and static routes remain outside the quota.

playground/main.py

rate_limiter.pyAdd configurable, thread-safe sliding-window limiter +190/-0

Add configurable, thread-safe sliding-window limiter

• Tracks per-client request timestamps using a monotonic clock and lock, calculates retry delays, and prunes retained client keys. Parses PLAYGROUND_RATE_LIMIT with a 30/5min default and a disabled mode, and extracts client hosts without trusting forwarded headers.

playground/rate_limiter.py

Tests (1) +390 / -0
test_rate_limiter.pyCover limiter behavior and execution-endpoint rejection +390/-0

Cover limiter behavior and execution-endpoint rejection

• Adds unit tests for quotas, expiry, configuration, client isolation, concurrency, pruning, and IP fallback. Integration tests cover shared quotas, rejection before execution, disabled mode, HTTP and WebSocket responses, and unaffected listing and static routes.

playground/test_rate_limiter.py

@Subodh-17

Copy link
Copy Markdown
Contributor Author

Hi! I've pushed the follow-up fix for PR #229 (\3aa4e9e) addressing the Ruff E402 errors. The repository-wide lint and formatting checks pass locally with the pinned Ruff 0.15.10, and the relevant test suites pass. GitHub Actions run 40067553312 is currently marked \�ction_required. Could you please approve and run the workflow when convenient? Thanks!

@Karan-Raj-KR Karan-Raj-KR added recipe: improvement Enhancement to an existing recipe backend ECSoC26-L3 ECSoC26 — hard task (Level 3) labels Oct 24, 2026 — with Meta Muse AI
@Karan-Raj-KR
Karan-Raj-KR merged commit fece08a into Karan-Raj-KR:main Oct 24, 2026
45 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend ECSoC26-L3 ECSoC26 — hard task (Level 3) recipe: improvement Enhancement to an existing recipe

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Playground: basic in-memory rate limiting to prevent runaway loops

2 participants