Fix memory leak in fakeredis Lua script execution#300
Conversation
The fakeredis `memory://` backend was leaking memory when executing Lua scripts. Every `eval()` call created new `functools.partial` objects for the Lua callbacks (redis.call, redis.pcall, redis.log, cjson.encode, cjson.decode), and these were held by the Lua runtime, preventing GC. This restructures the monkeypatch to: - Create callback wrappers once at runtime init that look up the current socket dynamically - Cache static partials for log/cjson functions on the server - Only update KEYS/ARGV per call via a lightweight Lua function - Call `collectgarbage()` after each script to clean up Lua tables Testing methodology: Created a harness running a perpetual task at 10 evals/second. Before the fix, memory grew several MB per minute. After the fix, RSS stays flat at ~200KB above baseline even after 1000+ evals. The same fix has been submitted upstream: cunla/fakeredis-py#452 Also adds objgraph to dev dependencies for memory debugging. Co-Authored-By: Claude Opus 4.5 <[email protected]>
|
📚 Documentation has been built for this PR! You can download the documentation directly here: |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #300 +/- ##
=======================================
Coverage 99.64% 99.64%
=======================================
Files 98 98
Lines 9766 9767 +1
Branches 468 468
=======================================
+ Hits 9731 9732 +1
Misses 20 20
Partials 15 15
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 69e4791ea6
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| # Only update KEYS and ARGV per call (callbacks are already set) | ||
| self._server._lua_set_keys_argv( | ||
| lua_runtime.table_from(keys_and_args[:numkeys]), | ||
| lua_runtime.table_from(keys_and_args[numkeys:]), |
There was a problem hiding this comment.
Because set_globals_init (which assigns redis, cjson, and module globals) now runs only when _lua_runtime is first created, subsequent EVAL calls only update KEYS/ARGV. Any script that mutates those globals (e.g., redis.call = nil, cjson = {}, or overwriting module globals) will leave the Lua environment corrupted for later scripts, whereas the previous implementation reset these globals on every call. This can cause later scripts to fail or behave incorrectly; consider re-running the globals setup each eval while reusing cached callbacks.
Useful? React with 👍 / 👎.
This was added during memory leak debugging but isn't needed as a permanent dev dependency. Co-Authored-By: Claude Opus 4.5 <[email protected]>
The test was checking caplog.text for absence of unlogged values, but fakeredis DEBUG logs contain raw pickle data that includes those values. Filter to only docket-originated logs for the assertion. Co-Authored-By: Claude Opus 4.5 <[email protected]>
|
Thanks for the review! This is a fair observation about the changed semantics. The old code did reset However, I think this is fine because:
The memory leak was causing real problems (MB/minute growth under heavy Lua usage), and the fix is worth the theoretical edge case of scripts that corrupt their own globals. |
pydocket 0.17.2 includes a fix for a memory leak in fakeredis's Lua script execution that was causing unbounded memory growth when using the `memory://` URL for the Docket broker. Each `eval()` call was creating `functools.partial` objects that got held by the Lua runtime and never collected. Related to #18605 See also: - chrisguidry/docket#300 - cunla/fakeredis-py#452 Co-Authored-By: Claude Opus 4.5 <[email protected]>
2 participants
Summary
functools.partialobjects on everyeval()callcollectgarbage()after each script execution to clean up Lua tablesProblem
The
memory://backend was leaking memory when executing Lua scripts. Everyeval()created new partial objects for callbacks that were held by the Lua runtime, preventing garbage collection.Related: PrefectHQ/prefect#18605
Testing
Created a test harness running a perpetual task at ~10 evals/second:
The same fix has been submitted upstream: cunla/fakeredis-py#452
🤖 Generated with Claude Code