fix: prevent OOM crash when loading large EPUBs (2000+ chapters)#457
fix: prevent OOM crash when loading large EPUBs (2000+ chapters)#457DChells wants to merge 1 commit intocrosspoint-reader:masterfrom
Conversation
Remove the call to loadAllFileStatSlims() which pre-loads all ZIP central directory entries into memory. For EPUBs with 2000+ chapters (like webnovels), this exhausts the ESP32-C3's ~380KB RAM and causes abort(). The existing loadFileStatSlim() function already handles individual lookups by scanning the central directory per-file when the cache is empty. This is O(n*m) instead of O(n), but prevents memory exhaustion. Fixes crosspoint-reader#134
Test Results ✅Tested on Xteink X4 e-reader with CrossPoint community firmware. Test Case: Shadow Slave (2768 chapters, 2773 ZIP entries)Before this PR: Device crashed immediately during initial indexing due to OOM when After this PR: Book indexed successfully!
AnalysisThe fix correctly addresses the root cause: By removing this call and falling back to sequential ZIP reads, we trade indexing speed for memory safety. This is the correct tradeoff for a memory-constrained device. |
daveallie
left a comment
daveallie
left a comment
There was a problem hiding this comment.
Can you please compare the initial load times for some other epubs. I specifically wrote loadAllFileStatSlims and added it here to avoid the O(n^2) issue of scanning through the zip central directory for each file.
imo a 25 minute load time is pretty unacceptable and we should just crash out. Especially if it significantly slows down opening previously ok files.
|
Closing as superseded by #458, which includes both the OOM fix (skipping |
2 participants
Summary
Fixes #134 - Large EPUBs with 2000+ chapters (common in webnovels) crash the device during initial indexing.
Problem
When loading a large EPUB,
buildBookBin()callsloadAllFileStatSlims()which pre-loads ALL ZIP central directory entries into astd::unordered_map<std::string, FileStatSlim>. For EPUBs like "Shadow Slave" (2773 entries) or "Lord of Mysteries" (1400+ entries), this exhausts ESP32-C3's ~380KB RAM and triggersabort().Serial log evidence:
Solution
Remove the
loadAllFileStatSlims()call. The existingloadFileStatSlim()function already handles individual lookups by scanning the ZIP central directory per-file when the cache is empty.Trade-off: This is O(n×m) instead of O(n) for lookups, making initial indexing slower for large EPUBs. However, it prevents memory exhaustion and allows the book to load successfully. A future optimization could cache only spine item stats rather than all ZIP entries.
Testing
Notes
This addresses the TODO comment that was already in the code acknowledging this issue.