Skip to content

Limiting the number of concurrent conversion tasks to reduce memory usage #75

@xlfjn

Description

@xlfjn

Is your feature request related to a problem? Please describe.
The memory consumption of webp-server may grow to >500MB when processing 5 concurrent requests that require webp conversion. Such workload causes a great burden on conventional virtual servers (with 1GB ram) and may trigger the OOM killer.

Describe the solution you'd like
Implement a limit on the number of concurrent webp conversion tasks.

Describe alternatives you've considered
The memory usage problem can be partially mitigated with the following options:

  1. Limit the concurrent request numbers in HTTP servers.
    This is a common strategy to prevent the backend from overloading. However, it's ill-suited for webp-server, as conversion requests will block subsequent requests that can be directly fulfilled by returning the original image file.

  2. Convert every image file to webp in advance with -prefetch
    However, users still need to restart webp-server to perform pre-conversion every time they add some image files, or some concurrent requests might still cause excessive memory usage.

Both methods are not optimal enough for real-world usage, hence I think this problem should be addressed in webp-server itself.

Metadata

Metadata

Assignees

Labels

bugSomething isn't workingdiscussionThis topic needs some discussion.enhancementNew feature or requesthelp wantedExtra attention is needed

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions