API optimisation: more streaming of outputs by Alkarex · Pull Request #8041 · FreshRSS/FreshRSS

Alkarex · 2025-10-01T11:40:02Z

I spotted a memory issue when testing #7714
Attempt to stream results more, instead of keeping too much in memory.
Could be further improved.

I spotted a memory issue when testing FreshRSS#7714 Attempt to stream results more, instead of keeping too much in memory. Could be further improved.

Alkarex · 2025-10-01T11:41:08Z

cli/check.translation.php

 	$regex = new RegexIterator($iterator, '/^.+\.(php|phtml)$/i', RecursiveRegexIterator::GET_MATCH);
 	$usedI18n = [];
-	foreach (array_keys(iterator_to_array($regex)) as $file) {
+	foreach ($regex as $file => $value) {


@aledeg Please double-check.

In this PR, I have tried to remove some of the iterator_to_array() constructs

aledeg

Good job!

app/Models/Feed.php

p/api/greader.php

Co-authored-by: Alexis Degrugillier <[email protected]>

Alkarex · 2025-10-01T18:20:05Z

Tested to be able to return 14MB payloads with our default Docker image (which memory settings are 128MB at the moment).

"GET /api/greader.php/reader/api/0/stream/contents/user/-/label/Actus%20DK?output=json&n=9999 HTTP/1.1" 200 10345967 "-" "YourNews/1.10.1 (https://yournews.app/contact/)"

"GET /api/greader.php/reader/api/0/stream/contents/user/-/label/Actus%20techno%20Int.?output=json&n=9999 HTTP/1.1" 200 14987110 "-" "YourNews/1.10.1 (https://yournews.app/contact/)"

Before this PR, it used to return an error:

PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 20480 bytes) in /var/www/FreshRSS/app/Models/Entry.php on line 1149
2025-10-01T09:19:23.520779140Z 2a05:f6c7:346:0:69ab:fe5e:e581:8a29 - - [01/Oct/2025:11:19:17 +0200] "GET /api/greader.php/reader/api/0/stream/contents/user/-/label/Actus%20DK?output=json&n=9999 HTTP/1.1" 500 - "-" "YourNews/1.10.1 (https://yournews.app/contact/)"

Ping @TijnvandenEijnde , you should check on your side. This can be very costly for the client in terms of bandwidth and battery consumption to rely on such voluminous requests. The content of the articles is only supposed to be downloaded once (or maybe when it changes). Make sure to read some synchronisation strategy tips:

https://freshrss.github.io/FreshRSS/en/developers/06_GoogleReader_API.html#synchronisation-strategy

Due to high bandwidth in current sync FreshRSS#8041 (comment) Follow-up of FreshRSS#7714

Alkarex · 2025-10-01T18:32:57Z

API client "Your News": one star sync #8051

Due to high bandwidth in current sync #8041 (comment) Follow-up of #7714

TijnvandenEijnde · 2025-10-02T06:57:25Z

@Alkarex Thanks for the heads up! I was already planning on improving it and I will definitely have a look at the synchronisation strategy tips.

I will create a new PR once I made the improvements.

Thanks again!

This PR contains the following updates: | Package | Update | Change | |---|---|---| | [freshrss/freshrss](https://freshrss.org/) ([source](https://github.com/FreshRSS/FreshRSS)) | minor | `1.27.1` -> `1.28.0` | --- ### Release Notes <details> <summary>FreshRSS/FreshRSS (freshrss/freshrss)</summary> ### [`v1.28.0`](https://github.com/FreshRSS/FreshRSS/blob/HEAD/CHANGELOG.md#2025-12-24-FreshRSS-1280) [Compare Source](FreshRSS/FreshRSS@1.27.1...1.28.0) - Features - New sorting and filtering by date of *User modified* [#7886](FreshRSS/FreshRSS#7886), [#8090](FreshRSS/FreshRSS#8090), [#8105](FreshRSS/FreshRSS#8105), [#8118](FreshRSS/FreshRSS#8118), [#8130](FreshRSS/FreshRSS#8130) - Corresponding search operator, e.g. `userdate:PT1H` for the past hour [#8093](FreshRSS/FreshRSS#8093) - Allows finding articles marked by the local user as read/unread or starred/unstarred at specific dates for e.g. undo action. - New sorting by article length [#8119](FreshRSS/FreshRSS#8119) - New advanced search form [#8103](FreshRSS/FreshRSS#8103), [#8122](FreshRSS/FreshRSS#8122), [#8226](FreshRSS/FreshRSS#8226) - Add compatibility with PCRE word boundary `\b` and `\B` for regex search using PostgreSQL [#8141](FreshRSS/FreshRSS#8141) - More uniform SQL search and PHP search for accents and case-sensitivity (e.g. for automatically marking as read) [#8329](FreshRSS/FreshRSS#8329) - New overview of dates with most unread articles [#8089](FreshRSS/FreshRSS#8089) - Allow marking as read articles older than 1 or 7 days also when sorting by publication date [#8163](FreshRSS/FreshRSS#8163) - New option to show user labels instead of tags in RSS share [#8112](FreshRSS/FreshRSS#8112) - Add new feed visibility (priority) *Show in its feed* [#7972](FreshRSS/FreshRSS#7972) - New ability to share feed visibility through API (implemented by e.g. Capy Reader) [#7583](FreshRSS/FreshRSS#7583), [#8158](FreshRSS/FreshRSS#8158) - Configurable notification timeout [#7942](FreshRSS/FreshRSS#7942) - OPML export/import of unicity criteria [#8243](FreshRSS/FreshRSS#8243) - Ensure stable IDs (categories, feeds, labels) during export/import [#7988](FreshRSS/FreshRSS#7988) - Add username and timestamp to SQLite export from Web UI [#8169](FreshRSS/FreshRSS#8169) - Add option to apply filter actions to existing articles [#7959](FreshRSS/FreshRSS#7959), [#8259](FreshRSS/FreshRSS#8259) - Support CSS selector `~` *subsequent-sibling* [#8154](FreshRSS/FreshRSS#8154) - Upstream PR [phpgt/CssXPath#231](phpgt/CssXPath#231) - Rework saving of configuration files for more reliability in case of e.g. full disk [#8220](FreshRSS/FreshRSS#8220) - Web scraping support date format as milliseconds for Unix epoch [#8266](FreshRSS/FreshRSS#8266) - Allow negative category sort numbers [#8330](FreshRSS/FreshRSS#8330) - Performance - Improve SQL speed for updating cached information [#6957](FreshRSS/FreshRSS#6957), [#8207](FreshRSS/FreshRSS#8207), [#8255](FreshRSS/FreshRSS#8255), [#8254](FreshRSS/FreshRSS#8254), [#8255](FreshRSS/FreshRSS#8255) - Fix SQL performance issue with MySQL, using an index hint [#8211](FreshRSS/FreshRSS#8211) - Scaling of user statistics in Web UI and CLI, to help instances with 1k+ users [#8277](FreshRSS/FreshRSS#8277) - API streaming of large responses for reducing memory consumption and increasing speed [#8041](FreshRSS/FreshRSS#8041) - Security - 💥 Move unsafe autologin to an extension [#7958](FreshRSS/FreshRSS#7958) - Fix some CSRFs [#8035](FreshRSS/FreshRSS#8035) - Strengthen some crypto (login, tokens, nonces) [#8061](FreshRSS/FreshRSS#8061), [#8320](FreshRSS/FreshRSS#8320) - Create separate HTTP `Retry-After` rules for proxies [#8029](FreshRSS/FreshRSS#8029), [#8218](FreshRSS/FreshRSS#8218) - Add `data:` to CSP in subscription controller [#8253](FreshRSS/FreshRSS#8253) - Improve anonymous authentication logic [#8165](FreshRSS/FreshRSS#8165) - Enable GitHub [release immutability](https://github.blog/changelog/2025-10-28-immutable-releases-are-now-generally-available/) [#8205](FreshRSS/FreshRSS#8205) - Bug fixing - Exclude local networks for domain-wide HTTP `Retry-After` [#8195](FreshRSS/FreshRSS#8195) - Fix OpenID Connect with Debian 13 [#8032](FreshRSS/FreshRSS#8032) - Fix MySQL / MariaDB bug wrongly sorting new articles [#8223](FreshRSS/FreshRSS#8223) - Fix MySQL / MariaDB database size calculation [#8282](FreshRSS/FreshRSS#8282) - Fix SQLite bind bug when adding user label [#8101](FreshRSS/FreshRSS#8101) - Fix SQL auto-update of field `f.kind` to ease migrations from FreshRSS versions older than 1.20.0 [#8148](FreshRSS/FreshRSS#8148) - Fix search encoding and quoting [#8311](FreshRSS/FreshRSS#8311), [#8324](FreshRSS/FreshRSS#8324), [#8338](FreshRSS/FreshRSS#8338) - Fix handling of database unexpected null content (during migrations) [#8319](FreshRSS/FreshRSS#8319), [#8321](FreshRSS/FreshRSS#8321) - Fix drag & drop of user query losing information [#8113](FreshRSS/FreshRSS#8113) - Fix DOM error while filtering retrieved full content [#8132](FreshRSS/FreshRSS#8132), [#8161](FreshRSS/FreshRSS#8161) - Fix `config.custom.php` during install [#8033](FreshRSS/FreshRSS#8033) - Fix do not mark important feeds as read from category [#8067](FreshRSS/FreshRSS#8067) - Fix regression of warnings in Web browser console due to lack of `window.bcrypt` object [#8166](FreshRSS/FreshRSS#8166) - Fix chart resize regression due to `chart.js` v4 update [#8298](FreshRSS/FreshRSS#8298) - Fix CLI user creation warning when language is not given [#8283](FreshRSS/FreshRSS#8283) - Fix merging of custom HTTP headers [#8251](FreshRSS/FreshRSS#8251) - Fix bug in the case of duplicated mark-as-read filters [#8322](FreshRSS/FreshRSS#8322) - SimplePie - Fix support of HTTP trailer headers [#7983](FreshRSS/FreshRSS#7983), [simplepie#943](simplepie/simplepie#943) - Apply HTTPS policy also on GUIDs and permalinks [#8037](FreshRSS/FreshRSS#8037), [simplepie#951](simplepie/simplepie#951) - Fix `WordPress.com` HTTP duplicates with WebSub [Automattic/pushpress#16](Automattic/pushpress#16) - Implement HTML whitelist for SimplePie sanitizer [#7924](FreshRSS/FreshRSS#7924), [simplepie#947](simplepie/simplepie#947) - Various upstream contributions [simplepie#940](simplepie/simplepie#940), [simplepie#944](simplepie/simplepie#944) - Deployment - Docker default image updated to Debian 13 Trixie with PHP 8.4.11 and Apache 2.4.65 [#8032](FreshRSS/FreshRSS#8032) - Docker alternative image updated to Alpine 3.23 with PHP 8.4.15 and Apache 2.4.65 [#8285](FreshRSS/FreshRSS#8285) - Fix Docker healthcheck `cli/health.php` compatibility with OpenID Connect [#8040](FreshRSS/FreshRSS#8040) - Improve Docker for compatibility with other base images such as Arch Linux [#8299](FreshRSS/FreshRSS#8299) - Improve `cli/access-permissions.sh` to detect the correct permission Web group such as `www-data`, `apache`, or `http` - Update PostgreSQL volume for Docker [#8216](FreshRSS/FreshRSS#8216), [#8224](FreshRSS/FreshRSS#8224) - Catch lack of `exec()` function for git update [#8228](FreshRSS/FreshRSS#8228) - Work around `DOMDocument::saveHTML()` scrambling charset encoding in some versions of libxml2 [#8296](FreshRSS/FreshRSS#8296) - Improve configuration checks for PHP extensions (in Web UI and CLI), including recommending e.g. `php-intl` [#8334](FreshRSS/FreshRSS#8334) - UI - New button for toggling sidebar on desktop view [#8201](FreshRSS/FreshRSS#8201), [#8286](FreshRSS/FreshRSS#8286) - Better transitions between groups of articles [#8174](FreshRSS/FreshRSS#8174) - New links in transitions and jump ⏭ to next transition [#8294](FreshRSS/FreshRSS#8294) - More visible selected article [#8230](FreshRSS/FreshRSS#8230) - Show the parsed search query instead of the original user input [#8293](FreshRSS/FreshRSS#8293), [#8306](FreshRSS/FreshRSS#8306), [#8341](FreshRSS/FreshRSS#8341) - Show search query in the page title [#8217](FreshRSS/FreshRSS#8217) - Scroll into filtered feed/category on page load in the sidebar [#8281](FreshRSS/FreshRSS#8281), [#8307](FreshRSS/FreshRSS#8307) - Fix autocomplete issues in change password form [#7812](FreshRSS/FreshRSS#7812) - Fix navigating between read feeds using shortcut <kbd>shift</kbd>+<kbd>j</kbd>/<kbd>k</kbd> [#8057](FreshRSS/FreshRSS#8057) - Dark background in Web app manifest to avoid white flash when opening [#8140](FreshRSS/FreshRSS#8140) - Increase button visibility in UI to change theme [#8149](FreshRSS/FreshRSS#8149) - Replace arrow navigation in theme switcher with `<select>` [#8190](FreshRSS/FreshRSS#8190) - Improve scroll of article after load of user labels [#7962](FreshRSS/FreshRSS#7962) - Keep scroll state of page when closing the slider [#8295](FreshRSS/FreshRSS#8295), [#8301](FreshRSS/FreshRSS#8301) - Scroll into filtered feed/category on page load [#8281](FreshRSS/FreshRSS#8281) - Display sidebar dropdowns above if no space below [#8335](FreshRSS/FreshRSS#8335), [#8336](FreshRSS/FreshRSS#8336) - Use native CSS instead of SCSS [#8200](FreshRSS/FreshRSS#8200), [#8241](FreshRSS/FreshRSS#8241) - Using [CSS nesting](https://developer.mozilla.org/en-US/docs/Web/CSS/Guides/Nesting) and [relative colours](https://developer.mozilla.org/en-US/docs/Web/CSS/Guides/Colors/Using_relative_colors). - Various UI and style improvements: [#8171](FreshRSS/FreshRSS#8171), [#8185](FreshRSS/FreshRSS#8185), [#8196](FreshRSS/FreshRSS#8196) - JavaScript finalise migration from `Promise` to `async`/`await`: [#8182](FreshRSS/FreshRSS#8182) - API - API performance optimisation: streaming of large responses [#8041](FreshRSS/FreshRSS#8041) - Fever API: Add `with_ids` parameter to mass-change read/unread/saved/unsaved on lists of articles [#8312](FreshRSS/FreshRSS#8312) - Misc API: better REST error semantics [#8232](FreshRSS/FreshRSS#8232) - Extensions - Add support for extension priority [#8038](FreshRSS/FreshRSS#8038) - Add support for extension compatibility [#8081](FreshRSS/FreshRSS#8081) - Improve PHP code with hook enums [#8036](FreshRSS/FreshRSS#8036) - New hook `nav_entries` [#8054](FreshRSS/FreshRSS#8054) - Rename [Extensions](https://github.com/FreshRSS/Extensions) default branch from *master* to *main* [#8194](FreshRSS/FreshRSS#8194) - I18n - Translation status as text in README [#7842](FreshRSS/FreshRSS#7842) - Add new translate CLI commands `move` [#8214](FreshRSS/FreshRSS#8214) - Change some regional language codes to comply with RFC 5646 / IETF BCP 47 / ISO 3166 / ISO 639-1 [#8065](FreshRSS/FreshRSS#8065) - Improve German [#8028](FreshRSS/FreshRSS#8028) - Improve Greek [#8146](FreshRSS/FreshRSS#8146) - Improve Finnish [#8073](FreshRSS/FreshRSS#8073), [#8092](FreshRSS/FreshRSS#8092) - Improve Hungarian [#8244](FreshRSS/FreshRSS#8244) - Improve Italian [#8115](FreshRSS/FreshRSS#8115), [#8186](FreshRSS/FreshRSS#8186) - Improve Polish [#8134](FreshRSS/FreshRSS#8134), [#8135](FreshRSS/FreshRSS#8135) - Improve Russian [#8155](FreshRSS/FreshRSS#8155), [#8197](FreshRSS/FreshRSS#8197) - Improve Simplified Chinese [#8308](FreshRSS/FreshRSS#8308), [#8313](FreshRSS/FreshRSS#8313) - Misc. - Add code to modify a search expression [#8293](FreshRSS/FreshRSS#8293) - Remove *Pocket* sharing service [#8127](FreshRSS/FreshRSS#8127), [#8128](FreshRSS/FreshRSS#8128) - Update to PHPMailer 7.0.1 [#8048](FreshRSS/FreshRSS#8048), [#8180](FreshRSS/FreshRSS#8180), [#8272](FreshRSS/FreshRSS#8272) - 💥 Housekeeping of `lib_rss.php` with potential breaking changes for some extensions [#8193](FreshRSS/FreshRSS#8193), - Use native PHP `#[Deprecated]` [#8325](FreshRSS/FreshRSS#8325) - Improve PHP code [#8156](FreshRSS/FreshRSS#8156), [#8203](FreshRSS/FreshRSS#8203), [#8284](FreshRSS/FreshRSS#8284), [#8292](FreshRSS/FreshRSS#8292), [#8297](FreshRSS/FreshRSS#8297) - GitHub Actions: `--no-progress` [#8315](FreshRSS/FreshRSS#8315) - Update dev dependencies [#8043](FreshRSS/FreshRSS#8043), [#8044](FreshRSS/FreshRSS#8044), [#8045](FreshRSS/FreshRSS#8045), [#8046](FreshRSS/FreshRSS#8046), [#8047](FreshRSS/FreshRSS#8047), [#8052](FreshRSS/FreshRSS#8052), [#8176](FreshRSS/FreshRSS#8176), [#8177](FreshRSS/FreshRSS#8177), [#8178](FreshRSS/FreshRSS#8178), [#8179](FreshRSS/FreshRSS#8179), [#8210](FreshRSS/FreshRSS#8210), [#8270](FreshRSS/FreshRSS#8270), [#8271](FreshRSS/FreshRSS#8271), [#8273](FreshRSS/FreshRSS#8273), [#8274](FreshRSS/FreshRSS#8274), [#8275](FreshRSS/FreshRSS#8275), [#8276](FreshRSS/FreshRSS#8276) </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR is behind base branch, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).  Reviewed-on: https://gitea.alexlebens.dev/alexlebens/infrastructure/pulls/2851 Co-authored-by: Renovate Bot <[email protected]> Co-committed-by: Renovate Bot <[email protected]>

API optimisation: more streaming of outputs

a2f044b

I spotted a memory issue when testing FreshRSS#7714 Attempt to stream results more, instead of keeping too much in memory. Could be further improved.

Alkarex added API 🤝 API for other clients php Pull requests that update Php code labels Oct 1, 2025

Alkarex added this to the 1.28.0 milestone Oct 1, 2025

Alkarex commented Oct 1, 2025

View reviewed changes

aledeg reviewed Oct 1, 2025

View reviewed changes

app/Models/Feed.php Show resolved Hide resolved

p/api/greader.php Outdated Show resolved Hide resolved

p/api/greader.php Outdated Show resolved Hide resolved

p/api/greader.php Outdated Show resolved Hide resolved

aledeg reviewed Oct 1, 2025

View reviewed changes

p/api/greader.php Show resolved Hide resolved

Alkarex and others added 3 commits October 1, 2025 13:53

Apply suggestions from code review

f0a4c3c

Co-authored-by: Alexis Degrugillier <[email protected]>

Minor optimisations

4e3bcfb

Merge branch 'edge' into api-optimise-streaming

9b24e7e

aledeg approved these changes Oct 1, 2025

View reviewed changes

Alkarex added 3 commits October 1, 2025 16:58

Fix JSON syntax

c50c962

Minor whitespace JSON formatting

12f5dc3

Minor whitespace JSON formatting

54539e6

Alkarex marked this pull request as ready for review October 1, 2025 15:08

aledeg approved these changes Oct 1, 2025

View reviewed changes

Alkarex merged commit 2601897 into FreshRSS:edge Oct 1, 2025
1 check passed

Alkarex deleted the api-optimise-streaming branch October 1, 2025 18:21

Alkarex added a commit to Alkarex/FreshRSS that referenced this pull request Oct 1, 2025

API client "Your News": one star sync

03cad1a

Due to high bandwidth in current sync FreshRSS#8041 (comment) Follow-up of FreshRSS#7714

Alkarex mentioned this pull request Oct 1, 2025

API client "Your News": one star sync #8051

Merged

Alkarex added a commit that referenced this pull request Oct 1, 2025

API client "Your News": one star sync (#8051)

0eab229

Due to high bandwidth in current sync #8041 (comment) Follow-up of #7714

Alkarex mentioned this pull request Dec 31, 2025

[Bug] Most 3rd party clients failing to sync due to timeout errors since 1.28.0 update #8370

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

API optimisation: more streaming of outputs#8041

API optimisation: more streaming of outputs#8041
Alkarex merged 7 commits intoFreshRSS:edgefrom
Alkarex:api-optimise-streaming

Alkarex commented Oct 1, 2025

Uh oh!

Alkarex Oct 1, 2025

Uh oh!

aledeg left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Alkarex commented Oct 1, 2025

Uh oh!

Uh oh!

Alkarex commented Oct 1, 2025

Uh oh!

TijnvandenEijnde commented Oct 2, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Comments

Uh oh!

Conversation

Alkarex commented Oct 1, 2025

Uh oh!

Alkarex Oct 1, 2025

Choose a reason for hiding this comment

Uh oh!

aledeg left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Alkarex commented Oct 1, 2025

Uh oh!

Uh oh!

Alkarex commented Oct 1, 2025

Uh oh!

TijnvandenEijnde commented Oct 2, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Comments