feat: Split data store into chunks#14315
Conversation
|
github-actions
bot
added
the
pkg: astro
CodSpeed Performance ReportMerging #14315 will not alter performanceComparing Summary
Footnotes |
|
This approach solves the issue for Cloudflare chunks, but I think it's a bit narrow. I'd rather also try to solve the issue of stringified objects being too large, by serialising the chunks individually and splitting by number of entries. Once doing that it would probably make sense to split by collection too. |
|
@ascorbic I've changed the splitting approach to cover more use cases, as you've explained. It's now split by collection -> chunks of 1000 entries -> devalue -> further split string to not exceed 20MB. I've ensured to cover all places, including the Vite virtual module. It now uses dynamic imports, to make sure it's not combined into one big module file during build. Feel free to take another look at the PR. Unfortunately, I'm stuck with some failing tests. I'd like to ask you for help with debugging 🙏 It looks like the failing tests are related to changes/invalidations/restarts. I struggle to recreate that manually, so maybe it's some kind of race condition? Or duplicate watching? IDK, I'm not sure. |
|
FYI, this will need to be behind an experimental flag for now, so you're going to need to add that into the logic. |
ascorbic
left a comment
ascorbic
left a comment
There was a problem hiding this comment.
Thanks. I've not tested it yet, but it's looking promising. Have you been able to detect any difference in performance? It might be good to add test cases in the benchmarks dir (see benchmark/make-project/markdown-cc2.js). This will need a minor changeset too. See https://contribute.docs.astro.build/docs-for-code-changes/changesets/ and https://contribute.docs.astro.build/docs-for-code-changes/experimental-feature-docs/
| * @version 5.14 | ||
| * @description | ||
| * | ||
| * When enabled, ... |
|
Please merge this PR into new version as soon as possible. |
|
I was wondering if there has been any progress on this pull request? It has been approximately three months since it was opened, and I wanted to check in on its status.
Separating the issue resolution from the proposal might help avoid unnecessary delays in addressing the core problem. |
3 participants
Changes
devalueto stringify the chunk of 1000 entriesManifest format:
{ "collectionName": [ ["collectionName.hash1.js", "collectionName.hash2.js"], // 1000 entries, split into two strings ["collectionName.hash3.js"], // rest of entries ], "anotherCollection": ["..."] }Testing
Content layer tests have already great coverage, I've added test cases where relevant to run the suite with experimental flag as well.
Docs
Needs a new experimental feature page.
/cc @withastro/maintainers-docs