feat: add retainBody option to the glob() loader#15022
Conversation
🦋 Changeset detectedLatest commit: be0976d The changes in this PR will be included in the next version bump. Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
CodSpeed Performance ReportMerging this PR will not alter performanceComparing Summary
Footnotes |
|
Would it make sense to have this enabled by default in Astro 6? |
matthewp
left a comment
There was a problem hiding this comment.
Blocked as a minor change.
|
@matthewp yeah, I was going to suggest that |
|
While thinking about content store size, it would be good to get #14315 over the line too. Even with an option like this, at some point things can get too big, so chunking feels like a good solution to that general problem. |
|
@delucis yeah, it needs some manual testing and benchmarking though. I would like to be confident it doesn't cause performance regressions. |
commit: |
Thanks. Hopefully it works! While this option is a nice one for advanced optimizations, the “Hit limit => discover obscure option => double check you’re not actually using body anywhere” flow is not the nicest DX, so if Astro can make it rarer to hit those limits, that might be nice. (Although we could also potentially one day consider defaulting to not storing the |
|
Yes, as @matthewp said, I would suggest we make this the default in Astro 6 |
|
Added this to the 5.17 milestone. Not yet sure when that will be. |
|
(Just adding myself here, and nothing that we'll need docs when it gets closer to time!) |
sarah11918
left a comment
There was a problem hiding this comment.
Approving the changeset for docs, and I have a docs PR ready to go (suggestions welcome if you got 'em)!
This PR contains the following updates: | Package | Change | [Age](https://docs.renovatebot.com/merge-confidence/) | [Confidence](https://docs.renovatebot.com/merge-confidence/) | |---|---|---|---| | [astro](https://astro.build) ([source](https://github.com/withastro/astro/tree/HEAD/packages/astro)) | [`5.16.15` → `5.17.1`](https://renovatebot.com/diffs/npm/astro/5.16.15/5.17.1) |  |  | --- ### Release Notes <details> <summary>withastro/astro (astro)</summary> ### [`v5.17.1`](https://github.com/withastro/astro/blob/HEAD/packages/astro/CHANGELOG.md#5171) [Compare Source](https://github.com/withastro/astro/compare/[email protected]@5.17.1) ##### Patch Changes - [#​15334](withastro/astro#15334) [`d715f1f`](withastro/astro@d715f1f) Thanks [@​florian-lefebvre](https://github.com/florian-lefebvre)! - **BREAKING CHANGE to the experimental Fonts API only** Removes the `getFontBuffer()` helper function exported from `astro:assets` when using the experimental Fonts API This experimental feature introduced in v15.6.13 ended up causing significant memory usage during build. This feature has been removed and will be reintroduced after further exploration and testing. If you were relying on this function, you can replicate the previous behavior manually: - On prerendered routes, read the file using `node:fs` - On server rendered routes, fetch files using URLs from `fontData` and `context.url` ### [`v5.17.0`](https://github.com/withastro/astro/blob/HEAD/packages/astro/CHANGELOG.md#5170) [Compare Source](https://github.com/withastro/astro/compare/[email protected]@5.17.0) ##### Minor Changes - [#​14932](withastro/astro#14932) [`b19d816`](withastro/astro@b19d816) Thanks [@​patrickarlt](https://github.com/patrickarlt)! - Adds support for returning a Promise from the `parser()` option of the `file()` loader This enables you to run asynchronous code such as fetching remote data or using async parsers when loading files with the Content Layer API. For example: ```js import { defineCollection } from 'astro:content'; import { file } from 'astro/loaders'; const blog = defineCollection({ loader: file('src/data/blog.json', { parser: async (text) => { const data = JSON.parse(text); // Perform async operations like fetching additional data const enrichedData = await fetch(`https://api.example.com/enrich`, { method: 'POST', body: JSON.stringify(data), }).then((res) => res.json()); return enrichedData; }, }), }); export const collections = { blog }; ``` See [the `parser()` reference documentation](https://docs.astro.build/en/reference/content-loader-reference/#parser) for more information. - [#​15171](withastro/astro#15171) [`f220726`](withastro/astro@f220726) Thanks [@​mark-ignacio](https://github.com/mark-ignacio)! - Adds a new, optional `kernel` configuration option to select a resize algorithm in the Sharp image service By default, Sharp resizes images with the `lanczos3` kernel. This new config option allows you to set the default resizing algorithm to any resizing option supported by [Sharp](https://sharp.pixelplumbing.com/api-resize/#resize) (e.g. `linear`, `mks2021`). Kernel selection can produce quite noticeable differences depending on various characteristics of the source image - especially drawn art - so changing the kernel gives you more control over the appearance of images on your site: ```js export default defineConfig({ image: { service: { entrypoint: 'astro/assets/services/sharp', config: { kernel: "mks2021" } } }) ``` This selection will apply to all images on your site, and is not yet configurable on a per-image basis. For more information, see [Sharps documentation on resizing images](https://sharp.pixelplumbing.com/api-resize/#resize). - [#​15063](withastro/astro#15063) [`08e0fd7`](withastro/astro@08e0fd7) Thanks [@​jmortlock](https://github.com/jmortlock)! - Adds a new `partitioned` option when setting a cookie to allow creating partitioned cookies. [Partitioned cookies](https://developer.mozilla.org/en-US/docs/Web/Privacy/Guides/Privacy_sandbox/Partitioned_cookies) can only be read within the context of the top-level site on which they were set. This allows cross-site tracking to be blocked, while still enabling legitimate uses of third-party cookies. You can create a partitioned cookie by passing `partitioned: true` when setting a cookie. Note that partitioned cookies must also be set with `secure: true`: ```js Astro.cookies.set('my-cookie', 'value', { partitioned: true, secure: true, }); ``` For more information, see the [`AstroCookieSetOptions` API reference](https://docs.astro.build/en/reference/api-reference/#astrocookiesetoptions). - [#​15022](withastro/astro#15022) [`f1fce0e`](withastro/astro@f1fce0e) Thanks [@​ascorbic](https://github.com/ascorbic)! - Adds a new `retainBody` option to the `glob()` loader to allow reducing the size of the data store. Currently, the `glob()` loader stores the raw body of each content file in the entry, in addition to the rendered HTML. The `retainBody` option defaults to `true`, but you can set it to `false` to prevent the raw body of content files from being stored in the data store. This significantly reduces the deployed size of the data store and helps avoid hitting size limits for sites with very large collections. The rendered body will still be available in the `entry.rendered.html` property for markdown files, and the `entry.filePath` property will still point to the original file. ```js import { defineCollection } from 'astro:content'; import { glob } from 'astro/loaders'; const blog = defineCollection({ loader: glob({ pattern: '**/*.md', base: './src/content/blog', retainBody: false, }), }); ``` When `retainBody` is `false`, `entry.body` will be `undefined` instead of containing the raw file contents. - [#​15153](withastro/astro#15153) [`928529f`](withastro/astro@928529f) Thanks [@​jcayzac](https://github.com/jcayzac)! - Adds a new `background` property to the `<Image />` component. This optional property lets you pass a background color to flatten the image with. By default, Sharp uses a black background when flattening an image that is being converted to a format that does not support transparency (e.g. `jpeg`). Providing a value for `background` on an `<Image />` component, or passing it to the `getImage()` helper, will flatten images using that color instead. This is especially useful when the requested output format doesn't support an alpha channel (e.g. `jpeg`) and can't support transparent backgrounds. ```astro --- import { Image } from 'astro:assets'; --- <Image src="/transparent.png" alt="A JPEG with a white background!" format="jpeg" background="#ffffff" /> ``` See more about this new property in [the image reference docs](https://docs.astro.build/en/reference/modules/astro-assets/#background) - [#​15015](withastro/astro#15015) [`54f6006`](withastro/astro@54f6006) Thanks [@​tony](https://github.com/tony)! - Adds optional `placement` config option for the dev toolbar. You can now configure the default toolbar position (`'bottom-left'`, `'bottom-center'`, or `'bottom-right'`) via `devToolbar.placement` in your Astro config. This option is helpful for sites with UI elements (chat widgets, cookie banners) that are consistently obscured by the toolbar in the dev environment. You can set a project default that is consistent across environments (e.g. dev machines, browser instances, team members): ```js // astro.config.mjs export default defineConfig({ devToolbar: { placement: 'bottom-left', }, }); ``` User preferences from the toolbar UI (stored in `localStorage`) still take priority, so this setting can be overridden in individual situations as necessary. ### [`v5.16.16`](https://github.com/withastro/astro/blob/HEAD/packages/astro/CHANGELOG.md#51616) [Compare Source](https://github.com/withastro/astro/compare/[email protected]@5.16.16) ##### Patch Changes - [#​15281](withastro/astro#15281) [`a1b80c6`](withastro/astro@a1b80c6) Thanks [@​matthewp](https://github.com/matthewp)! - Ensures server island requests carry an encrypted component export identifier so they do not accidentally resolve to the wrong component. - [#​15304](withastro/astro#15304) [`02ee3c7`](withastro/astro@02ee3c7) Thanks [@​cameronapak](https://github.com/cameronapak)! - Fix: Remove await from getActionResult example - [#​15324](withastro/astro#15324) [`ab41c3e`](withastro/astro@ab41c3e) Thanks [@​Princesseuh](https://github.com/Princesseuh)! - Fixes an issue where certain unauthorized links could be rendered as clickable in the error overlay </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0Mi45Mi4xMiIsInVwZGF0ZWRJblZlciI6IjQyLjkyLjEyIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6W119--> Co-authored-by: Renovate Bot <[email protected]> Co-committed-by: Renovate Bot <[email protected]>
Changes
Adds a new
retainBodyoption to theglob()loader to allow reducing the size of the data store.Currently, the
glob()loader stores the raw body of each content file inentry.body, in addition to the rendered HTML inentry.rendered.html.The
retainBodyoption defaults totrue, but you can set it tofalseto prevent the raw body of content files from being stored in the data store. This significantly reduces the deployed size of the data store and helps avoid hitting size limits for sites with very large collections. This has caused issues for some very large sites, which have hit the maximum JSON string size.The rendered body will still be available in the
entry.rendered.htmlproperty for markdown files, and theentry.filePathproperty will still point to the original file. For MDX collections this will dramatically reduce the size of the collection, as there will no longer be any body retained in the store.When
retainBodyisfalse,entry.bodywill beundefinedinstead of containing the raw file contents.Testing
Docs
withastro/docs#13150