feat: allow for async parsing in file loader#14932
feat: allow for async parsing in file loader#14932Princesseuh merged 8 commits intowithastro:mainfrom
Conversation
🦋 Changeset detectedLatest commit: e0b0267 The changes in this PR will be included in the next version bump. Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
CodSpeed Performance ReportMerging this PR will not alter performanceComparing Summary
|
Princesseuh
left a comment
There was a problem hiding this comment.
That seems fine to me! Any objection you see @ascorbic perhaps?
|
From a docs standpoint, just checking here: is this "allows the option for the Here's the existing docs for the This change is a little tricker than normal because the v6 docs have been significantly overhauled for content collections, and don't match up cleanly with our existing ones. So, I would also ask for a second docs PR made to the v6 branch too, so we can port the changes over smoothly for our revised documentation over there. Once we've clarified exactly what the impact of this change is, then @ArmandPhilippot and I can help with the changeset, and with suggesting what should happen in docs! |
Looking at the type, this is "allows" (nothing is required, unless there is an uncaught side effect somewhere in the implementation... but I don't think so, and this would be a bug). So, the user should not need to do any updates and I don't think this requires a lot of updates in docs. But, yes, we would need to update the which is bit long (3 lines on desktop view with my resolution, I think we have at least another type on 3 lines but this is not the most readable unfortunately). I wonder if it would make sense to have a public type alias for Which I think improves the readability. And we could use a link on |
|
OK, thanks! So, if I'm understanding this correctly, it's necessary to update the |
There was a problem hiding this comment.
Looks great, @Princesseuh !
Regarding updating the docs themselves, since currently we describe using the parser() function for situations where Astro doesn't automatically parse for you (e.g. .json files), it might be worth adding this example in the Content Collections guide because now there's a reason you might write a parser function even if Astro can parse the file type.
This could be a small section before/after ##### Nested json that is something like ##### Async parsing (??? something short) to describe this use case. (whatever the existing heading level is)
v5 current docs: https://docs.astro.build/en/guides/content-collections/#nested-json-documents
The structure is a little different on the overall page in v6 docs, but the same sections exist: https://v6.docs.astro.build/en/guides/content-collections/#nested-json-documents
If not a full section, we can simply update the existing text in these sections where it talks about the file loader's parser (not actually added in bold), something like:
The file() loader creates multiple entries from a single local file. Each entry in the file must have a unique id key property. It accepts a base file path to your file and optionally a parser function for data files it cannot parse automatically, or to parse data asynchronously.
Support for parsing single JSON, YAML, and TOML files into collection entries with the
file()loader is built-in (unless you have a nested JSON document) or need to return a Promise to parse data asyncronously. To load your collection from unsupported file types (e.g..csv) or to await data before parsing, you will need to create a parser function.
(The v6 docs don't have the nice compact sentence of the first one. So we may decide we want a section for v6 docs so we can just link like we do re: nested json. But that doesn't have to block this PR, as long as the idea is added, and docs can always be updated later in v6 if we so decide!)
So to recap, the reference entry for parser() at https://docs.astro.build/en/reference/content-loader-reference/#parser needs updating (both in current docs and in v6 docs), and in each of those PRs, I'd suggest adding a new tiny section described above. (Doesn't have to be big! Just basically what's in this changelog, except not from the "now you can..." perspective. Instead framed like, "You can also..."
Co-authored-by: Sarah Rainsberger <[email protected]>
ascorbic
left a comment
There was a problem hiding this comment.
Thanks! Excellent change
|
v5 docs: withastro/docs#13108 |
sarah11918
left a comment
There was a problem hiding this comment.
Approving this PR for docs! (Left a comment in the docs PR re: reference still needs updating) but this should just be a quick thing!
This PR contains the following updates: | Package | Change | [Age](https://docs.renovatebot.com/merge-confidence/) | [Confidence](https://docs.renovatebot.com/merge-confidence/) | |---|---|---|---| | [astro](https://astro.build) ([source](https://github.com/withastro/astro/tree/HEAD/packages/astro)) | [`5.16.15` → `5.17.1`](https://renovatebot.com/diffs/npm/astro/5.16.15/5.17.1) |  |  | --- ### Release Notes <details> <summary>withastro/astro (astro)</summary> ### [`v5.17.1`](https://github.com/withastro/astro/blob/HEAD/packages/astro/CHANGELOG.md#5171) [Compare Source](https://github.com/withastro/astro/compare/[email protected]@5.17.1) ##### Patch Changes - [#​15334](withastro/astro#15334) [`d715f1f`](withastro/astro@d715f1f) Thanks [@​florian-lefebvre](https://github.com/florian-lefebvre)! - **BREAKING CHANGE to the experimental Fonts API only** Removes the `getFontBuffer()` helper function exported from `astro:assets` when using the experimental Fonts API This experimental feature introduced in v15.6.13 ended up causing significant memory usage during build. This feature has been removed and will be reintroduced after further exploration and testing. If you were relying on this function, you can replicate the previous behavior manually: - On prerendered routes, read the file using `node:fs` - On server rendered routes, fetch files using URLs from `fontData` and `context.url` ### [`v5.17.0`](https://github.com/withastro/astro/blob/HEAD/packages/astro/CHANGELOG.md#5170) [Compare Source](https://github.com/withastro/astro/compare/[email protected]@5.17.0) ##### Minor Changes - [#​14932](withastro/astro#14932) [`b19d816`](withastro/astro@b19d816) Thanks [@​patrickarlt](https://github.com/patrickarlt)! - Adds support for returning a Promise from the `parser()` option of the `file()` loader This enables you to run asynchronous code such as fetching remote data or using async parsers when loading files with the Content Layer API. For example: ```js import { defineCollection } from 'astro:content'; import { file } from 'astro/loaders'; const blog = defineCollection({ loader: file('src/data/blog.json', { parser: async (text) => { const data = JSON.parse(text); // Perform async operations like fetching additional data const enrichedData = await fetch(`https://api.example.com/enrich`, { method: 'POST', body: JSON.stringify(data), }).then((res) => res.json()); return enrichedData; }, }), }); export const collections = { blog }; ``` See [the `parser()` reference documentation](https://docs.astro.build/en/reference/content-loader-reference/#parser) for more information. - [#​15171](withastro/astro#15171) [`f220726`](withastro/astro@f220726) Thanks [@​mark-ignacio](https://github.com/mark-ignacio)! - Adds a new, optional `kernel` configuration option to select a resize algorithm in the Sharp image service By default, Sharp resizes images with the `lanczos3` kernel. This new config option allows you to set the default resizing algorithm to any resizing option supported by [Sharp](https://sharp.pixelplumbing.com/api-resize/#resize) (e.g. `linear`, `mks2021`). Kernel selection can produce quite noticeable differences depending on various characteristics of the source image - especially drawn art - so changing the kernel gives you more control over the appearance of images on your site: ```js export default defineConfig({ image: { service: { entrypoint: 'astro/assets/services/sharp', config: { kernel: "mks2021" } } }) ``` This selection will apply to all images on your site, and is not yet configurable on a per-image basis. For more information, see [Sharps documentation on resizing images](https://sharp.pixelplumbing.com/api-resize/#resize). - [#​15063](withastro/astro#15063) [`08e0fd7`](withastro/astro@08e0fd7) Thanks [@​jmortlock](https://github.com/jmortlock)! - Adds a new `partitioned` option when setting a cookie to allow creating partitioned cookies. [Partitioned cookies](https://developer.mozilla.org/en-US/docs/Web/Privacy/Guides/Privacy_sandbox/Partitioned_cookies) can only be read within the context of the top-level site on which they were set. This allows cross-site tracking to be blocked, while still enabling legitimate uses of third-party cookies. You can create a partitioned cookie by passing `partitioned: true` when setting a cookie. Note that partitioned cookies must also be set with `secure: true`: ```js Astro.cookies.set('my-cookie', 'value', { partitioned: true, secure: true, }); ``` For more information, see the [`AstroCookieSetOptions` API reference](https://docs.astro.build/en/reference/api-reference/#astrocookiesetoptions). - [#​15022](withastro/astro#15022) [`f1fce0e`](withastro/astro@f1fce0e) Thanks [@​ascorbic](https://github.com/ascorbic)! - Adds a new `retainBody` option to the `glob()` loader to allow reducing the size of the data store. Currently, the `glob()` loader stores the raw body of each content file in the entry, in addition to the rendered HTML. The `retainBody` option defaults to `true`, but you can set it to `false` to prevent the raw body of content files from being stored in the data store. This significantly reduces the deployed size of the data store and helps avoid hitting size limits for sites with very large collections. The rendered body will still be available in the `entry.rendered.html` property for markdown files, and the `entry.filePath` property will still point to the original file. ```js import { defineCollection } from 'astro:content'; import { glob } from 'astro/loaders'; const blog = defineCollection({ loader: glob({ pattern: '**/*.md', base: './src/content/blog', retainBody: false, }), }); ``` When `retainBody` is `false`, `entry.body` will be `undefined` instead of containing the raw file contents. - [#​15153](withastro/astro#15153) [`928529f`](withastro/astro@928529f) Thanks [@​jcayzac](https://github.com/jcayzac)! - Adds a new `background` property to the `<Image />` component. This optional property lets you pass a background color to flatten the image with. By default, Sharp uses a black background when flattening an image that is being converted to a format that does not support transparency (e.g. `jpeg`). Providing a value for `background` on an `<Image />` component, or passing it to the `getImage()` helper, will flatten images using that color instead. This is especially useful when the requested output format doesn't support an alpha channel (e.g. `jpeg`) and can't support transparent backgrounds. ```astro --- import { Image } from 'astro:assets'; --- <Image src="/transparent.png" alt="A JPEG with a white background!" format="jpeg" background="#ffffff" /> ``` See more about this new property in [the image reference docs](https://docs.astro.build/en/reference/modules/astro-assets/#background) - [#​15015](withastro/astro#15015) [`54f6006`](withastro/astro@54f6006) Thanks [@​tony](https://github.com/tony)! - Adds optional `placement` config option for the dev toolbar. You can now configure the default toolbar position (`'bottom-left'`, `'bottom-center'`, or `'bottom-right'`) via `devToolbar.placement` in your Astro config. This option is helpful for sites with UI elements (chat widgets, cookie banners) that are consistently obscured by the toolbar in the dev environment. You can set a project default that is consistent across environments (e.g. dev machines, browser instances, team members): ```js // astro.config.mjs export default defineConfig({ devToolbar: { placement: 'bottom-left', }, }); ``` User preferences from the toolbar UI (stored in `localStorage`) still take priority, so this setting can be overridden in individual situations as necessary. ### [`v5.16.16`](https://github.com/withastro/astro/blob/HEAD/packages/astro/CHANGELOG.md#51616) [Compare Source](https://github.com/withastro/astro/compare/[email protected]@5.16.16) ##### Patch Changes - [#​15281](withastro/astro#15281) [`a1b80c6`](withastro/astro@a1b80c6) Thanks [@​matthewp](https://github.com/matthewp)! - Ensures server island requests carry an encrypted component export identifier so they do not accidentally resolve to the wrong component. - [#​15304](withastro/astro#15304) [`02ee3c7`](withastro/astro@02ee3c7) Thanks [@​cameronapak](https://github.com/cameronapak)! - Fix: Remove await from getActionResult example - [#​15324](withastro/astro#15324) [`ab41c3e`](withastro/astro@ab41c3e) Thanks [@​Princesseuh](https://github.com/Princesseuh)! - Fixes an issue where certain unauthorized links could be rendered as clickable in the error overlay </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0Mi45Mi4xMiIsInVwZGF0ZWRJblZlciI6IjQyLjkyLjEyIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6W119--> Co-authored-by: Renovate Bot <[email protected]> Co-committed-by: Renovate Bot <[email protected]>
|
@ascorbic @sarah11918 @Princesseuh thanks for getting this through! I kinda forgot about it for a few weeks so it was great to come back and see it merged. |
Changes
parser()option on thefile()loader to return a Promise so that more complex processing can be done if needed.Testing
See included unit test.
Docs
/cc @withastro/maintainers-docs for feedback!
I'm happy to do a matching PR to the docs if preferred.