Skip to content

feat: allow for async parsing in file loader#14932

Merged
Princesseuh merged 8 commits intowithastro:mainfrom
patrickarlt:async-file-parse
Jan 28, 2026
Merged

feat: allow for async parsing in file loader#14932
Princesseuh merged 8 commits intowithastro:mainfrom
patrickarlt:async-file-parse

Conversation

@patrickarlt
Copy link
Contributor

Changes

Testing

See included unit test.

Docs

/cc @withastro/maintainers-docs for feedback!

I'm happy to do a matching PR to the docs if preferred.

@changeset-bot
Copy link

changeset-bot bot commented Nov 29, 2025

🦋 Changeset detected

Latest commit: e0b0267

The changes in this PR will be included in the next version bump.

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@github-actions github-actions bot added the pkg: astro Related to the core `astro` package (scope) label Nov 29, 2025
@codspeed-hq
Copy link

codspeed-hq bot commented Nov 29, 2025

CodSpeed Performance Report

Merging this PR will not alter performance

Comparing patrickarlt:async-file-parse (e0b0267) with main (edabeaa)

Summary

✅ 9 untouched benchmarks

Copy link
Member

@Princesseuh Princesseuh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That seems fine to me! Any objection you see @ascorbic perhaps?

@Princesseuh Princesseuh added this to the 5.17.0 milestone Jan 15, 2026
@sarah11918
Copy link
Member

From a docs standpoint, just checking here: is this "allows the option for the parser() function to be async if you add await" or is it "updates the parser() function to require await and always run asynchronously"? And, if it's now updated, could this possibly break anyone's existing use of the parser() function? Is there anything they might have to check for to be sure?

Here's the existing docs for the parser() function that we would need to figure out how this affects, and we'd also have to check for any examples (e.g. in the Content collections guide, astro:assets API, and perhaps elsewhere throughout docs that if we show examples. So we'd need a PR to current live docs that does a sweep just to make sure we haven't got places where we've relied on the old behaviour.

This change is a little tricker than normal because the v6 docs have been significantly overhauled for content collections, and don't match up cleanly with our existing ones. So, I would also ask for a second docs PR made to the v6 branch too, so we can port the changes over smoothly for our revised documentation over there.

Once we've clarified exactly what the impact of this change is, then @ArmandPhilippot and I can help with the changeset, and with suggesting what should happen in docs!

@ArmandPhilippot
Copy link
Member

is this "allows the option for the parser() function to be async if you add await" or is it "updates the parser() function to require await and always run asynchronously"?

Looking at the type, this is "allows" (nothing is required, unless there is an uncaught side effect somewhere in the implementation... but I don't think so, and this would be a bug). So, the user should not need to do any updates and I don't think this requires a lot of updates in docs.

But, yes, we would need to update the Type: on parser() because this is now async. However, because we tend to show only public types (ie. importable)... This would be something like:

(text: string) => Record<string, Record<string, unknown>> | Array<Record<string, unknown>> | Promise<Record<string, Record<string, unknown>> | Array<Record<string, unknown>>>

which is bit long (3 lines on desktop view with my resolution, I think we have at least another type on 3 lines but this is not the most readable unfortunately).

I wonder if it would make sense to have a public type alias for data (unless I'm wrong and this doesn't match data?). We could keep Record<string, unknown> under the data section and have something like CollectionData (a bit shorter, or something similar but shorter than Record<string, unknown>) for the other types. This would allows us to have something like:

(text: string) => Record<string, CollectionData> | Array<CollectionData> | Promise<Record<string, CollectionData> | Array<CollectionData>>

Which I think improves the readability. And we could use a link on CollectionData for those who would like to see the full type.

@sarah11918
Copy link
Member

OK, thanks! So, if I'm understanding this correctly, it's necessary to update the parser() docs (separate Docs PR needed), and this PR can follow our normal "minor" PR formula! Will leave a suggestion there!

Copy link
Member

@sarah11918 sarah11918 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In addition to making a docs PR to both v5 and v6, some guidance below on the changeset!

@Princesseuh Princesseuh self-assigned this Jan 21, 2026
Copy link
Member

@sarah11918 sarah11918 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great, @Princesseuh !

Regarding updating the docs themselves, since currently we describe using the parser() function for situations where Astro doesn't automatically parse for you (e.g. .json files), it might be worth adding this example in the Content Collections guide because now there's a reason you might write a parser function even if Astro can parse the file type.

This could be a small section before/after ##### Nested json that is something like ##### Async parsing (??? something short) to describe this use case. (whatever the existing heading level is)

v5 current docs: https://docs.astro.build/en/guides/content-collections/#nested-json-documents

The structure is a little different on the overall page in v6 docs, but the same sections exist: https://v6.docs.astro.build/en/guides/content-collections/#nested-json-documents

If not a full section, we can simply update the existing text in these sections where it talks about the file loader's parser (not actually added in bold), something like:

v5
https://docs.astro.build/en/guides/content-collections/#built-in-loaders

The file() loader creates multiple entries from a single local file. Each entry in the file must have a unique id key property. It accepts a base file path to your file and optionally a parser function for data files it cannot parse automatically, or to parse data asynchronously.

v6
https://v6.docs.astro.build/en/guides/content-collections/#the-file-loader

Support for parsing single JSON, YAML, and TOML files into collection entries with the file() loader is built-in (unless you have a nested JSON document) or need to return a Promise to parse data asyncronously. To load your collection from unsupported file types (e.g. .csv) or to await data before parsing, you will need to create a parser function.

(The v6 docs don't have the nice compact sentence of the first one. So we may decide we want a section for v6 docs so we can just link like we do re: nested json. But that doesn't have to block this PR, as long as the idea is added, and docs can always be updated later in v6 if we so decide!)

So to recap, the reference entry for parser() at https://docs.astro.build/en/reference/content-loader-reference/#parser needs updating (both in current docs and in v6 docs), and in each of those PRs, I'd suggest adding a new tiny section described above. (Doesn't have to be big! Just basically what's in this changelog, except not from the "now you can..." perspective. Instead framed like, "You can also..."

Co-authored-by: Sarah Rainsberger <[email protected]>
Copy link
Contributor

@ascorbic ascorbic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Excellent change

@Princesseuh
Copy link
Member

v5 docs: withastro/docs#13108
I pushed to v6 directly because Git is too hard for me: withastro/docs@98c833d, but it's also being reviewed!

Copy link
Member

@sarah11918 sarah11918 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving this PR for docs! (Left a comment in the docs PR re: reference still needs updating) but this should just be a quick thing!

@Princesseuh Princesseuh merged commit b19d816 into withastro:main Jan 28, 2026
25 of 26 checks passed
@astrobot-houston astrobot-houston mentioned this pull request Jan 28, 2026
dadezzz pushed a commit to dadezzz/ice-notes that referenced this pull request Feb 7, 2026
This PR contains the following updates:

| Package | Change | [Age](https://docs.renovatebot.com/merge-confidence/) | [Confidence](https://docs.renovatebot.com/merge-confidence/) |
|---|---|---|---|
| [astro](https://astro.build) ([source](https://github.com/withastro/astro/tree/HEAD/packages/astro)) | [`5.16.15` → `5.17.1`](https://renovatebot.com/diffs/npm/astro/5.16.15/5.17.1) | ![age](https://developer.mend.io/api/mc/badges/age/npm/astro/5.17.1?slim=true) | ![confidence](https://developer.mend.io/api/mc/badges/confidence/npm/astro/5.16.15/5.17.1?slim=true) |

---

### Release Notes

<details>
<summary>withastro/astro (astro)</summary>

### [`v5.17.1`](https://github.com/withastro/astro/blob/HEAD/packages/astro/CHANGELOG.md#5171)

[Compare Source](https://github.com/withastro/astro/compare/[email protected]@5.17.1)

##### Patch Changes

- [#&#8203;15334](withastro/astro#15334) [`d715f1f`](withastro/astro@d715f1f) Thanks [@&#8203;florian-lefebvre](https://github.com/florian-lefebvre)! - **BREAKING CHANGE to the experimental Fonts API only**

  Removes the `getFontBuffer()` helper function exported from `astro:assets` when using the experimental Fonts API

  This experimental feature introduced in v15.6.13 ended up causing significant memory usage during build. This feature has been removed and will be reintroduced after further exploration and testing.

  If you were relying on this function, you can replicate the previous behavior manually:

  - On prerendered routes, read the file using `node:fs`
  - On server rendered routes, fetch files using URLs from `fontData` and `context.url`

### [`v5.17.0`](https://github.com/withastro/astro/blob/HEAD/packages/astro/CHANGELOG.md#5170)

[Compare Source](https://github.com/withastro/astro/compare/[email protected]@5.17.0)

##### Minor Changes

- [#&#8203;14932](withastro/astro#14932) [`b19d816`](withastro/astro@b19d816) Thanks [@&#8203;patrickarlt](https://github.com/patrickarlt)! - Adds support for returning a Promise from the `parser()` option of the `file()` loader

  This enables you to run asynchronous code such as fetching remote data or using async parsers when loading files with the Content Layer API.

  For example:

  ```js
  import { defineCollection } from 'astro:content';
  import { file } from 'astro/loaders';

  const blog = defineCollection({
    loader: file('src/data/blog.json', {
      parser: async (text) => {
        const data = JSON.parse(text);

        // Perform async operations like fetching additional data
        const enrichedData = await fetch(`https://api.example.com/enrich`, {
          method: 'POST',
          body: JSON.stringify(data),
        }).then((res) => res.json());

        return enrichedData;
      },
    }),
  });

  export const collections = { blog };
  ```

  See [the `parser()` reference documentation](https://docs.astro.build/en/reference/content-loader-reference/#parser) for more information.

- [#&#8203;15171](withastro/astro#15171) [`f220726`](withastro/astro@f220726) Thanks [@&#8203;mark-ignacio](https://github.com/mark-ignacio)! - Adds a new, optional `kernel` configuration option to select a resize algorithm in the Sharp image service

  By default, Sharp resizes images with the `lanczos3` kernel. This new config option allows you to set the default resizing algorithm to any resizing option supported by [Sharp](https://sharp.pixelplumbing.com/api-resize/#resize) (e.g. `linear`, `mks2021`).

  Kernel selection can produce quite noticeable differences depending on various characteristics of the source image - especially drawn art - so changing the kernel gives you more control over the appearance of images on your site:

  ```js
  export default defineConfig({
    image: {
      service: {
        entrypoint: 'astro/assets/services/sharp',
        config: {
          kernel: "mks2021"
        }
    }
  })
  ```

  This selection will apply to all images on your site, and is not yet configurable on a per-image basis. For more information, see [Sharps documentation on resizing images](https://sharp.pixelplumbing.com/api-resize/#resize).

- [#&#8203;15063](withastro/astro#15063) [`08e0fd7`](withastro/astro@08e0fd7) Thanks [@&#8203;jmortlock](https://github.com/jmortlock)! - Adds a new `partitioned` option when setting a cookie to allow creating partitioned cookies.

  [Partitioned cookies](https://developer.mozilla.org/en-US/docs/Web/Privacy/Guides/Privacy_sandbox/Partitioned_cookies) can only be read within the context of the top-level site on which they were set. This allows cross-site tracking to be blocked, while still enabling legitimate uses of third-party cookies.

  You can create a partitioned cookie by passing `partitioned: true` when setting a cookie. Note that partitioned cookies must also be set with `secure: true`:

  ```js
  Astro.cookies.set('my-cookie', 'value', {
    partitioned: true,
    secure: true,
  });
  ```

  For more information, see the [`AstroCookieSetOptions` API reference](https://docs.astro.build/en/reference/api-reference/#astrocookiesetoptions).

- [#&#8203;15022](withastro/astro#15022) [`f1fce0e`](withastro/astro@f1fce0e) Thanks [@&#8203;ascorbic](https://github.com/ascorbic)! - Adds a new `retainBody` option to the `glob()` loader to allow reducing the size of the data store.

  Currently, the `glob()` loader stores the raw body of each content file in the entry, in addition to the rendered HTML.

  The `retainBody` option defaults to `true`, but you can set it to `false` to prevent the raw body of content files from being stored in the data store. This significantly reduces the deployed size of the data store and helps avoid hitting size limits for sites with very large collections.

  The rendered body will still be available in the `entry.rendered.html` property for markdown files, and the `entry.filePath` property will still point to the original file.

  ```js
  import { defineCollection } from 'astro:content';
  import { glob } from 'astro/loaders';

  const blog = defineCollection({
    loader: glob({
      pattern: '**/*.md',
      base: './src/content/blog',
      retainBody: false,
    }),
  });
  ```

  When `retainBody` is `false`, `entry.body` will be `undefined` instead of containing the raw file contents.

- [#&#8203;15153](withastro/astro#15153) [`928529f`](withastro/astro@928529f) Thanks [@&#8203;jcayzac](https://github.com/jcayzac)! - Adds a new `background` property to the `<Image />` component.

  This optional property lets you pass a background color to flatten the image with. By default, Sharp uses a black background when flattening an image that is being converted to a format that does not support transparency (e.g. `jpeg`). Providing a value for `background` on an `<Image />` component, or passing it to the `getImage()` helper, will flatten images using that color instead.

  This is especially useful when the requested output format doesn't support an alpha channel (e.g. `jpeg`) and can't support transparent backgrounds.

  ```astro
  ---
  import { Image } from 'astro:assets';
  ---

  <Image
    src="/transparent.png"
    alt="A JPEG with a white background!"
    format="jpeg"
    background="#ffffff"
  />
  ```

  See more about this new property in [the image reference docs](https://docs.astro.build/en/reference/modules/astro-assets/#background)

- [#&#8203;15015](withastro/astro#15015) [`54f6006`](withastro/astro@54f6006) Thanks [@&#8203;tony](https://github.com/tony)! - Adds optional `placement` config option for the dev toolbar.

  You can now configure the default toolbar position (`'bottom-left'`, `'bottom-center'`, or `'bottom-right'`) via `devToolbar.placement` in your Astro config. This option is helpful for sites with UI elements (chat widgets, cookie banners) that are consistently obscured by the toolbar in the dev environment.

  You can set a project default that is consistent across environments (e.g. dev machines, browser instances, team members):

  ```js
  // astro.config.mjs
  export default defineConfig({
    devToolbar: {
      placement: 'bottom-left',
    },
  });
  ```

  User preferences from the toolbar UI (stored in `localStorage`) still take priority, so this setting can be overridden in individual situations as necessary.

### [`v5.16.16`](https://github.com/withastro/astro/blob/HEAD/packages/astro/CHANGELOG.md#51616)

[Compare Source](https://github.com/withastro/astro/compare/[email protected]@5.16.16)

##### Patch Changes

- [#&#8203;15281](withastro/astro#15281) [`a1b80c6`](withastro/astro@a1b80c6) Thanks [@&#8203;matthewp](https://github.com/matthewp)! - Ensures server island requests carry an encrypted component export identifier so they do not accidentally resolve to the wrong component.

- [#&#8203;15304](withastro/astro#15304) [`02ee3c7`](withastro/astro@02ee3c7) Thanks [@&#8203;cameronapak](https://github.com/cameronapak)! - Fix: Remove await from getActionResult example

- [#&#8203;15324](withastro/astro#15324) [`ab41c3e`](withastro/astro@ab41c3e) Thanks [@&#8203;Princesseuh](https://github.com/Princesseuh)! - Fixes an issue where certain unauthorized links could be rendered as clickable in the error overlay

</details>

---

### Configuration

📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box

---

This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0Mi45Mi4xMiIsInVwZGF0ZWRJblZlciI6IjQyLjkyLjEyIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6W119-->

Co-authored-by: Renovate Bot <[email protected]>
Co-committed-by: Renovate Bot <[email protected]>
@patrickarlt
Copy link
Contributor Author

@ascorbic @sarah11918 @Princesseuh thanks for getting this through! I kinda forgot about it for a few weeks so it was great to come back and see it merged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pkg: astro Related to the core `astro` package (scope)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants