Add `lockfileContents` option to `loadPyodide()` by hoodmane · Pull Request #5764 · pyodide/pyodide

hoodmane · 2025-07-14T13:18:43Z

And also packageBaseUrl option:

If lockfileContents is provided and not packageBaseUrl, then in the browser attempting to load a lock entry with a relative path fails.
In Node, packageBaseUrl is used in place of jsdelivr as the cdn url if passed.

See discussion in #5736.

Add a CHANGELOG entry
Add / update tests
Add new / update outdated documentation

And also `packageBaseUrl` option: 1. If `lockfileContents` is provided and not `packageBaseUrl`, then in the browser attempting to load a lock entry with a relative path fails. 2. In Node, `packageBaseUrl` is used in place of jsdelivr as the cdn url if passed.

for more information, see https://pre-commit.ci

ryanking13

Thanks. Ideally, I would like to deprecate LockFileURL instead of having two different options doing the same thing. WDYT?

src/js/types.ts

src/js/pyodide.ts

src/js/load-package.ts

src/tests/test_package_loading.py

src/js/pyodide.ts

hoodmane · 2025-07-15T14:47:27Z

Thanks for the review @ryanking13!

hoodmane · 2025-07-15T16:48:14Z

I would like to deprecate LockFileURL instead of having two different options doing the same thing.

I don't think it's worth the churn to deprecate this since it causes no extra maintenance burden.

WebReflection · 2025-07-22T08:27:37Z

If lockfileContents is provided and not packageBaseUrl, then in the browser attempting to load a lock entry with a relative path fails.

does it mean that with content created via micropip.freeze() we still need to create an object URL to pass as packageBaseURL ?

I am trying to implement the dance that must work on any pyodide version and this detail seems to be very important to understand properly ... I am not sure micropip.freeze() does, or will, provide relative paths internally so it'd be great to understand if our JSON Blob artifact is needed at all with lockfileContents, thank you!

ryanking13 · 2025-07-22T10:26:01Z

@WebReflection

Here are some changes related to the lockfile.

Before 0.28.0: We assumed that the Pyodide runtime and packages are always served from the same place. So if the lockfile has relative URLs in it, it always loads the packages from the indexURL + filename.
Pyodide 0.28.0 (micropip 0.10.0): This assumption has changed and we now allow loading Pyodide runtime and packages in different places. Therefore, we changed the behavior related to lockFileURL parameter. If the lockfile has relative URLs in it, and if indexURL is provided, the pacakges are loaded from parent(lockFileURL) + filename.
Pyodide 0.28.1 (micropip 0.10.1) (to be)
- The behavior of micropip.freeze() has been changed. micropip.freeze() will now output the lockfile with absolute URLs. So if you generate a lockfile using micropip.freeze() and serve it through indexedDB, it will load the packages correctly (which was an error in Pyodide 0.28).
- If the lockfile contains relative URLs, it will still load the packages from parent(lockFileURL) + filename.
- This PR adds lockFileContents and packageBaseURL.
  - If lockFileContents is provided without packageBaseURL and if the lockfile has no absolute URLs, we fail (we don't know where to load packages)
  - If packageBaseURL is provided, then we load from packageBaseURL + filename.

So...

does it mean that with content created via micropip.freeze() we still need to create an object URL to pass as packageBaseURL ?

In Pyodide 0.28.1, no. micropip.freeze() will output absolute URLs for all the packages in the lockfile, so packageBaseURL will not be necessary.

I am not sure micropip.freeze() does, or will, provide relative paths internally so it'd be great to understand if our JSON Blob artifact is needed at all with lockfileContents, thank you!

micropip.freeze() will generate absolute URLs by default from 0.10.1 (Pyodide 0.28.1). And you'll be able to provide it directly to lockfileContents. Otherwise, creating the Blob URL and passing it to lockfileURL will also work.

WebReflection · 2025-07-22T13:29:45Z

@ryanking13 crystal clear, thank you!

…ing>

for more information, see https://pre-commit.ci

ryanking13

Thanks. There is a test failure, otherwise looks good to me.

ryanking13 · 2025-07-29T13:24:10Z

src/js/pyodide.ts

+  options_.cdnUrl =
+    options_.packageBaseUrl ??
+    `https://cdn.jsdelivr.net/pyodide/v${version}/full/`;


Shouldn't it always be https://cdn.jsdelivr.net/pyodide/v${version}/full/, as we use this only for a fallback?

I am not sure. But:

the caching system only works if file_name is relative, otherwise we will try to download the url every time and won't do any caching

if the user has a custom lock file, then any additional packages definitely won't be available on jsdelivr

So if they are to have any chance of using the caching mechanism with their own lock file, they need to be able to adjust the cdnURL.

ryanking13 · 2025-07-29T13:25:34Z

src/js/pyodide.ts

  const emscriptenSettings = createSettings(config);
  const API = emscriptenSettings.API;
-  API.lockFilePromise = loadLockFile(config.lockFileURL);
+  API.lockFilePromise = Promise.resolve(options_.lockFileContents);


It is a bit confusing to have all of config, option, and option_ here. But I guess I'll be able clean it up in #5746.

Well options_ and options are identical things just with different types.

hoodmane · 2025-07-30T10:46:13Z

The failing test doesn't actually pass for me on the main branch...

hoodmane · 2025-07-30T10:47:03Z

Maybe my micropip is out of date...

This reverts commit f4939cb.

for more information, see https://pre-commit.ci

And also `packageBaseUrl` option: 1. If `lockfileContents` is provided and not `packageBaseUrl`, then in the browser attempting to load a lock entry with a relative path fails. 2. In Node, `packageBaseUrl` is used in place of jsdelivr as the cdn url if passed. See discussion in pyodide#5736.

ryanking13 · 2025-07-31T11:40:47Z

Thanks!

And also `packageBaseUrl` option: 1. If `lockfileContents` is provided and not `packageBaseUrl`, then in the browser attempting to load a lock entry with a relative path fails. 2. In Node, `packageBaseUrl` is used in place of jsdelivr as the cdn url if passed. See discussion in pyodide#5736.

WebReflection · 2025-07-31T23:55:51Z

any eta for a new release? we're going on vacation and we'd love to be able to bring in latest Pyodide before doing that, so that anyone playing around our project could benefit from its latest/greatest, thank you!

And also `packageBaseUrl` option: 1. If `lockfileContents` is provided and not `packageBaseUrl`, then in the browser attempting to load a lock entry with a relative path fails. 2. In Node, `packageBaseUrl` is used in place of jsdelivr as the cdn url if passed. See discussion in pyodide#5736.

hoodmane force-pushed the lockfile-contents branch from 02d380a to 7de7090 Compare July 14, 2025 13:42

hoodmane force-pushed the lockfile-contents branch from 9613447 to df050ea Compare July 14, 2025 13:45

pre-commit-ci bot and others added 4 commits July 14, 2025 13:47

[pre-commit.ci] auto fixes from pre-commit.com hooks

a0e115c

for more information, see https://pre-commit.ci

Add back lockfileBaseUrl public api

1fc091c

Try to fix test

32c0bc6

Fix again

34a36ba

ryanking13 self-requested a review July 15, 2025 09:41

hoodmane added 2 commits July 15, 2025 12:27

Merge branch 'main' into lockfile-contents

fe2b06e

Merge branch 'main' into lockfile-contents

654ad14

ryanking13 reviewed Jul 15, 2025

View reviewed changes

WebReflection mentioned this pull request Jul 22, 2025

Preparing better lockFileContents for Pyodide 0.28.1 pyscript/polyscript#144

Merged

hoodmane added 6 commits July 28, 2025 17:36

name field is unnormalized

e825434

Make lockfileContents type Lockfile | string | Promise<Lockfile | str…

35dd28e

…ing>

Update

8da9e45

Clearer error message

dc2d126

Merge branch 'main' into lockfile-contents

08feb85

Update changelog

0b04001

hoodmane added this to the 0.28.1 milestone Jul 29, 2025

hoodmane added the needs backport label Jul 29, 2025

pre-commit-ci bot and others added 3 commits July 29, 2025 09:46

[pre-commit.ci] auto fixes from pre-commit.com hooks

57ce7db

for more information, see https://pre-commit.ci

Fix docs

6432835

Update expected docs

f124a78

hoodmane force-pushed the lockfile-contents branch from eee6723 to f124a78 Compare July 29, 2025 10:13

ryanking13 approved these changes Jul 29, 2025

View reviewed changes

Revert some stuff

f4939cb

hoodmane and others added 2 commits July 30, 2025 14:35

Revert "Revert some stuff"

55cc16f

This reverts commit f4939cb.

[pre-commit.ci] auto fixes from pre-commit.com hooks

0beb3e5

for more information, see https://pre-commit.ci

hoodmane merged commit 427d2b1 into pyodide:main Jul 31, 2025
38 of 40 checks passed

hoodmane deleted the lockfile-contents branch July 31, 2025 10:01

ryanking13 mentioned this pull request Aug 7, 2025

Error loading micropip with pyodide version 0.28.1 #5821

Closed

Copilot AI mentioned this pull request Aug 7, 2025

Fix micropip loading regression in Node.js environments #5823

Closed

ryanking13 mentioned this pull request Aug 7, 2025

Fix cdnURL in Node #5824

Merged

2 tasks

hoodmane removed the needs backport label Aug 18, 2025

Uh oh!

Conversation

hoodmane commented Jul 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ryanking13 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hoodmane commented Jul 15, 2025

Uh oh!

hoodmane commented Jul 15, 2025

Uh oh!

WebReflection commented Jul 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ryanking13 commented Jul 22, 2025

Uh oh!

WebReflection commented Jul 22, 2025

Uh oh!

ryanking13 left a comment

Choose a reason for hiding this comment

Uh oh!

ryanking13 Jul 29, 2025

Choose a reason for hiding this comment

Uh oh!

hoodmane Jul 29, 2025

Choose a reason for hiding this comment

Uh oh!

ryanking13 Jul 29, 2025

Choose a reason for hiding this comment

Uh oh!

hoodmane Jul 29, 2025

Choose a reason for hiding this comment

Uh oh!

hoodmane commented Jul 30, 2025

Uh oh!

hoodmane commented Jul 30, 2025

Uh oh!

Uh oh!

ryanking13 commented Jul 31, 2025

Uh oh!

WebReflection commented Jul 31, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

hoodmane commented Jul 14, 2025 •

edited

Loading

WebReflection commented Jul 22, 2025 •

edited

Loading