Remove editable from the lock-file

Problem description

Currently, the lock-file stores whether a Pypi package should be installed as editable or not. However, this has some downsides:

If you change a package from editable to non-editable or vice-versa, the lock file has to be regenerated.
Whether a package is editable or not is not defined in the name but it is currently stored in the package data part of the lock file. E.g.
```
- kind: pypi
  name: foo
  version: 0.1.0
  path: foo
  sha256: 5eb7d27a464b108fef2bb090ef1d2f62a2750e7bd625c70d2b00ef3f19a2edf7
  editable: true
```
This means that for every environment the package is now editable.

I see two solutions:

Add editable to a per environment section (similar to extras) and remove it from the package data. This still requires the lock file to be regenerated if the editableness of a package changes.
Remove it from the lock-file all together. This means we have to check at installation time if the package needs to be "re-installed".

I think I'm in favor of option 2 but I'm curious about other people's opinions.

@pavelzw

Also @pavelzw this would fix the some of the issues you were facing as well. Where you include them into the same solve group.

It does slightly break one of the goals of the lock-file:

//! * To be complete. The lock-file should contain all the information needed to recreate
//!   environments even years after it was created. As long as the package data persists that a
//!   lock-file refers to, it should be possible to recreate the environment.

But I think its fine because the environment is still valid, just not perse editable.

I'm facing this issue. I don't know if an opt-in/out option to track the SHA of the editable package would make sense.

I can see use-case that you want to track the exact package installed through the SHA or sometimes, you are in a dev setting where the editable is your current project in development where you don't want to bother with this info.

@baszalmstra

@baszalmstra Any news on this one?
I just encountered this problem where the sha256 of my (editable) package is the only change in my pixi.lock file....

There is a related/duplicate issue; #2512

As I commented there, the sha256 should be dropped since should not be required and creates lockfile churn.

We recently switched our dev envs to pixi and are using setuptools-scm. This means whenever you run pixi without the --frozen argument on a feature branch, the pixi.lock file changes. Meaning, unless people know about this and explicitly revert the change, we are seeing a diff on every single PR, e.g. here.

I'm very much in favor of removing editable packages from the lock file. Note that uv recently did the same for basically the same reason. See astral-sh/uv#7533 and astral-sh/uv#10622.

@baszalmstra

Yeah this makes a lot of sense to me. So I think we should take the version into account when resolving but leave it out of the lock. I think only in the case of a dynamic version, I'm guessing that's what uv is doing when reading the PR description.

Then we should maybe also emit both the editable Boolean and the sha in case of an editable. But check the state of things on install.

That would give us:

No lock file changes when switching between editable and non-editable.
No resolve when changing something other than the version in the metadata. But perhaps a reinstallation.
When using a dynamic version, even changing the version should not trigger a re-solve. I'm not sure what would happen when this version would suddenly start conflicting with a dependency though. We might need to look closer to how uv is handling this.

As an aside:
We need to make this change so that old lock files with this info keep working.

wdyt @baszalmstra @ruben-arts?

All for this improvement!

I'm only worried about the requirement for a resolve/installation. When do we now call for a resolve? Should pixi install or at least pixi lock still see it as not satisfied and try a resolve? My gut says yes. But we then have different behavior from pixi run and pixi shell.

Sorry could you maybe re-phrase. I don't see why it should function differently for the different subcommands :)

I have two issues with the current behaviour:

In my CI when I install an env pixi is re-resolving solely because of the editable install which takes up a huge amount of time.
When I install an env the re-resolve updates the hash in my (committed) pixi.lock file causing git describe to show as dirty.

I would like to be able to do a pixi install --locked -e <env> and have it Just Work when I have an editable install such that no time consuming and lock-file-changing re-resolve occurs.

Hey Dave! So are you using something like a dynamic version ala hatch-vcs or something? Because currently it should only re-solve when the metadata of the editable changes. Otherwise it's another bug :)

Yeah - using dynamic version via hatch-vcs so that the package versions are strongly tied to the source code and are orderable.

Cool that make sense then, we need to implement something similar to uv's cache-keys as well so that we do rebuild on changes and this also needs to be applied to pixi build :)

I'm really close to implementing pixi (really awesome btw) in a few of my repos but got really bit hard by this one and became a bit of a showstopper.

I have a repo with a bunch of dependencies that equate to roughly ~2GB. Since we do dynamic versioning and bump version on every commit to main, the lock file gets updated which means our Docker images trigger a layer rebuild forcing us to transfer ~2GB every time v reusing a cached layer even though it would just be a file or two of source code that changed.

Not really sure if there's a good way to work around it.

@jeromehmu

@jeromehmu (and others on this thread) please make sure to give a 👍 at the top of this issue as they're often used to prioritise work which impacts the most users.

For myself, this also is the biggest problem I have with rolling out pixi more broadly at our org so I'm watching any updates closely...

@jeromehmu

Hey @jeromehmu and @dhirschfeld. That you require this totally makes sense to me, I've written up what needs to be changed for the PyPI use-case. Which I can share, I'll split it up in two comments (this write-up includes hash-based pyproject validation):

So there are couple of problems when using editable PyPI path dependencies, specifically when using dynamic versions. These mostly are relevant to PyPI editables, but are also relevant for pixi build.

Introduction

The main problems are:

People want to switch between editable/non-editable without lock-file changes or resolving

Currently, locking is incompatible with dynamic versioning. This is because the version changes with source changes like when working with hatch-vcs. Resulting in re-solve and lock file change

Like noted here: #3762 (comment), but there are other issues. People want the least amount of churn, as possible while developing, changing the author field in a pyproject.toml now causes a full lock file change for example.

Things what are causing these problems are:

Hash based on pyproject.toml and pixi.toml so a lot of changes cause a re-solve
Because editable is included in the lock file, when changing this the lock file changes. Causing a lot of churn.
-> Same holds for dynamic versions

For some prior art see:

astral-sh/uv#7533

Solution uv for dynamic versions: astral-sh/uv#10622.

For caching: https://docs.astral.sh/uv/reference/settings/#cache-keys

Solutions

We should omit both the editable boolean and the sha for path dependencies. But check the state of things on install.

That would give us:

No lock file changes when switching between editable and non-editable.
No resolve when changing something other than the version in the metadata. But perhaps a reinstallation.

When using a dynamic version, even changing the version should not trigger a re-solve. I'm not sure what would happen when this version would suddenly start conflicting with a dependency though. We might need to look closer to how uv is handling this.

We would need to maybe store the metadata explictly, uv is doing this when having a path dependency on click:

[[package]]
name = "click"
version = "8.3.1"
source = { directory = "click" }
dependencies = [
    { name = "colorama", marker = "sys_platform == 'win32'" },
]

[package.metadata]
requires-dist = [{ name = "colorama", marker = "sys_platform == 'win32'" }]

[package.metadata.requires-dev]
dev = [
    { name = "ruff" },
    { name = "tox" },
    { name = "tox-uv" },
]
docs = [
    { name = "myst-parser" },
    { name = "pallets-sphinx-themes" },
    { name = "sphinx" },
    { name = "sphinx-tabs" },
    { name = "sphinxcontrib-log-cabinet" },
]
docs-auto = [{ name = "sphinx-autobuild" }]
gha-update = [{ name = "gha-update", marker = "python_full_version >= '3.12'" }]
pre-commit = [
    { name = "pre-commit" },
    { name = "pre-commit-uv" },
]
tests = [{ name = "pytest" }]
typing = [
    { name = "mypy" },
    { name = "pyright" },
    { name = "pytest" },
]

Some extra points:

We need an extra way to determine when a package needs to be rebuild, something similar to: https://docs.astral.sh/uv/concepts/cache/#dynamic-metadata

Pixi build

We are different from uv in that we need this for our own build-system as well (I, Tim) think all of this should be kind of transferable to pixi-build. Especially the cache-keys feature is important here.

The thing is I thought we wanted to figure all this out for pixi build first. But seeing the amount of thumbs-up to just your comment Dave, we might need to prioritize this more. I'll just cc @baszalmstra and @ruben-arts, so they see this coming up. And @nichmor which works on PyPI issues as well.

@pavelzw

First off, for a good write-up on docker practices @pavelzw has a great one here: https://tech.quantco.com/blog/pixi-production.

Next up the current workarounds, which are not ideal:

Split your environments into a docker and non-docker feature, this is mostly pertaining to the editable. So you can have a non-editable built into your docker. The big caveat: this does not allow you to use a solve-group though, as this can only select a single package instance and the editable and non-editable are considered to be different. Again, like always, we have ideas to fix this but its not a trival change. We can make another issue for this though.
You can use a postinstall task that does a uv pip install --no-deps /your-project, the downside of-course is that it totally bypasses locking and that you need to add the project dependencies to the pixi.toml.

Maybe there are others as well.

Thanks @jeromehmu for chiming in with a very relevant example, and as always thanks @dhirschfeld I value your opinion a lot :) Let me know if you see any holes in the proposal. Sorry to not be able to help more currently!

I'm not sure what would happen when this version would suddenly start conflicting with a dependency though

I see editable installs as for development so I think they should be very loose in what they accept - i.e. you shouldn't prevent development because of mis-specified dependencies. Given that, I think an editable install of package XYZ should always satisfy any constraint by other packages on XYZ, even if those other constraints are in conflict - i.e. an editable install should satisfy any and all constraints against it.

Not sure if that's technically feasible. In any case, I can't think of package that depends on a package that depens on it, so maybe that's a moot point?

I am new to the concept of lockfile workflows, so I'm still trying to figure out how things can work. I have wondered about the distinction between development and deployment and am not clear on how both can be well supported wrt lockfiles 🤔

In developent I want to install my package in editable mode. The version will come from hatch-vcs which will include the SHA of the commit. This is critical for provenance/reproducibility - you need to know what code generated any given result. For development packages that's the SHA of the git commit, for production packages that's the annotated tag.

In my CI I want to cache GitHub actions / docker builds based on the contents of the manifest - if there are no changes to the manifest then it should be good to load the previous environment from the cache.

I have been looking at caching based on the pixi.lock file but have run into the issue here of the lockfile changing with every commit. Writing this out, I'm now wondering if caching on the manifest rather than the lock file is the solution to that problem?

The re-resolve caused by the changes to the lockfile on every commit does add a lot of redundant time to the CI so it would be good to also solve the problem of triggering a re-resolve based on dynamic versions tied to the git commit.

A bit of a late reply but there has been such thoughtful and thorough responses so thank you so much!

In regards to possible workarounds

I've tried approach 1 and it doesn't quite work for my use case since it isolates the environment so it gets tricky having to recognize things with PATH, if they're across the environment. I also tried installing a feature like envs/xyz and then trying to copy the contents of some other feature in there (e.g. cp -r envs/abc/* envs/xyz) but that doesn't work since the shebangs in some files in bin/ are hardcoded to the environment folder path.

Another thing, since I have a mixture of PyPi and conda dependencies, having my core project say as feature xyz will also automatically include python which makes installing feature xyz by itself with no dependencies ~310MB based on a du -d1 -h .pixi/envs/xyz

I haven't gotten to trying 2 yet. I believe that means I have to explicitly add uv as a dependency as well. This could maybe, be an option if Docker and uv would be smart enough to cache the layer with no changes correctly. I'll have to toy with this option a bit.

Appreciate the suggestions!

I am also not quite versed with dealing with lockfiles. We currently use a combination of micromamba and poetry. What I noticed is that poetry.lock for example only has

other package dependency information (not the current package you're working on itself)
metadata for which version their locking system is using
python version
a sha of the lock file itself

Similarly, conda-lock also seems to only store those four things in its lock file from what I can tell. Is that how pixi could go about it ? or is that essentially what is being proposed?

Closed by #5106

Navigation Menu

Repository navigation

Remove `editable` from the lock-file #1274

Problem description

Introduction

For some prior art see:

Solutions

Pixi build

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Remove editable from the lock-file #1274

Description

Problem description

Activity

tdejager commented on Apr 25, 2024

baszalmstra commented on Apr 25, 2024

glemaitre commented on Jun 17, 2024

noamgot commented on Feb 13, 2025

cas-- commented on Apr 8, 2025

pmeier commented on May 28, 2025

tdejager commented on May 28, 2025

ruben-arts commented on May 30, 2025

tdejager commented on May 30, 2025

dhirschfeld commented on May 31, 2025

tdejager commented on May 31, 2025

dhirschfeld commented on May 31, 2025

tdejager commented on May 31, 2025

jeromehmu commented on Jul 17, 2025

dhirschfeld commented on Jul 17, 2025

tdejager commented on Jul 18, 2025

Introduction

For some prior art see:

Solutions

Pixi build

tdejager commented on Jul 18, 2025

dhirschfeld commented on Jul 18, 2025

dhirschfeld commented on Jul 18, 2025

dhirschfeld commented on Jul 18, 2025

jeromehmu commented on Jul 24, 2025

baszalmstra commented on Dec 17, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Remove `editable` from the lock-file #1274