Skip to content

Comments

fetchToStore(): Avoid duplicate copying if the input is already a store path#10511

Open
edolstra wants to merge 2 commits intoNixOS:masterfrom
edolstra:optimize-fetchToStore
Open

fetchToStore(): Avoid duplicate copying if the input is already a store path#10511
edolstra wants to merge 2 commits intoNixOS:masterfrom
edolstra:optimize-fetchToStore

Conversation

@edolstra
Copy link
Member

@edolstra edolstra commented Apr 15, 2024

Motivation

This is needed for the path:// input scheme (until it returns a FSInputAccessor to the original path, but that's blocked by #10089) and the Mercurial input scheme.

Note: it would be cleaner to have fetchToStore() as an overridable method on InputAccessor (as was originally the case), but the latter got moved to libutil so we can't do that anymore.

Context

Priorities and Process

Add 👍 to pull requests you find important.

The Nix maintainer team uses a GitHub project board to schedule and track reviews.

@github-actions github-actions bot added the fetching Networking with the outside (non-Nix) world, input locking label Apr 15, 2024
Copy link
Member

@Ericson2314 Ericson2314 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need isStorePath? I don't think so

// an FSInputAccessor pointing to a store path.
if (path.accessor->isStorePath
&& path.path.isRoot()
&& method == FileIngestionMethod::Recursive
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What goes wrong if we skip this? I am trying to special-case NAR hashing less in Nix currently.

Suggested change
&& method == FileIngestionMethod::Recursive

Copy link
Member

@Ericson2314 Ericson2314 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Meant that to not be approval

@Ericson2314 Ericson2314 self-assigned this Apr 15, 2024
@edolstra
Copy link
Member Author

Do we need isStorePath? I don't think so

We do need that to ensure it's only used for store paths where we know their ingestion method.

As mentioned above, the architecturally cleaner way is to make fetchToStore() a method in InputAccessor (which is pretty much why that class exists in the first place), but it would require moving InputAccessor back to libfetcher. Which would be a much bigger change.

@Ericson2314
Copy link
Member

I think I rather solve this in a different way:

  1. SourceAccessor has optional content address (on the FSO level, so (FileIngestionMethod, Hash) pair)
  2. fetchToStore can use that to construct a store path and see if it is in the store
  3. only add to store if it isn't.

I am still hoping InputAccessor can go away entirely.

@Ericson2314
Copy link
Member

If you could take a stab at #10467 (review) @edolstra, I would be happy to take on this one per my plan above.

…re path

This is needed for the path:// input scheme (until it returns a
FSInputAccessor to the original path, but that's blocked by NixOS#10089)
and the Mercurial input scheme.
@edolstra edolstra force-pushed the optimize-fetchToStore branch from 893c383 to 8d08846 Compare April 19, 2024 14:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fetching Networking with the outside (non-Nix) world, input locking

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants