Result of nixpkgs-review pr 327651 run on x86_64-linux 1

Looks good to me so far.

we currently output .nupkg files next to the unpacked packages (as in ~/.nuget/packages). could this be avoided?

Maybe if in fetch-deps we generated a nuget lockfile we could avoid this, but other than that I don't see much of an alternative.

The issue is that nuget lockfiles don't support multi-RID restores, as it only works for the current RID, and if you put in a dependency for a different RID in the lockfile it'll complain that it doesn't need it.
Having one lockfile per RID also wasn't possible last time I tried.

maybe the .nupkgs can be empty and still function as a source for the purpose of version detection?

Maybe if we keep only the nuspec file in there we could somehow hack the restore algo into supporting it, but I don't know how I feel about it, feels a bit flaky.

maybe source and packages should be separate outputs?

I like this idea, I think it'd make things less confusing as one is supposed to be the target directory and the other a source.

Maybe if we keep only the nuspec file in there we could somehow hack the restore algo into supporting it, but I don't know how I feel about it, feels a bit flaky.

I'll do some more testing around this. What's in here right now might not be required for all packages. We could possibly allow multiple options (best first):

1. add all dependency package dirs to fallbackPackageFolders in nuget.config
2. create NUGET_FALLBACK_PACKAGES with symlinks to dependencies
3. create NUGET_PACKAGES with symlinks to dependencies
4. do (3) and also add a package source with all the dependencies .nupkgs (pretty much what this PR currently does)

maybe source and packages should be separate outputs?

One problem with this is that it obfuscates runtime deps, unless we make the .nupkgs uncompressed, or extract the dependencies.

I'll do some more testing around this. What's in here right now might not be required for all packages. We could possibly allow multiple options (best first):

1. add all dependency package dirs to fallbackPackageFolders in nuget.config
2. create NUGET_FALLBACK_PACKAGES with symlinks to dependencies
3. create NUGET_PACKAGES with symlinks to dependencies
4. do (3) and also add a package source with all the dependencies .nupkgs (pretty much what this PR currently does)

I might be mistaken, but 1 and 2 seem to be the same in practice? They do seem like the best option considering that it's immutable and dotnet restore won't try to modify them, but considering we don't want any networking at all in dotnet restore and already have all the dependencies listed and downloaded, I don't see much of a difference between 1, 2 and 3 as in theory none of them would modify the source dir.

Given that most projects don't have lockfiles at all, I don't think we'll be able to get rid of overriding the source with a directory full of nupkgs as it'll try to use online sources instead if we don't do it.

For now I think the most realistic path would be to proceed with 4 as it works on projects without lockfiles and maybe look into requiring lockfiles for projects or something similar so that we can get rid of the directory full of .nupkgs entirely.

I might be mistaken, but 1 and 2 seem to be the same in practice?

Yeah, more or less. It's just that NUGET_FALLBACK_PACKAGES only supports a single dir.

They do seem like the best option considering that it's immutable and dotnet restore won't try to modify them,

Currently nuget-to-nix has to search through one package dir with a mix of built and downloaded packages, so in theory it would simplify that.

Given that most projects don't have lockfiles at all, I don't think we'll be able to get rid of overriding the source with a directory full of nupkgs as it'll try to use online sources instead if we don't do it.

One thing I'm not sure of, and was going to test, is if we can do something like put empty files in place of the .nupkg files. If it's only using the source to determine which versions are available, maybe that'll work?

Another crazier option might be to make a fake http source using a daemon?

Yeah, more or less. It's just that NUGET_FALLBACK_PACKAGES only supports a single dir.

It actually supports multiple, but they need to be separated by semicolons:

The value must contain the full path or paths to the fallback package folders. Multiple paths may be included by setting the value to a ; delimited list.

Currently nuget-to-nix has to search through one package dir with a mix of built and downloaded packages, so in theory it would simplify that.

How so? I thought it only searched through stuff that dotnet restore pulled, so if there's any dependencies on csprojs or nix-built stuff then there wouldn't be any .nupkgs there for it to find.

One thing I'm not sure of, and was going to test, is if we can do something like put empty files in place of the .nupkg files. If it's only using the source to determine which versions are available, maybe that'll work?

Another crazier option might be to make a fake http source using a daemon?

I've asked a question in the .NET Discord to the nuget folks about it, will report back if I get anything, but I think we could get away with putting in a .nupkg with just the .nuspec inside. It needs testing though.

The fake HTTP daemon wouldn't work because it'd still try to download .nupkgs from the daemon so it'd be the same as using a directory as a source.

It actually supports multiple, but they need to be separated by semicolons:

Ah yeah, I don't know why I thought that. I swear I looked it up in the code before, but it's definitely splitting on ;.

How so? I thought it only searched through stuff that dotnet restore pulled, so if there's any dependencies on csprojs or nix-built stuff then there wouldn't be any .nupkgs there for it to find.

It searches $NUGET_PACKAGES, which includes all the things that have been symlinked from dependencies (sdk packages, etc). It looks at .nupkg.metadata, which is empty in packages from the nix store. Interestingly, (unlike dotnet restore) dotnet tool restore doesn't seem to fill in .nupkg.metadata with the source it used to download.

The fake HTTP daemon wouldn't work because it'd still try to download .nupkgs from the daemon so it'd be the same as using a directory as a source.

I wasn't sure if it would actually download them or just enumerate the available versions.

I think we could get away with putting in a .nupkg with just the .nuspec inside. It needs testing though.

This is a good idea to try.

I think we could get away with putting in a .nupkg with just the .nuspec inside. It needs testing though.

It does seem to work with an uncompressed zip containing only the nuspec. Something like that would be safe to put in outputs.

Result of nixpkgs-review pr 327651 run on x86_64-linux 1

I've updated the description to match the latest version. Now we:

• point $NUGET_FALLBACK_PACKAGES at the dependencies

We also unconditionally:

• create a 'stub' source and add it to nuget.config

In theory this would be optional if all dependencies were locked, but that's rare enough that I'm not sure it makes sense for it to be configurable yet.

If linkNugetPackages is set, we:

• link all $NUGET_FALLBACK_PACKAGES into $NUGET_PACKAGES

This is required to work around upstream bugs like tunnelvisionlabs/ReferenceAssemblyAnnotator#94

If installable is passed to mkNugetDeps, we:

• make full .nupkg files in the source instead of stubs.

This is required for dotnet tool install, because it can only install from the .nupkg, and not from an installed package. It's also required for some VMR dependencies.

Result of nixpkgs-review pr 327651 run on x86_64-linux 1

(I'm currently on holiday and diligently not doing anything that looks like work, but would in principle be interested in 1. this PR and 2. being on the dotnet team!)

@Smaug123 have a good holiday and feel free to make a PR to add yourself here:

By the way, don't take my "I'm interested" as a desire to block this PR on my account if it's ready to go in before I get back!

Result of nixpkgs-review run on x86_64-linux 1

This is actually breaking some tests, like nexusmods-app.tests.help:

Unhandled exception. System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.Extensions.Hosting.Abstractions, Version=8.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'. The system cannot find the file specified.

File name: 'Microsoft.Extensions.Hosting.Abstractions, Version=8.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'
/build/.attr-0l2nkwhif96f51f4amnlf414lhl4rv9vh8iffyp431v6s28gsr90: line 1:     5 Aborted                 (core dumped) /nix/store/riv0cmlbvkfydvglbgcc5iip6fdhjnm7-nexusmods-app-0.4.1/bin/NexusMods.App --help

Working on that now.

I fixed another problem with nexusmods-app, and now all tests in the packages built by nixpkgs-review pass on linux. I also updated the docs to reflect the projectReferences change.

I'm planning on merging this soon. I just wanted to check out some closure sizes. Some things will be larger uncompressed, but should be similar compressed.

I also checked a few packages to confirm that build times are improved, and it seems likely to save in the neighborhood of 30s from restore in a package with a reasonable number of dependencies.

Apologies for necro, I was really hoping the weird bug would magically solve itself and I wouldn't have to bisect all of Nixpkgs. But it didn't and I'm finally here.
Something in d3ca502 caused my <Move/>s within the project output directory to fail with MSB3677 re: read-only... and only when passing in dotnet-sdk = nixpkgs24_05.dotnet-sdk_6;. Before, my whole expression including .NET 8 and .NET 6 was drawn from Nixpkgs 24.05, and I was plugging in .NET 5 from 22.11 without issue.
I noticed you added -p:OverwriteReadOnlyFiles=true when invoking dotnet build so I tried that (since my expression overrides buildPhase and invokes dotnet build directly), but it didn't work, nor did export MSBUILDALWAYSOVERWRITEREADONLYFILES=1.
These files are assemblies created during MSBuild's execution. Do you know why it would suddenly be creating them read-only?

edit: I patched in OverwriteReadOnlyFiles="true" and .NET 6 is now working, but .NET 5 is still failing with Unable to load the service index for source https://api.nuget.org/v3/index.json during dotnetConfigureHook. I copied in the --source like it used to have but then building fails with error CS0518: Predefined type 'System.String' is not defined or imported. Is there a better way to build old projects with the old SDKs but current Nix infrastructure?