-
Notifications
You must be signed in to change notification settings - Fork 564
[One .NET] simplify native library inclusion #5386
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[One .NET] simplify native library inclusion #5386
Conversation
783d16f to
896f410
Compare
| Include="@(ResolvedFileToPublish)" | ||
| Condition=" '%(Extension)' == '.so' and '%(Filename)' != 'libmono-android.debug' and '%(Filename)' != 'libmono-android.release' " | ||
| /> | ||
| </ItemGroup> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How are we going to handle _AndroidNativeLibraryForFastDev in the .net 6 world? This is used when we fast deploy most native libraries to the override directory rather than include them in the apk.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hopefully some test will fail here, as this was removed for .NET 6:
<_AndroidNativeLibraryForFastDev Condition=" '$(_InstantRunEnabled)' == 'True' And '$(AndroidUseDebugRuntime)' == 'True' And '$(_AndroidCheckedBuild)' == '' " Include="%(_TargetLibDir.Identity)\libxamarin-debug-app-helper.so" />I'll need to add an equivalent for these.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
libxamarin-debug-app-helper.so is in our runtime packs and will end up as a @(FrameworkNativeLibrary), and those items aren't fast deployed. Before fb727f0, these files were hard coded in the <BuildApk/> task.
It looks like only @(AndroidNativeLibrary) items are added to @(_AndroidNativeLibraryForFastDev): https://github.com/xamarin/monodroid/blob/27736a7ffc48d606ab45598f761e873f8572f46a/tools/msbuild/Xamarin.Android.Common.Debugging.targets#L224.
Maybe we should look at fast deploying @(FrameworkNativeLibrary) in a future PR? If we try to fast deploy both customer libs and framework libs we'll need to rework: https://github.com/xamarin/xamarin-android/blob/aaa37c3df94490ee9befb8381f4dd7aa18226355/src/Xamarin.Android.Build.Tasks/Tasks/BuildApk.cs#L589
As for the three $(_AndroidNativeLibraryForFastDev) items explicitly mentioned here, two files are used for native static code analysis (which I don't think we ever need to put into customer apps). The other one is for the interpreter, and I am not sure if that is available in .NET 6? I am not sure if we need to worry about any of these in a .NET 6 context?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
certain native libs still need to be in the apk even when fast deploying. specifically monodroid.so and its immediate dependencies. This is because it is loaded as part of the runtime startup. So we can't just fast dev everything.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The latest changes shouldn't change the current logic for .NET 6, except the block:
<!-- HACK remove any unexpected runtime components -->
<FrameworkNativeLibrary
Remove="@(ResolvedFileToPublish)"
Condition=" $([System.Text.RegularExpressions.Regex]::IsMatch('%(FullPath)', '.*linux-.*')) "
/>Instead of silently ignoring, you get a XA4301 warning now.
The tests we have for Fast Deployment should be passing, so I don't think there is actually anything broken?
|
I was seeing an issue with SQLite NuGet packages, I think we'll need this addressed before we can finish this: |
266b540 to
d0ff9e8
Compare
| {0} - The file path</comment> | ||
| </data> | ||
| <data name="XA4301_ABI_RID" xml:space="preserve"> | ||
| <value>Cannot determine ABI of native library '{0}'. The RuntimeIdentifier '{1}' does not target Android platforms.</value> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This message isn't actionable; what should the user do to fix this?
For that matter, what's it mean? This reads like two separate bits of information: we can't determine the ABI, a'la "normal" XA4301_ABI, in which case the library needs to be moved into a different directory (?), but what's that have to do with RuntimeIdentifier?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This message isn't actionable; what should the user do to fix this?
I don't think there is much they can do? I could log the name of the NuGet package it came from, and suggest they remove it?
See details here: dotnet/runtime#45927
Basically, if you add a NuGet package that is missing android-* RID for native libraries, the dotnet/sdk will potentially fall back to including linux-* native libraries. I tried to get them to change the behavior, but it is setup this way for other reasons.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looking closer at the item metadata available:
We don't even have %(RuntimeIdentifier), so I changed the message to be:
warning XA4301: Cannot determine ABI of native library 'C:\src\xamarin-android\packages\sqlitepclraw.lib.e_sqlite3.linux\1.1.11\runtimes\linux-arm64\native\libe_sqlite3.so'. Remove the 'SQLitePCLRaw.lib.e_sqlite3.linux' NuGet package, or notify the library author.
Let me know if you have other ideas, thanks.
|
|
||
| foreach (var library in InputLibraries) { | ||
| var abi = MonoAndroidHelper.GetNativeLibraryAbi (library); | ||
| if (string.IsNullOrEmpty (abi)) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
…especially when I look at this: XA4301_ABI_RID won't be emitted if the lib is in a directory named after an ABI we know about, %(RuntimeIdentifier) be damned.
Presumably if %(RuntimeIdentifier) doesn't "match", it's unusable, e.g. trying to use a "desktop" Linux .so on Android is doomed, because libc.so is in the wrong location (among other things). Should we "fail early" and error, and check before calling MonoAndroidHelper.GetNativeLibraryAbi()? Or should we "fail at runtime" and warn -- as is done in this PR -- if the .so can't actually be used?
Which dovetails back with the warning message: if an unsupported %(RuntimeIdentifier) means "this has no chance of working", then the message needs to state that, and the fix is to remove the offending .so file. If it has a chance of working, then XA4301_ABI_RID should mirror XA4301_ABI, and mention that moving the .so into a directory with the ABI name will work.
As-is, the XA4301_ABI_RID warning means nothing.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't believe the %(RuntimeIdentifier) metadata would be present in user projects. It would only be present from NuGet packages.
Do you think the existing checks here needs to be changed?
I think it will already check %(RuntimeIdentifier) before directory names.
d0ff9e8 to
5b575fc
Compare
I've noticed the `_IncludeNativeSystemLibraries` MSBuild target can
take 40ms because it is running this workaround:
<!-- HACK remove any unexpected runtime components -->
<FrameworkNativeLibrary
Remove="@(ResolvedFileToPublish)"
Condition=" $([System.Text.RegularExpressions.Regex]::IsMatch('%(FullPath)', '.*linux-.*')) "
/>
The `_IncludeNativeSystemLibraries` also can be split up between
"legacy" Xamarin.Android and .NET 6. The .NET 6 version can simply
check the `@(ResolvedFileToPublish)` item group, as the other parts
support the intepreter, profiler, etc., which are not available in .NET
6 yet.
I created a new `<ProcessNativeLibraries/>` MSBuild task to simplify
the work in C#:
* We can iterate over all `.so` files from `@(ResolvedFileToPublish)`.
* We can fix up `libmonodroid.so` as needed.
* We can emit a warning if the ABI is unknown without using a `Regex`.
It is possible for Linux `.so` files to be brought in from NuGet
packages as discussed in: dotnet/runtime#45927
The warning is a new alternate message for XA4301, such as:
warning XA4301: Cannot determine ABI of native library 'C:\src\xamarin-android\packages\sqlitepclraw.lib.e_sqlite3.linux\1.1.11\runtimes\linux-arm64\native\libe_sqlite3.so'. Remove the 'SQLitePCLRaw.lib.e_sqlite3.linux' NuGet package, or notify the library author.
As part of this change, .NET 6 will no longer need to do:
<PropertyGroup>
<_Assemblies>@(_ResolvedFrameworkAssemblies)</_Assemblies>
So we can avoid creating a ~14K character string.
~~ Results ~~
Timing the project fromm the `DotNetBuild(android.21-arm,false)` test,
in a build with no changes:
Before:
40 ms _IncludeNativeSystemLibraries 1 calls
After:
4 ms _IncludeNativeSystemLibraries 1 calls
This should save ~36ms from all .NET 6 builds.
5b575fc to
fc0de47
Compare

I've noticed the
_IncludeNativeSystemLibrariesMSBuild target cantake 40ms because it is running this workaround:
The
_IncludeNativeSystemLibrariesalso can be split up between"legacy" Xamarin.Android and .NET 6. The .NET 6 version can simply
check the
@(ResolvedFileToPublish)item group, as the other partssupport the intepreter, profiler, etc., which are not available in .NET
6 yet.
I created a new
<ProcessNativeLibraries/>MSBuild task to simplifythe work in C#:
.sofiles from@(ResolvedFileToPublish).libmonodroid.soas needed.Regex.It is possible for Linux
.sofiles to be brought in from NuGetpackages as discussed in: Use unix as the parent for the Android RID graph runtime#45927
The warning is a new alternate message for XA4301, such as:
As part of this change, .NET 6 will no longer need to do:
So we can avoid creating a ~14K character string.
Results
Timing the project fromm the
DotNetBuild(android.21-arm,false)test,in a build with no changes:
This should save ~36ms from all .NET 6 builds.