Add Lzma library files to the repository by rzikm · Pull Request #124003 · dotnet/runtime

rzikm · 2026-02-04T13:22:41Z

Preparatory PR for #1542

Copilot

Pull request overview

This PR adds the XZ Utils library (version 5.8.2) to the dotnet/runtime repository as preparatory work for issue #1542. The addition includes the complete LZMA compression library source code, build configurations, documentation, and integration scripts.

Changes:

Adds XZ Utils 5.8.2 library files from the tukaani-project/xz repository
Includes CMake and Autotools build system configurations
Adds documentation, examples, and internationalization support files
Configures the library for static linking with multithreading disabled

Reviewed changes

Copilot reviewed 75 out of 497 changed files in this pull request and generated no comments.

File	Description
src/native/external/xz/*	Complete XZ Utils 5.8.2 library source code including build scripts, documentation, examples, and localization files
src/native/external/xz.cmake	CMake configuration to integrate XZ Utils into dotnet/runtime build system with specific feature flags
src/native/external/xz.version	Version tracking file indicating XZ Utils v5.8.2 from tukaani-project
src/native/external/cgmanifest.json	Component governance manifest entry for XZ Utils dependency

dotnet-policy-service · 2026-02-04T13:25:43Z

Tagging subscribers to this area: @karelz, @dotnet/area-system-io-compression
See info in area-owners.md if you want to be subscribed.

src/native/libs/System.IO.Compression.Native/CMakeLists.txt

Copilot

Pull request overview

Copilot reviewed 79 out of 284 changed files in this pull request and generated 7 comments.

src/native/external/xz/src/common/tuklib_open_stdxxx.c

src/native/external/xz/src/common/tuklib_open_stdxxx.h

src/native/external/xz/cmake/tuklib_physmem.cmake

src/native/external/xz/cmake/tuklib_cpucores.cmake

src/native/external/xz.version

src/native/external/xz/src/common/tuklib_gettext.h

src/native/external/xz.cmake

rzikm · 2026-02-09T12:21:55Z

I am getting compile errors on linux:

  ERROR: /source/dotnet/runtime/src/native/libs/System.IO.Compression.Native/entrypoints.c file did not match entries exported from /source/dotnet/runtime/artifacts/obj/native/net11.0-linux-Debug-x64/System.IO.Compression.Native/libSystem.IO.Compression.Native.so
  DIFFERENCES FOUND: 
  lzma_stream_decoder

Seems like there is something wrong with the exports specification:

System.IO.Compression.Native$ nm libSystem.IO.Compression.Native.so | grep " lzma_stream_decoder"
00000000003c51c0 t lzma_stream_decoder
00000000003c4410 t lzma_stream_decoder_init

All the other exported entries have T in the nm output, e.g.

000000000019dd90 T BrotliDecoderDecompress

But the exports file seems to be generated correctly:

        ZSTD_CCtx_setPledgedSrcSize;
        ZDICT_trainFromBuffer;
        lzma_stream_decoder;
    local: *;
};

I am not sure what is going on here, maybe there is a problem because the library functions are declared as extern?

runtime/src/native/external/xz/src/liblzma/api/lzma/container.h

Lines 838 to 840 in d8731e0

    
           extern LZMA_API(lzma_ret) lzma_alone_decoder( 
        
           		lzma_stream *strm, uint64_t memlimit) 
        
           		lzma_nothrow lzma_attr_warn_unused_result;

I will need to investigate a bit more

Copilot

Pull request overview

Copilot reviewed 79 out of 284 changed files in this pull request and generated 2 comments.

src/native/external/xz/src/common/tuklib_open_stdxxx.h

src/native/external/xz.version

Co-authored-by: Copilot <[email protected]>

rzikm · 2026-02-10T17:38:54Z

I will need to investigate a bit more

Should be working now

Copilot

Pull request overview

Copilot reviewed 79 out of 284 changed files in this pull request and generated 3 comments.

src/native/external/xz.cmake

src/native/external/xz/src/common/w32_application.manifest.comments.txt

src/native/external/xz/src/common/tuklib_gettext.h

Co-authored-by: Copilot <[email protected]>

Copilot

Pull request overview

Copilot reviewed 79 out of 284 changed files in this pull request and generated no new comments.

Copilot

Pull request overview

Copilot reviewed 79 out of 230 changed files in this pull request and generated no new comments.

ericstj · 2026-02-19T19:14:38Z

What sort of size increase do we see to our system.io.compression.native with this, across different platforms? Have you confirmed we are linking correctly and getting the minimal set of things we need?

rzikm · 2026-02-20T09:11:58Z

What sort of size increase do we see to our system.io.compression.native with this, across different platforms? Have you confirmed we are linking correctly and getting the minimal set of things we need?

the PR as-is increases the native .so by about 60 kB in release on my desktop, there is a reference to lzma_stream_decoder_init in the entryponts.c to validate if it links properly. I don't think we can whittle down any more of the compiled source without manual modifications to the build scripts.

jkotas · 2026-02-20T16:36:56Z

src/native/external/xz/src/scripts/xzgrep.in

@@ -0,0 +1,302 @@
+#!@POSIX_SHELL@
+# SPDX-License-Identifier: GPL-2.0-or-later


Do we really need these GPL licensed sources?

Unfortunately, the configure-time step fails without them even though they are not necessary for the liblzma target build, we could get rid of them after some CMakeLists.txt modifications.

Would it be a small change that can be up streamed?

I expect that this will get flagged by license scanning tools like https://github.com/dotnet/dotnet/blob/main/docs/license-scanning.md . If you get rid of all GPL sources, it should make it easier to deal with the license scanning tools.

Done: tukaani-project/xz#210

jkotas · 2026-02-20T16:51:43Z

the PR as-is increases the native .so by about 60 kB in release on my desktop, there is a reference to lzma_stream_decoder_init in the entryponts.c to validate if it links properly

Does lzma_stream_decoder_init bring the full compression / decompression implementation into the binary?

FWIW, I do not see a problem with the extra code that this brings. We add several megabytes of code to netcoreapp every release. The problem I see is that the overall compression design is not trimming friendly. The moment you enable compression for your HttpClient, you get all compression algorithms in our portfolio in your (AOT compiled) binary - discussed at #123531 (comment) .

rzikm · 2026-02-21T17:02:07Z

Does lzma_stream_decoder_init bring the full compression / decompression implementation into the binary?

I am not 100% sure. I just assumed that the other code does not get eliminated and 60 kB seems reasonable compared to about 400 kB added by zstd when looking at the included code size

FWIW, I do not see a problem with the extra code that this brings. We add several megabytes of code to netcoreapp every release. The problem I see is that the overall compression design is not trimming friendly. The moment you enable compression for your HttpClient, you get all compression algorithms in our portfolio in your (AOT compiled) binary - discussed at #123531 (comment) .

Can you see something that we could do about the design to improve the trimming situation? I don't have enough knowledge in that area

jkotas · 2026-02-22T01:06:04Z

I just assumed that the other code does not get eliminated

Linkers for native code trim unreachable code by default. If the actual compression / decompression implementation is unreachable code, it is likely stripped by the linker in the final binary.

Can you see something that we could do about the design to improve the trimming situation?

We have solved this problem for encodings in globalization by having an API that allows registering encoding implementation https://learn.microsoft.com/dotnet/api/system.text.encoding.registerprovider?view=net-10.0#system-text-encoding-registerprovider(system-text-encodingprovider) . We can consider something similar for compression algorithms.

Copilot

Pull request overview

Copilot reviewed 82 out of 227 changed files in this pull request and generated 1 comment.

src/native/external/xz/src/liblzma/check/crc32_fast.c

HAVE_SYS_PARAM_H was misspelled HAVE_PARAM_H. This might have broken the build on systems where sysctl() is used to detect the amount of RAM or the number of processor cores/threads. The cpuset code for FreeBSD doesn't need the macro it so it was removed. Fixes: dotnet/runtime#124003 (comment) Fixes: dotnet/runtime#124003 (comment) Fixes: 7e3493d ("Build: Add very limited experimental CMake support.")

It made no difference in this package, but it's good to fix it still. Fixes: dotnet/runtime#124003 (comment)

Fixes: dotnet/runtime#124003 (comment) Fixes: dotnet/runtime#124003 (comment) Fixes: dotnet/runtime#124003 (comment)

HAVE_SYS_PARAM_H was misspelled HAVE_PARAM_H. This might have broken the build on systems where sysctl() is used to detect the amount of RAM or the number of processor cores/threads. The cpuset code for FreeBSD doesn't need the macro it so it was removed. Fixes: dotnet/runtime#124003 (comment) Fixes: dotnet/runtime#124003 (comment) Fixes: 7e3493d ("Build: Add very limited experimental CMake support.")

It made no difference in this package, but it's good to fix it still. Fixes: dotnet/runtime#124003 (comment)

Fixes: dotnet/runtime#124003 (comment) Fixes: dotnet/runtime#124003 (comment) Fixes: dotnet/runtime#124003 (comment)

Fixes: dotnet/runtime#124003 (comment) Fixes: 96b663f ("liblzma: Refactor CRC comments.")

HAVE_SYS_PARAM_H was misspelled HAVE_PARAM_H. This might have broken the build on systems where sysctl() is used to detect the amount of RAM or the number of processor cores/threads. The cpuset code for FreeBSD doesn't need the macro it so it was removed. Fixes: dotnet/runtime#124003 (comment) Fixes: dotnet/runtime#124003 (comment) Fixes: 7e3493d ("Build: Add very limited experimental CMake support.")

It doesn't matter in this package, but it's good to fix it still. Fixes: dotnet/runtime#124003 (comment)

Fixes: dotnet/runtime#124003 (comment) Fixes: dotnet/runtime#124003 (comment) Fixes: dotnet/runtime#124003 (comment)

Fixes: dotnet/runtime#124003 (comment) Fixes: 96b663f ("liblzma: Refactor CRC comments.")

rzikm added 2 commits January 23, 2026 13:31

Include xz library in the repo

d8731e0

Link lzma to System.IO.Compression.Native

2b5700f

Copilot AI review requested due to automatic review settings February 4, 2026 13:22

rzikm added the area-System.IO.Compression label Feb 4, 2026

Copilot AI reviewed Feb 4, 2026

View reviewed changes

rzikm marked this pull request as draft February 4, 2026 13:23

dotnet-policy-service bot assigned rzikm Feb 4, 2026

build-analysis bot mentioned this pull request Feb 4, 2026

[android][arm64] System.Net.Sockets.Tests.SendTo_SyncForceNonBlocking.Datagram_UDP_ShouldImplicitlyBindLocalEndpoint fails with NetworkUnreachable #120526

Open

iremyux reviewed Feb 6, 2026

View reviewed changes

src/native/libs/System.IO.Compression.Native/CMakeLists.txt Outdated Show resolved Hide resolved

rzikm added 4 commits February 9, 2026 10:42

Remove unnecessary xz repo files

6b2729e

Don't copy unnecessary files during install

9e22f20

Attempt to fix build on linux

b577d95

Fix newlines at the end of file

829ad77

Copilot AI review requested due to automatic review settings February 9, 2026 12:14

Copilot AI reviewed Feb 9, 2026

View reviewed changes

rzikm added 2 commits February 10, 2026 18:06

Fix build on Linux

daf697d

Remove libgnu workaround

4a58132

Copilot AI review requested due to automatic review settings February 10, 2026 17:24

Copilot AI reviewed Feb 10, 2026

View reviewed changes

src/native/external/xz/src/common/tuklib_open_stdxxx.h Show resolved Hide resolved

src/native/external/xz.version Outdated Show resolved Hide resolved

Update src/native/external/xz.version

29847da

Co-authored-by: Copilot <[email protected]>

Copilot AI review requested due to automatic review settings February 10, 2026 17:38

Copilot AI reviewed Feb 10, 2026

View reviewed changes

src/native/external/xz.cmake Outdated Show resolved Hide resolved

src/native/external/xz/src/common/w32_application.manifest.comments.txt Show resolved Hide resolved

src/native/external/xz/src/common/tuklib_gettext.h Show resolved Hide resolved

Update src/native/external/xz.cmake

5abc214

Co-authored-by: Copilot <[email protected]>

Copilot AI review requested due to automatic review settings February 10, 2026 17:49

Copilot AI reviewed Feb 10, 2026

View reviewed changes

This was referenced Feb 10, 2026

"We stopped hearing from agent Azure Pipelines 32. Verify the agent machine is running and has a healthy network connection" dotnet/dnceng#1886

Open

Test failure: baseservices/exceptions/stackoverflow/stackoverflowtester/stackoverflowtester.cmd #110173

Open

Copilot AI review requested due to automatic review settings February 19, 2026 18:16

Copilot AI reviewed Feb 19, 2026

View reviewed changes

build-analysis bot mentioned this pull request Feb 19, 2026

[mono] The type initializer for System.Drawing.DrawingCom threw an exception: Operation is not supported on this platform #110150

Open

Restore scripts folder

8aa31aa

jkotas reviewed Feb 20, 2026

View reviewed changes

rzikm added 2 commits February 23, 2026 17:46

Remove GPL-licensed files

402ae6c

More local patches

e948d00

Copilot AI review requested due to automatic review settings February 23, 2026 16:54

Copilot AI reviewed Feb 23, 2026

View reviewed changes

src/native/external/xz/src/liblzma/check/crc32_fast.c Show resolved Hide resolved

build-analysis bot mentioned this pull request Feb 23, 2026

Wasm build terminated with exit code 137 #108299

Open

rzikm requested a review from a team February 24, 2026 15:53

iremyux approved these changes Feb 24, 2026

View reviewed changes

Larhzu added a commit to tukaani-project/xz that referenced this pull request Feb 24, 2026

tuklib_open_stdxxx: Fix a typo in macro name

e546f0d

It made no difference in this package, but it's good to fix it still. Fixes: dotnet/runtime#124003 (comment)

Larhzu added a commit to tukaani-project/xz that referenced this pull request Feb 24, 2026

Fix typos and grammar in comments and docs

93ea4aa

Fixes: dotnet/runtime#124003 (comment) Fixes: dotnet/runtime#124003 (comment) Fixes: dotnet/runtime#124003 (comment)

Larhzu mentioned this pull request Feb 24, 2026

CMake: Guard script configure_file() calls with ENABLE_SCRIPTS tukaani-project/xz#210

Closed

Larhzu added a commit to tukaani-project/xz that referenced this pull request Feb 24, 2026

tuklib_open_stdxxx: Fix a typo in macro name

0510b17

It made no difference in this package, but it's good to fix it still. Fixes: dotnet/runtime#124003 (comment)

Larhzu added a commit to tukaani-project/xz that referenced this pull request Feb 24, 2026

Fix typos and grammar in comments and docs

dee4a43

Fixes: dotnet/runtime#124003 (comment) Fixes: dotnet/runtime#124003 (comment) Fixes: dotnet/runtime#124003 (comment)

Larhzu added a commit to tukaani-project/xz that referenced this pull request Feb 24, 2026

liblzma: Fix a copy-paste error in commented-out code

fc3ffbc

Fixes: dotnet/runtime#124003 (comment) Fixes: 96b663f ("liblzma: Refactor CRC comments.")

Larhzu added a commit to tukaani-project/xz that referenced this pull request Feb 25, 2026

tuklib_open_stdxxx: Fix a typo in macro name

cd8a623

It doesn't matter in this package, but it's good to fix it still. Fixes: dotnet/runtime#124003 (comment)

Larhzu added a commit to tukaani-project/xz that referenced this pull request Feb 25, 2026

Fix typos and grammar in comments and docs

4e76651

Fixes: dotnet/runtime#124003 (comment) Fixes: dotnet/runtime#124003 (comment) Fixes: dotnet/runtime#124003 (comment)

Larhzu added a commit to tukaani-project/xz that referenced this pull request Feb 25, 2026

liblzma: Fix a copy-paste error in commented-out code

5553509

Fixes: dotnet/runtime#124003 (comment) Fixes: 96b663f ("liblzma: Refactor CRC comments.")

iremyux mentioned this pull request Feb 25, 2026

Add support for 7z/LZMA1/LZMA2 to System.IO.Compression #1542

Open

		@@ -0,0 +1,302 @@
		#!@POSIX_SHELL@
		# SPDX-License-Identifier: GPL-2.0-or-later

Conversation

rzikm commented Feb 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

dotnet-policy-service bot commented Feb 4, 2026

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rzikm commented Feb 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

rzikm commented Feb 10, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

ericstj commented Feb 19, 2026

Uh oh!

rzikm commented Feb 20, 2026

Uh oh!

jkotas Feb 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rzikm Feb 23, 2026

Choose a reason for hiding this comment

Uh oh!

jkotas Feb 23, 2026

Choose a reason for hiding this comment

Uh oh!

rzikm Feb 23, 2026

Choose a reason for hiding this comment

Uh oh!

jkotas commented Feb 20, 2026

Uh oh!

rzikm commented Feb 21, 2026

Uh oh!

jkotas commented Feb 22, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

rzikm commented Feb 4, 2026 •

edited

Loading

rzikm commented Feb 9, 2026 •

edited

Loading

jkotas Feb 20, 2026 •

edited

Loading