RFC: Cleanup xdiff and begin its rustification #1980

ezekielnewren · 2025-05-30T00:09:34Z

In order to facilitate the quick adoption of am/xdiff-hash-tweak, this round drops the changes to hashing in xdiff and instead modifies another part of xdiff.

A high level overview of v3:

patch 1: add a policy for using Rust (brian's patch, with a small tweak)
patch 2: introduce Rust to the codebase
patches 3-5: adapt CI (github workflows) to build Git with Rust
patch 6: introduce the ivec type
patches 7-14: xdiff code cleanup in preparation for translating to Rust
patch 15: translate a C function into Rust and call it from C

I'm particularly interested in what folks think of the new ivec type for sharing data across the language barrier. Thoughts?

Build results for these changes: https://github.com/git/git/actions/runs/17170212383

Links to older versions, which focused on hashing in xdiff:

cc: Elijah Newren [email protected]
cc: "brian m. carlson" [email protected]
cc: Taylor Blau [email protected]
cc: Christian Brabandt [email protected]
cc: Phillip Wood [email protected]
cc: Eli Schwartz [email protected]
cc: "Haelwenn (lanodan) Monnier" [email protected]
cc: Johannes Schindelin [email protected]
cc: Matthias Aßhauer [email protected]
cc: Patrick Steinhardt [email protected]
cc: Sam James [email protected]
cc: Collin Funk [email protected]
cc: Mike Hommey [email protected]
cc: Pierre-Emmanuel Patry [email protected]
cc: Ben Knoble [email protected]
cc: Ramsay Jones [email protected]
cc: "Kristoffer Haugsbakk" [email protected]
cc: [email protected]
cc: Josh Steadmon [email protected]
cc: Jeff King [email protected]
cc: Yee Cheng Chin [email protected]

gitgitgadget-git · 2025-05-30T00:11:02Z

Welcome to GitGitGadget

Hi @ezekielnewren, and welcome to GitGitGadget, the GitHub App to send patch series to the Git mailing list from GitHub Pull Requests.

Please make sure that either:

Your Pull Request has a good description, if it consists of multiple commits, as it will be used as cover letter.
Your Pull Request description is empty, if it consists of a single commit, as the commit message should be descriptive enough by itself.

You can CC potential reviewers by adding a footer to the PR description with the following syntax:

CC: Revi Ewer <[email protected]>, Ill Takalook <[email protected]>

NOTE: DO NOT copy/paste your CC list from a previous GGG PR's description,
because it will result in a malformed CC list on the mailing list. See
example.

Also, it is a good idea to review the commit messages one last time, as the Git project expects them in a quite specific form:

the lines should not exceed 76 columns,
the first line should be like a header and typically start with a prefix like "tests:" or "revisions:" to state which subsystem the change is about, and
the commit messages' body should be describing the "why?" of the change.
Finally, the commit messages should end in a Signed-off-by: line matching the commits' author.

It is in general a good idea to await the automated test ("Checks") in this Pull Request before contributing the patches, e.g. to avoid trivial issues such as unportable code.

Contributing the patches

Before you can contribute the patches, your GitHub username needs to be added to the list of permitted users. Any already-permitted user can do that, by adding a comment to your PR of the form /allow. A good way to find other contributors is to locate recent pull requests where someone has been /allowed:

Search: is:pr is:open "/allow"

Both the person who commented /allow and the PR author are able to /allow you.

An alternative is the channel #git-devel on the Libera Chat IRC network:

<newcontributor> I've just created my first PR, could someone please /allow me? https://github.com/gitgitgadget/git/pull/12345
<veteran> newcontributor: it is done
<newcontributor> thanks!

Once on the list of permitted usernames, you can contribute the patches to the Git mailing list by adding a PR comment /submit.

If you want to see what email(s) would be sent for a /submit request, add a PR comment /preview to have the email(s) sent to you. You must have a public GitHub email address for this. Note that any reviewers CC'd via the list in the PR description will not actually be sent emails.

After you submit, GitGitGadget will respond with another comment that contains the link to the cover letter mail in the Git mailing list archive. Please make sure to monitor the discussion in that thread and to address comments and suggestions (while the comments and suggestions will be mirrored into the PR by GitGitGadget, you will still want to reply via mail).

If you do not want to subscribe to the Git mailing list just to be able to respond to a mail, you can download the mbox from the Git mailing list archive (click the (raw) link), then import it into your mail program. If you use GMail, you can do this via:

curl -g --user "<EMailAddress>:<Password>" \
    --url "imaps://imap.gmail.com/INBOX" -T /path/to/raw.txt

To iterate on your change, i.e. send a revised patch or patch series, you will first want to (force-)push to the same branch. You probably also want to modify your Pull Request description (or title). It is a good idea to summarize the revision by adding something like this to the cover letter (read: by editing the first comment on the PR, i.e. the PR description):

Changes since v1:
- Fixed a typo in the commit message (found by ...)
- Added a code comment to ... as suggested by ...
...

To send a new iteration, just add another PR comment with the contents: /submit.

Need help?

New contributors who want advice are encouraged to join [email protected], where volunteers who regularly contribute to Git are willing to answer newbie questions, give advice, or otherwise provide mentoring to interested contributors. You must join in order to post or view messages, but anyone can join.

You may also be able to find help in real time in the developer IRC channel, #git-devel on Libera Chat. Remember that IRC does not support offline messaging, so if you send someone a private message and log out, they cannot respond to you. The scrollback of #git-devel is archived, though.

gitgitgadget-git · 2025-05-30T00:11:03Z

There are issues in commit 2178ac1:
xdiff: conditionally use Rust's implementation of xxhash
Lines in the body of the commit messages should be wrapped between 60 and 76 characters.
Indented lines, and lines without whitespace, are exempt

gitgitgadget-git · 2025-05-30T20:57:42Z

There are issues in commit 2178ac1:
xdiff: conditionally use Rust's implementation of xxhash
Lines in the body of the commit messages should be wrapped between 60 and 76 characters.
Indented lines, and lines without whitespace, are exempt

gitgitgadget-git · 2025-05-30T20:57:43Z

There are issues in commit 67f6c92:
Random stab in the dark with Rust building in ci
Commit checks stopped - the message is too short
Commit not signed off

gitgitgadget-git · 2025-05-30T21:14:16Z

There are issues in commit d647348:
xdiff: conditionally use Rust's implementation of xxhash
Lines in the body of the commit messages should be wrapped between 60 and 76 characters.
Indented lines, and lines without whitespace, are exempt

gitgitgadget-git · 2025-05-30T22:08:28Z

There are issues in commit d647348:
xdiff: conditionally use Rust's implementation of xxhash
Lines in the body of the commit messages should be wrapped between 60 and 76 characters.
Indented lines, and lines without whitespace, are exempt

gitgitgadget-git · 2025-05-30T23:40:20Z

There are issues in commit d647348:
xdiff: conditionally use Rust's implementation of xxhash
Lines in the body of the commit messages should be wrapped between 60 and 76 characters.
Indented lines, and lines without whitespace, are exempt

gitgitgadget-git · 2025-05-31T01:23:04Z

There are issues in commit 9c59e82:
xdiff: conditionally use Rust's implementation of xxhash
Lines in the body of the commit messages should be wrapped between 60 and 76 characters.
Indented lines, and lines without whitespace, are exempt

dscho · 2025-06-01T15:50:47Z

/allow

gitgitgadget-git · 2025-06-01T15:51:45Z

User ezekielnewren is now allowed to use GitGitGadget.

WARNING: ezekielnewren has no public email address set on GitHub; GitGitGadget needs an email address to Cc: you on your contribution, so that you receive any feedback on the Git mailing list. Go to https://github.com/settings/profile to make your preferred email public to let GitGitGadget know which email address to use.

gitgitgadget-git · 2025-06-03T21:06:54Z

There are issues in commit 9c59e82:
xdiff: conditionally use Rust's implementation of xxhash
Lines in the body of the commit messages should be wrapped between 60 and 76 characters.
Indented lines, and lines without whitespace, are exempt

gitgitgadget-git · 2025-06-03T21:06:55Z

There are issues in commit 10a0eb8:
meson.build: make rust the first dependency for the libgit target
Commit checks stopped - the message is too short
Commit not signed off

gitgitgadget-git · 2025-06-03T21:29:27Z

There are issues in commit 9c59e82:
xdiff: conditionally use Rust's implementation of xxhash
Lines in the body of the commit messages should be wrapped between 60 and 76 characters.
Indented lines, and lines without whitespace, are exempt

gitgitgadget-git · 2025-06-03T21:29:28Z

There are issues in commit 10a0eb8:
meson.build: make rust the first dependency for the libgit target
Commit checks stopped - the message is too short
Commit not signed off

gitgitgadget-git · 2025-06-03T21:29:28Z

There are issues in commit 8116ad9:
meson.build: link libdl
Commit checks stopped - the message is too short
Commit not signed off

gitgitgadget-git · 2025-06-03T22:03:35Z

There are issues in commit 9c59e82:
xdiff: conditionally use Rust's implementation of xxhash
Lines in the body of the commit messages should be wrapped between 60 and 76 characters.
Indented lines, and lines without whitespace, are exempt

gitgitgadget-git · 2025-06-03T22:03:35Z

There are issues in commit 10a0eb8:
meson.build: make rust the first dependency for the libgit target
Commit checks stopped - the message is too short
Commit not signed off

gitgitgadget-git · 2025-06-03T22:03:36Z

There are issues in commit 8116ad9:
meson.build: link libdl
Commit checks stopped - the message is too short
Commit not signed off

gitgitgadget-git · 2025-06-03T22:03:36Z

There are issues in commit c28361b:
.github/workflows/main.yml: use github actions to setup rust
Commit checks stopped - the message is too short
Commit not signed off

gitgitgadget-git · 2025-06-05T18:19:35Z

There are issues in commit 9c59e82:
xdiff: conditionally use Rust's implementation of xxhash
Lines in the body of the commit messages should be wrapped between 60 and 76 characters.
Indented lines, and lines without whitespace, are exempt

gitgitgadget-git · 2025-06-05T18:19:36Z

There are issues in commit 10a0eb8:
meson.build: make rust the first dependency for the libgit target
Commit checks stopped - the message is too short
Commit not signed off

gitgitgadget-git · 2025-06-05T18:19:36Z

There are issues in commit 8116ad9:
meson.build: link libdl
Commit checks stopped - the message is too short
Commit not signed off

gitgitgadget-git · 2025-06-05T18:19:37Z

There are issues in commit c28361b:
.github/workflows/main.yml: use github actions to setup rust
Commit checks stopped - the message is too short
Commit not signed off

gitgitgadget-git · 2025-06-05T18:19:37Z

There are issues in commit 564a178:
Revert "meson.build: link libdl"
Commit not signed off

Make each ci workflow upload its Cargo.lock file as a build artifact so that we can audit build dependencies. Signed-off-by: Ezekiel Newren <[email protected]>

Trying to use Rust's Vec in C, or git's ALLOC_GROW() macros (via wrapper functions) in Rust is painful because: * C doing vector things the Rust way would require wrapper functions, and Rust doing vector things the C way would require wrapper functions, so ivec was created to ensure a consistent contract between the 2 languages for how to manipulate a vector. * Currently, Rust defines its own 'Vec' type that is generic, but its memory allocator and struct layout weren't designed for interoperability with C (or any language for that matter), meaning that the C side cannot push to or expand a 'Vec' without defining wrapper functions in Rust that C can call. Without special care, the two languages might use different allocators (malloc/free on the C side, and possibly something else in Rust), which would make it difficult for a function in one language to free elements allocated by a call from a function in the other language. * Similarly, git defines ALLOC_GROW() and related macros in git-compat-util.h. While we could add functions allowing Rust to invoke something similar to those macros, passing three variables (pointer, length, allocated_size) instead of a single variable (vector) across the language boundary requires more cognitive overhead for readers to keep track of and makes it easier to make mistakes. Further, for low-level components that we want to eventually convert to pure Rust, such triplets would feel very out of place. To address these issue, introduce a new type, ivec -- short for interoperable vector. (We refer to it as 'ivec' generally, though on the Rust side the struct is called IVec to match Rust style.) This new type is specifically designed for FFI purposes, so that both languages handle the vector in the same way, though it could be used on either side independently. This type is designed such that it can easily be replaced by a standard Rust 'Vec' once interoperability is no longer a concern. One particular item to note is that Git's macros to handle vec operations infer the amount that a vec needs to grow from the size of a pointer, but that makes it somewhat specific to the macros used in C. To avoid defining every ivec function as a macro I opted to also include an element_size field that allows concrete functions like push() to know how much to grow the memory. This element_size also helps in verifying that the ivec is correct when passing from C to Rust. Signed-off-by: Ezekiel Newren <[email protected]>

Move xdl_prepare_env() later in the file to avoid the need for forward declarations. Signed-off-by: Ezekiel Newren <[email protected]>

xrecord_t.next, xdfile_t.hbits, xdfile_t.rhash are initialized, but never used for anything by the code. Remove them. Signed-off-by: Ezekiel Newren <[email protected]>

A few commits ago, we added definitions for Rust primitive types, to facilitate interoperability between C and Rust. Switch a few variables to use these types. Which, for now, will require adding some casts. Also change xdlclass_t::ha to be u64 to match xrecord_t::ha, as pointed out by Johannes. Helped-by: Johannes Schindelin <[email protected]> Signed-off-by: Ezekiel Newren <[email protected]>

Simplify xdl_prepare_ctx() by using xdl_free_ctx() instead of using local variables with hand rolled memory management. Signed-off-by: Ezekiel Newren <[email protected]>

xdfile_t currently uses a chastore which functions as a memory pool and a vector which maps to the alocations created by the chastore. It seems like xrecord_t used to be a linked list until the recs and nrec fields were added. I think that xrecord_t.next was meant to be removed, but was overlooked. This dual data structure setup make the code somewhat confusing. Additionally the C type chastore_t isn't FFI friendly. While it could be implemented in Rust, since the data structure is confusing anyway, replace it with an ivec whose purpose is to be interoperable. This makes the fields nrec and recs in xdfile_t redundant, which will be removed in the next 2 commits. Signed-off-by: Ezekiel Newren <[email protected]>

Because of the data structure cleanup in the previous commit, the nrec field is no longer necessary. Use record.length in place of nrec. This commit is best viewed with --color-words. Signed-off-by: Ezekiel Newren <[email protected]>

Because of the change from chastore to ivec a few commits ago, recs now points to record's elements in a 1:1 mapping. Since both recs and record are vectors, this additional mapping is superfluous. Remove recs. This commit is best viewed with --color-words. Signed-off-by: Ezekiel Newren <[email protected]>

Convert the remaining ambiguous fields in xdfile_t from C types to Rust types for interoperability. Signed-off-by: Ezekiel Newren <[email protected]>

Replace the C implementation of xdl_trim_ends() with a Rust implementation. Signed-off-by: Ezekiel Newren <[email protected]>

ezekielnewren · 2025-08-23T03:55:07Z

/submit

gitgitgadget-git · 2025-08-23T03:56:47Z

Submitted as [email protected]

To fetch this version into FETCH_HEAD:

git fetch https://github.com/gitgitgadget/git/ pr-git-1980/ezekielnewren/xdiff_rust_speedup-v3

To fetch this version to local tag pr-git-1980/ezekielnewren/xdiff_rust_speedup-v3:

git fetch --no-tags https://github.com/gitgitgadget/git/ tag pr-git-1980/ezekielnewren/xdiff_rust_speedup-v3

gitgitgadget-git · 2025-08-23T08:18:13Z

Makefile


 # Guard against environment variables
 BUILTIN_OBJS =
 BUILT_INS =


On the Git mailing list, "Kristoffer Haugsbakk" wrote (reply to this):

On Sat, Aug 23, 2025, at 05:55, Ezekiel Newren via GitGitGadget wrote: > From: Ezekiel Newren <[email protected]> > > Trying to use Rust's Vec in C, or git's ALLOC_GROW() macros (via > wrapper functions) in Rust is painful because: nit: s/git's/Git's/ > [snip] > diff --git a/rust/interop/src/ivec.rs b/rust/interop/src/ivec.rs > [snip] > + // assert_eq!(vec.capacity, vec.slice.len()); Why are there three commented-out assertions? (all capacity/length) > + assert_eq!(expected, vec.length); > + assert!(vec.capacity >= expected); > + for i in 0..vec.length { > + assert_eq!(default_value, vec[i]); > + } > [snip] -- Kristoffer Haugsbakk

On the Git mailing list, Ezekiel Newren wrote (reply to this):

On Sat, Aug 23, 2025 at 2:13 AM Kristoffer Haugsbakk <[email protected]> wrote: > > On Sat, Aug 23, 2025, at 05:55, Ezekiel Newren via GitGitGadget wrote: > > From: Ezekiel Newren <[email protected]> > > > > Trying to use Rust's Vec in C, or git's ALLOC_GROW() macros (via > > wrapper functions) in Rust is painful because: > > nit: s/git's/Git's/ > > > [snip] > > diff --git a/rust/interop/src/ivec.rs b/rust/interop/src/ivec.rs > > [snip] > > + // assert_eq!(vec.capacity, vec.slice.len()); > > Why are there three commented-out assertions? (all capacity/length) > > > + assert_eq!(expected, vec.length); > > + assert!(vec.capacity >= expected); > > + for i in 0..vec.length { > > + assert_eq!(default_value, vec[i]); > > + } > > [snip] > > -- > Kristoffer Haugsbakk Good catch, I should have removed those commented out lines. Looking back through the code I also missed calling std::ptr::drop_in_place() if the IVec shrinks. I'll apply those changes in the next version.

gitgitgadget-git · 2025-08-23T08:18:16Z

User "Kristoffer Haugsbakk" <[email protected]> has been added to the cc: list.

gitgitgadget-git · 2025-08-23T13:47:56Z

.gitignore

 /git.VC.VC.opendb
 /git.VC.db
 *.dSYM
 /contrib/buildsystems/out


On the Git mailing list, wrote (reply to this):

On August 22, 2025 11:56 PM, Ezekiel Newren wrote: >From: Ezekiel Newren <[email protected]> > >Upcoming patches will simplify xdiff, while also porting parts of it to Rust. In >preparation, add some stubs and setup the Rust build. For now, it is easier to let >cargo build rust and have make or meson merely link against the static library that >cargo builds. In line with ongoing libification efforts, use multiple crates to allow >more modularity on the Rust side. xdiff is the crate that this series will focus on, but >we also introduce the interop crate for future patch series. > >In order to facilitate interoperability between C and Rust, introduce C definitions for >Rust primitive types in git-compat-util.h. > >Signed-off-by: Ezekiel Newren <[email protected]> >--- > .gitignore | 3 +++ > Makefile | 53 ++++++++++++++++++++++++++++---------- > build_rust.sh | 57 +++++++++++++++++++++++++++++++++++++++++ > git-compat-util.h | 17 ++++++++++++ > meson.build | 52 +++++++++++++++++++++++++++++++------ > rust/Cargo.toml | 6 +++++ > rust/interop/Cargo.toml | 14 ++++++++++ rust/interop/src/lib.rs | 0 > rust/xdiff/Cargo.toml | 15 +++++++++++ > rust/xdiff/src/lib.rs | 0 > 10 files changed, 196 insertions(+), 21 deletions(-) create mode 100755 >build_rust.sh create mode 100644 rust/Cargo.toml create mode 100644 >rust/interop/Cargo.toml create mode 100644 rust/interop/src/lib.rs create mode >100644 rust/xdiff/Cargo.toml create mode 100644 rust/xdiff/src/lib.rs > >diff --git a/.gitignore b/.gitignore >index 04c444404e4b..ff81e3580c4e 100644 >--- a/.gitignore >+++ b/.gitignore >@@ -254,3 +254,6 @@ Release/ > /contrib/buildsystems/out > /contrib/libgit-rs/target > /contrib/libgit-sys/target >+/.idea/ >+/rust/target/ >+/rust/Cargo.lock >diff --git a/Makefile b/Makefile >index 70d1543b6b86..1ec0c1ee6603 100644 >--- a/Makefile >+++ b/Makefile >@@ -919,6 +919,29 @@ TEST_SHELL_PATH = $(SHELL_PATH) > > LIB_FILE = libgit.a > XDIFF_LIB = xdiff/lib.a >+ >+EXTLIBS = >+ >+ifeq ($(DEBUG), 1) >+ RUST_BUILD_MODE = debug >+else >+ RUST_BUILD_MODE = release >+endif >+ >+RUST_TARGET_DIR = rust/target/$(RUST_BUILD_MODE) RUST_FLAGS_FOR_C = >+-L$(RUST_TARGET_DIR) >+ >+.PHONY: compile_rust >+compile_rust: >+ ./build_rust.sh . $(RUST_BUILD_MODE) xdiff >+ >+EXTLIBS += ./$(RUST_TARGET_DIR)/libxdiff.a >+ >+UNAME_S := $(shell uname -s) >+ifeq ($(UNAME_S),Linux) >+ EXTLIBS += -ldl >+endif >+ > REFTABLE_LIB = reftable/libreftable.a > > GENERATED_H += command-list.h >@@ -1390,7 +1413,7 @@ UNIT_TEST_OBJS += $(UNIT_TEST_DIR)/lib-reftable.o > > # xdiff and reftable libs may in turn depend on what is in libgit.a GITLIBS = >common-main.o $(LIB_FILE) $(XDIFF_LIB) $(REFTABLE_LIB) $(LIB_FILE) -EXTLIBS = >+ > > GIT_USER_AGENT = git/$(GIT_VERSION) > >@@ -2541,7 +2564,7 @@ git.sp git.s git.o: EXTRA_CPPFLAGS = \ > '-DGIT_MAN_PATH="$(mandir_relative_SQ)"' \ > '-DGIT_INFO_PATH="$(infodir_relative_SQ)"' > >-git$X: git.o GIT-LDFLAGS $(BUILTIN_OBJS) $(GITLIBS) >+git$X: git.o GIT-LDFLAGS $(BUILTIN_OBJS) $(GITLIBS) compile_rust > $(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) \ > $(filter %.o,$^) $(LIBS) > >@@ -2891,17 +2914,17 @@ headless-git.o: compat/win32/headless.c GIT- >CFLAGS > headless-git$X: headless-git.o git.res GIT-LDFLAGS > $(QUIET_LINK)$(CC) $(ALL_CFLAGS) $(ALL_LDFLAGS) -mwindows -o $@ >$< git.res > >-git-%$X: %.o GIT-LDFLAGS $(GITLIBS) >+git-%$X: %.o GIT-LDFLAGS $(GITLIBS) compile_rust > $(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) $(filter >%.o,$^) $(LIBS) > >-git-imap-send$X: imap-send.o $(IMAP_SEND_BUILDDEPS) GIT-LDFLAGS >$(GITLIBS) >+git-imap-send$X: imap-send.o $(IMAP_SEND_BUILDDEPS) GIT-LDFLAGS >+$(GITLIBS) compile_rust > $(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) $(filter >%.o,$^) \ > $(IMAP_SEND_LDFLAGS) $(LIBS) > >-git-http-fetch$X: http.o http-walker.o http-fetch.o GIT-LDFLAGS $(GITLIBS) >+git-http-fetch$X: http.o http-walker.o http-fetch.o GIT-LDFLAGS >+$(GITLIBS) compile_rust > $(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) $(filter >%.o,$^) \ > $(CURL_LIBCURL) $(LIBS) >-git-http-push$X: http.o http-push.o GIT-LDFLAGS $(GITLIBS) >+git-http-push$X: http.o http-push.o GIT-LDFLAGS $(GITLIBS) compile_rust > $(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) $(filter >%.o,$^) \ > $(CURL_LIBCURL) $(EXPAT_LIBEXPAT) $(LIBS) > >@@ -2911,11 +2934,11 @@ $(REMOTE_CURL_ALIASES): >$(REMOTE_CURL_PRIMARY) > ln -s $< $@ 2>/dev/null || \ > cp $< $@ > >-$(REMOTE_CURL_PRIMARY): remote-curl.o http.o http-walker.o GIT-LDFLAGS >$(GITLIBS) >+$(REMOTE_CURL_PRIMARY): remote-curl.o http.o http-walker.o GIT-LDFLAGS >+$(GITLIBS) compile_rust > $(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) $(filter >%.o,$^) \ > $(CURL_LIBCURL) $(EXPAT_LIBEXPAT) $(LIBS) > >-scalar$X: scalar.o GIT-LDFLAGS $(GITLIBS) >+scalar$X: scalar.o GIT-LDFLAGS $(GITLIBS) compile_rust > $(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) \ > $(filter %.o,$^) $(LIBS) > >@@ -2925,6 +2948,7 @@ $(LIB_FILE): $(LIB_OBJS) > $(XDIFF_LIB): $(XDIFF_OBJS) > $(QUIET_AR)$(RM) $@ && $(AR) $(ARFLAGS) $@ $^ > >+ > $(REFTABLE_LIB): $(REFTABLE_OBJS) > $(QUIET_AR)$(RM) $@ && $(AR) $(ARFLAGS) $@ $^ > >@@ -3294,7 +3318,7 @@ perf: all > > t/helper/test-tool$X: $(patsubst %,t/helper/%,$(TEST_BUILTINS_OBJS)) >$(UNIT_TEST_DIR)/test-lib.o > >-t/helper/test-%$X: t/helper/test-%.o GIT-LDFLAGS $(GITLIBS) >+t/helper/test-%$X: t/helper/test-%.o GIT-LDFLAGS $(GITLIBS) >+compile_rust > $(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) $(filter >%.o,$^) $(filter %.a,$^) $(LIBS) > > check-sha1:: t/helper/test-tool$X >@@ -3756,7 +3780,10 @@ cocciclean: > $(RM) -r .build/contrib/coccinelle > $(RM) contrib/coccinelle/*.cocci.patch > >-clean: profile-clean coverage-clean cocciclean >+rustclean: >+ cd rust && cargo clean >+ >+clean: profile-clean coverage-clean cocciclean rustclean > $(RM) -r .build $(UNIT_TEST_BIN) > $(RM) GIT-TEST-SUITES > $(RM) po/git.pot po/git-core.pot >@@ -3911,13 +3938,13 @@ FUZZ_CXXFLAGS ?= $(ALL_CFLAGS) > .PHONY: fuzz-all > fuzz-all: $(FUZZ_PROGRAMS) > >-$(FUZZ_PROGRAMS): %: %.o oss-fuzz/dummy-cmd-main.o $(GITLIBS) GIT- >LDFLAGS >+$(FUZZ_PROGRAMS): %: %.o oss-fuzz/dummy-cmd-main.o $(GITLIBS) >+GIT-LDFLAGS compile_rust > $(QUIET_LINK)$(FUZZ_CXX) $(FUZZ_CXXFLAGS) -o $@ $(ALL_LDFLAGS) \ > -Wl,--allow-multiple-definition \ > $(filter %.o,$^) $(filter %.a,$^) $(LIBS) $(LIB_FUZZING_ENGINE) > > $(UNIT_TEST_PROGS): $(UNIT_TEST_BIN)/%$X: $(UNIT_TEST_DIR)/%.o >$(UNIT_TEST_OBJS) \ >- $(GITLIBS) GIT-LDFLAGS >+ $(GITLIBS) GIT-LDFLAGS compile_rust > $(call mkdir_p_parent_template) > $(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) \ > $(filter %.o,$^) $(filter %.a,$^) $(LIBS) @@ -3936,7 +3963,7 @@ >$(UNIT_TEST_DIR)/clar.suite: $(UNIT_TEST_DIR)/clar-decls.h >$(UNIT_TEST_DIR)/gene > $(UNIT_TEST_DIR)/clar/clar.o: $(UNIT_TEST_DIR)/clar.suite > $(CLAR_TEST_OBJS): $(UNIT_TEST_DIR)/clar-decls.h > $(CLAR_TEST_OBJS): EXTRA_CPPFLAGS = -I$(UNIT_TEST_DIR) >-$(CLAR_TEST_PROG): $(UNIT_TEST_DIR)/clar.suite $(CLAR_TEST_OBJS) $(GITLIBS) >GIT-LDFLAGS >+$(CLAR_TEST_PROG): $(UNIT_TEST_DIR)/clar.suite $(CLAR_TEST_OBJS) >+$(GITLIBS) GIT-LDFLAGS compile_rust > $(call mkdir_p_parent_template) > $(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) $(filter >%.o,$^) $(LIBS) > >diff --git a/build_rust.sh b/build_rust.sh new file mode 100755 index >000000000000..192385a1d961 >--- /dev/null >+++ b/build_rust.sh >@@ -0,0 +1,57 @@ >+#!/bin/sh >+ >+ >+rustc -vV || exit $? >+cargo --version || exit $? >+ >+dir_git_root=${0%/*} >+dir_build=$1 >+rust_build_profile=$2 >+crate=$3 >+ >+dir_rust=$dir_git_root/rust >+ >+if [ "$dir_git_root" = "" ]; then >+ echo "did not specify the directory for the root of git" >+ exit 1 >+fi >+ >+if [ "$dir_build" = "" ]; then >+ echo "did not specify the build directory" >+ exit 1 >+fi >+ >+if [ "$rust_build_profile" = "" ]; then >+ echo "did not specify the rust_build_profile" >+ exit 1 >+fi >+ >+if [ "$rust_build_profile" = "release" ]; then >+ rust_args="--release" >+ export RUSTFLAGS='' >+elif [ "$rust_build_profile" = "debug" ]; then >+ rust_args="" >+ export RUSTFLAGS='-C debuginfo=2 -C opt-level=1 -C force-frame-pointers=yes' >+else >+ echo "illegal rust_build_profile value $rust_build_profile" >+ exit 1 >+fi >+ >+cd $dir_rust && cargo clean && pwd && cargo build -p $crate $rust_args; >+cd $dir_git_root >+ >+libfile="lib${crate}.a" >+if rustup show active-toolchain | grep windows-msvc; then >+ libfile="${crate}.lib" >+fi >+dst=$dir_build/$libfile >+ >+if [ "$dir_git_root" != "$dir_build" ]; then >+ src=$dir_rust/target/$rust_build_profile/$libfile >+ if [ ! -f $src ]; then >+ echo >&2 "::error:: cannot find path of static library $src is not a file or does not >exist" >+ exit 5 >+ fi >+ >+ rm $dst 2>/dev/null >+ mv $src $dst >+fi >diff --git a/git-compat-util.h b/git-compat-util.h index >4678e21c4cb8..82dc99764ac0 100644 >--- a/git-compat-util.h >+++ b/git-compat-util.h >@@ -196,6 +196,23 @@ static inline int is_xplatform_dir_sep(int c) #include >"compat/msvc.h" > #endif > >+/* rust types */ >+typedef uint8_t u8; >+typedef uint16_t u16; >+typedef uint32_t u32; >+typedef uint64_t u64; >+ >+typedef int8_t i8; >+typedef int16_t i16; >+typedef int32_t i32; >+typedef int64_t i64; >+ >+typedef float f32; >+typedef double f64; >+ >+typedef size_t usize; >+typedef ptrdiff_t isize; >+ > /* used on Mac OS X */ > #ifdef PRECOMPOSE_UNICODE > #include "compat/precompose_utf8.h" >diff --git a/meson.build b/meson.build >index 596f5ac7110e..324f968338b9 100644 >--- a/meson.build >+++ b/meson.build >@@ -267,6 +267,40 @@ version_gen_environment.set('GIT_DATE', >get_option('build_date')) version_gen_environment.set('GIT_USER_AGENT', >get_option('user_agent')) version_gen_environment.set('GIT_VERSION', >get_option('version')) > >+if get_option('optimization') in ['2', '3', 's', 'z'] >+ rust_build_profile = 'release' >+else >+ rust_build_profile = 'debug' >+endif >+ >+# Run `rustup show active-toolchain` and capture output rustup_out = >+run_command('rustup', 'show', 'active-toolchain', >+ check: true).stdout().strip() >+ >+rust_crates = ['xdiff'] >+rust_builds = [] >+ >+foreach crate : rust_crates >+ if rustup_out.contains('windows-msvc') >+ libfile = crate + '.lib' >+ else >+ libfile = 'lib' + crate + '.a' >+ endif >+ >+ rust_builds += custom_target( >+ 'rust_build_'+crate, >+ output: libfile, >+ build_by_default: true, >+ build_always_stale: true, >+ command: [ >+ meson.project_source_root() / 'build_rust.sh', >+ meson.current_build_dir(), rust_build_profile, crate, >+ ], >+ install: false, >+ ) >+endforeach >+ >+ > compiler = meson.get_compiler('c') > > libgit_sources = [ >@@ -1678,14 +1712,16 @@ version_def_h = custom_target( libgit_sources += >version_def_h > > libgit = declare_dependency( >- link_with: static_library('git', >- sources: libgit_sources, >- c_args: libgit_c_args + [ >- '-DGIT_VERSION_H="' + version_def_h.full_path() + '"', >- ], >- dependencies: libgit_dependencies, >- include_directories: libgit_include_directories, >- ), >+ link_with: [ >+ static_library('git', >+ sources: libgit_sources, >+ c_args: libgit_c_args + [ >+ '-DGIT_VERSION_H="' + version_def_h.full_path() + '"', >+ ], >+ dependencies: libgit_dependencies, >+ include_directories: libgit_include_directories, >+ ), >+ ] + rust_builds, > compile_args: libgit_c_args, > dependencies: libgit_dependencies, > include_directories: libgit_include_directories, diff --git a/rust/Cargo.toml >b/rust/Cargo.toml new file mode 100644 index 000000000000..ed3d79d7f827 >--- /dev/null >+++ b/rust/Cargo.toml >@@ -0,0 +1,6 @@ >+[workspace] >+members = [ >+ "xdiff", >+ "interop", >+] >+resolver = "2" >diff --git a/rust/interop/Cargo.toml b/rust/interop/Cargo.toml new file mode >100644 index 000000000000..045e3b01cfad >--- /dev/null >+++ b/rust/interop/Cargo.toml >@@ -0,0 +1,14 @@ >+[package] >+name = "interop" >+version = "0.1.0" >+edition = "2021" >+ >+[lib] >+name = "interop" >+path = "src/lib.rs" >+## staticlib to generate xdiff.a for use by gcc ## cdylib (optional) to >+generate xdiff.so for use by gcc ## rlib is required by the rust unit >+tests crate-type = ["staticlib", "rlib"] >+ >+[dependencies] >diff --git a/rust/interop/src/lib.rs b/rust/interop/src/lib.rs new file mode 100644 >index 000000000000..e69de29bb2d1 diff --git a/rust/xdiff/Cargo.toml >b/rust/xdiff/Cargo.toml new file mode 100644 index >000000000000..eb7966aada64 >--- /dev/null >+++ b/rust/xdiff/Cargo.toml >@@ -0,0 +1,15 @@ >+[package] >+name = "xdiff" >+version = "0.1.0" >+edition = "2021" >+ >+[lib] >+name = "xdiff" >+path = "src/lib.rs" >+## staticlib to generate xdiff.a for use by gcc ## cdylib (optional) to >+generate xdiff.so for use by gcc ## rlib is required by the rust unit >+tests crate-type = ["staticlib", "rlib"] >+ >+[dependencies] >+interop = { path = "../interop" } >diff --git a/rust/xdiff/src/lib.rs b/rust/xdiff/src/lib.rs new file mode 100644 index >000000000000..e69de29bb2d1 Does this introduce Rust as a mandatory dependency for git? If so, it cuts out numerous platforms.

On the Git mailing list, "Kristoffer Haugsbakk" wrote (reply to this):

On Sat, Aug 23, 2025, at 15:43, [email protected] wrote: > On August 22, 2025 11:56 PM, Ezekiel Newren wrote: >>From: Ezekiel Newren <[email protected]> >> >>Upcoming patches will simplify xdiff, while also porting parts of it to Rust. In >>preparation, add some stubs and setup the Rust build. For now, it is easier to let >>cargo build rust and have make or meson merely link against the static library that >>cargo builds. In line with ongoing libification efforts, use multiple crates to allow >>more modularity on the Rust side. xdiff is the crate that this series will focus on, but >>we also introduce the interop crate for future patch series. >> >>In order to facilitate interoperability between C and Rust, introduce C definitions for >>Rust primitive types in git-compat-util.h. >> >>Signed-off-by: Ezekiel Newren <[email protected]> >>[snip snip] > > Does this introduce Rust as a mandatory dependency for git? If so, it cuts out > numerous platforms. The proposed platform support policy is in patch 1. https://lore.kernel.org/git/6d065f550fe871cf010409f7bd2a63438cf52723.1755921357.git.gitgitgadget@gmail.com/

On the Git mailing list, Ezekiel Newren wrote (reply to this):

On Sat, Aug 23, 2025 at 7:44 AM <[email protected]> wrote: > Does this introduce Rust as a mandatory dependency for git? If so, it cuts out > numerous platforms. Yes it does.

On the Git mailing list, wrote (reply to this):

On August 23, 2025 10:26 AM, Kristoffer Haugsbakk wrote: >On Sat, Aug 23, 2025, at 15:43, [email protected] wrote: >> On August 22, 2025 11:56 PM, Ezekiel Newren wrote: >>>From: Ezekiel Newren <[email protected]> >>> >>>Upcoming patches will simplify xdiff, while also porting parts of it >>>to Rust. In preparation, add some stubs and setup the Rust build. For >>>now, it is easier to let cargo build rust and have make or meson >>>merely link against the static library that cargo builds. In line with >>>ongoing libification efforts, use multiple crates to allow more >>>modularity on the Rust side. xdiff is the crate that this series will focus on, but we >also introduce the interop crate for future patch series. >>> >>>In order to facilitate interoperability between C and Rust, introduce >>>C definitions for Rust primitive types in git-compat-util.h. >>> >>>Signed-off-by: Ezekiel Newren <[email protected]> [snip snip] >> >> Does this introduce Rust as a mandatory dependency for git? If so, it >> cuts out numerous platforms. > >The proposed platform support policy is in patch 1. > >https://lore.kernel.org/git/6d065f550fe871cf010409f7bd2a63438cf52723.1755 >[email protected]/ It is a very disappointing policy to be honest. It kicks me off git because Rust is not available on my platform, representing tens of thousands of users in North American alone. Rust is not available, but may be in a few years, but there is no guarantee that the hardware vendor (HPE) will provide support. I previously commented about the problem with Rust and was not taken seriously. This is disappointing and exclusionary. The assertion in the policy that Rust is easily interoperable is incorrect. Not Thanks, Randall

On the Git mailing list, Elijah Newren wrote (reply to this):

Hi Randall, On Sat, Aug 23, 2025 at 8:06 AM <[email protected]> wrote: > > On August 23, 2025 10:26 AM, Kristoffer Haugsbakk wrote: > >On Sat, Aug 23, 2025, at 15:43, [email protected] wrote: [...] > >> Does this introduce Rust as a mandatory dependency for git? If so, it > >> cuts out numerous platforms. > > > >The proposed platform support policy is in patch 1. > > > >https://lore.kernel.org/git/6d065f550fe871cf010409f7bd2a63438cf52723.1755 > >[email protected]/ > > It is a very disappointing policy to be honest. It kicks me off git because Rust is > not available on my platform, representing tens of thousands of users in North > American alone. Rust is not available, but may be in a few years, but there is no > guarantee that the hardware vendor (HPE) will provide support. I previously > commented about the problem with Rust and was not taken seriously. This is > disappointing and exclusionary. I don't think that's fair. A quick reminder on the history: There was lots of excitement about potentially introducing Rust two years ago at our virtual Git contributors conference. Taylor formally proposed adopting it on the mailing list a year and a half ago. And at Git Merge last year, among those in attendance, there was broad significant interest in adopting Rust with unanimous support for letting it move forward among those that were present (which, yes, we know wasn't everyone). And there's the three rounds so far of this patch series. At every discussion where you weren't present, someone else would always bring up you and NonStop, and point out how you've been a very positive long-term member of the Git community and how Rust adoption would likely negatively affect you, which would be regrettable. We waited years to adopt Rust precisely (and I believe solely) because of your objections. Josh and Calvin even went the route of making optional not-even-built-by-default Rust libraries (libgit-rs and libgit-sys) when they wanted to add some Rust bindings. If years of deference by other community members isn't considered taking you seriously, I don't know what is. I agree that it is disappointing that there isn't a clear way to both gain the compelling advantages of Rust while also retaining the full current extent of our widespread platform support. It's doubly unfortunate since you're such a positive contributing member of the community. But not allowing us to ever gain the advantages of Rust is problematic too. So, a decision has to be made, one way or the other. If it helps, here's the statements I've seen from long term community members on Ezekiel's proposal for a hard dependency so far, most of which call out the reduced platform support (whether in favor of the proposal or not): * Randall: https://lore.kernel.org/git/[email protected]/ * brian: https://lore.kernel.org/git/[email protected]/ * Taylor: https://lore.kernel.org/git/[email protected]/ * Patrick: https://lore.kernel.org/git/[email protected]/ * Phillip: https://lore.kernel.org/git/[email protected]/ There's also been some emails that can be read as implicitly making a position statement on the topic from long term community members: * Junio: https://lore.kernel.org/git/[email protected]/ * Johannes: https://lore.kernel.org/git/[email protected]/ * Elijah: https://lore.kernel.org/git/[email protected]/ (I figured my noted assistance of this series meant I didn't need to explicitly call out my support for it.) > The assertion in the policy that Rust is easily interoperable is incorrect. Are you mixing up interoperability with portability? Without further context than your email provides, it appears so to me. Rust code can call C code and vice-versa within the same process without huge amounts of serializing and deserializing of data structures, and without what amounts to something close to an operating system context switch in order to ensure call stacks are as expected for the language in question. To me, that means we can call the two languages easily interoperable. On the other hand, portability of those languages is about whether those languages have compilers supported on various hardware platforms. The document explicitly calls out that fewer systems have a Rust compiler than have a C compiler, and that Rust adoption would thus reduce how portable Git is. Are you referring to this lower portability that the document itself also calls out, or are you pointing out additional issues with interoperation between the languages on a platform where compilers for both languages exist? If the latter, could you provide more details? I know my email is probably disappointing to you, at a minimum. I'm sorry about that. I hope it's helpful, at least in having links to where various folks in the community stand if nothing else.

On the Git mailing list, wrote (reply to this):

On September 8, 2025 6:12 AM, Phillip Wood wrote: >On 07/09/2025 17:09, [email protected] wrote: >> On September 7, 2025 12:10 AM, Elijah Newren wrote: >>> Sorry for the delay; life outside of work is challenging at the moment... >>> >> >> I am going to address the critical point mentioned below and snip the rest for >brevity. >> >>> I still don't see why distributors _must_ ship the latest version of >>> Git and why folks on some platforms are considered broken if they are using a >slightly older version. >>> Let me ask again: has anyone answered why this is considered >>> mandatory? If they have, I've missed it, but I've asked multiple >>> times. Even if you want to lump "distributors cannot build a newer >>> version" under the umbrella of "breaking changes", I argue it's a >>> much different kind of break and one which merits different timelines for >handling than e.g. lumping it in with 3.0. >> >> I do not see that distributors _must_ ship the latest version. Suppose >> we are on >> 2.51.0 and a CVE comes out that prohibits its use in an organization >> that does not allow any medium-high to high CVEs. This represents >> hundreds of thousands of impacted users in my community alone. How >> does the CVE get applied if the latest cannot be built and the git >> team does not apply the CVE fixes to old versions. Personally, I do >> not care if git versions are different between work and home, or even >> between CI/CD and other platforms. I don't even care if I have to use >> JGit instead of git in some situations (which I see is a likely >> outcome of this discussion). Is there an official statement of what an LTS means? > >We're currently discussing what promises we can make about supporting a non-rust >version of git. > >> In other projects LTS is typically, and formally by policy 5 years. > >I know commercial linux distributions offer that kind of support but are there really >open source projects that guarantee 5 years of security updates without any kind >of support contract? OpenSSL provides 5 years of security fix support (at no cost) for LTS designated releases. Currently 3.0 ending Sept 2026 and 3.5 ending around October 2030. After those dates, there is a fee-based support arrangement available. >> From what others have said here, positions of 6 months, 3 years, and >> "apply it yourself if you want to continue to use git" have been made. > >Yes it is still being discussed, and no one is volunteering to offer five years of >support. > >> The core problem of adding a breaking dependency is when a CVE comes >> out that prohibits git from being used at all. If the git team is not >> going to provide a clear statement, one way or another, if how CVEs >> (at whatever severity level) will not have a commitment of any kind, >> then distributors are essentially cast adrift and on our own. It would >> be helpful of those of us who donate our time, for no compensation, >> are able to plan for this in a meaningful way. > >Doesn't your company make a front end to git? Are you saying that the >management does not allocate any staff time to work on git itself and expects the >community to provide it with free security updates? The REAL PROBLEM that is not being addressed in this thread is that large companies (the ones who process your credit cards, build your cars, manufacture your drugs, and tool your factories (a.k.a. NonStop customers), are generally unwilling to accept CVE fixes from third parties. The fixes have to be part of the official code base or the fixes will not accepted. That means that either: 1. the git team has to officially sanction the fixes; or 2. do the fixes themselves. A compromise may be possible to keep a support branch around in the official git repo, for those of us who do not have rust available to contribute to, specifically for post C-deprecation CVE fixes, but I am not sure this is practical. It would also require occasional assistance from the git team, to make sense of some of the fixes, If they apply, as none of us are rust experts. I am already allocated to spending between 10 and 20 hours a month to git, which usually involves running and verifying build/test cycles. Since git tests are flakey, in some cases, I have to manually examine each failure situation and decide whether the failures are sufficient to pass the releases. These have been reported previously without resolution and do not bear discussion here. It is important to understand that many git customers in high audit situations build git on their own because they do not trust third party builds, so this needs to remain an option. >> Please remember that >> we have to justify our participation to our management teams to be >> allowed to continue to participate. >I'm confused by this, as the sentence before say's you're donating your time for no >compensation. My company pretends to donates my time with very little direct benefit to them. My participation is because I feel it is important for my community. No, I do not get a salary for my git time. It is evenings and weekends. Any time I spend during working hours has to be made up during off hours. > >> Nothing is free from this end >> and if fixing (not just applying fixes) CVEs are now 100% our >> responsibility, if would be critical to know that when we build our >> business cases to our bosses, who I am fairly certain will say an >> emphatic no. > >In the long term, unless your platform gains a rust compiler I'm afraid I think that is >most likely outcome. > >> Also remember that without support from the git team, the code base is >> no longer the same, meaning the auditors will not necessarily accept >> fixes from third-party sources. > >I think I saw a suggestion/question about the possibility of hosting any long term >support branch that is maintained by interested parties within the main repository. >Would that help? DEFINITELY. With assistance as above. With some help, we may be able to make this work. It might require a deeper participation on my part and those on my team to approve changes, which we would consider. >I appreciate that any move to rust would be very disappointing and disruptive to >you but the community has to weigh up the benefits rust has to offer against that. The community has plans for Rust but they have not taken shape fully as of yet. I have personally been badgering product management to make this happen, and I might know more in a month or so. However, this takes time, and 6 months is not enough. --Randall

On the Git mailing list, wrote (reply to this):

On September 8, 2025 11:31 AM, Elijah Newren wrote: >On Sun, Sep 7, 2025 at 9:10 AM <[email protected]> wrote: >> >> On September 7, 2025 12:10 AM, Elijah Newren wrote: >> >Sorry for the delay; life outside of work is challenging at the moment... >> > >> >> I am going to address the critical point mentioned below and snip the rest for >brevity. >> >> >I still don't see why distributors _must_ ship the latest version of >> >Git and why folks on some platforms are considered broken if they are using a >slightly older version. >> >Let me ask again: has anyone answered why this is considered >> >mandatory? If they have, I've missed it, but I've asked multiple >> >times. Even if you want to lump "distributors cannot build a newer >> >version" under the umbrella of "breaking changes", I argue it's a >> >much different kind of break and one which merits different timelines for >handling than e.g. lumping it in with 3.0. >> >> I do not see that distributors _must_ ship the latest version. Suppose >> we are on >> 2.51.0 and a CVE comes out that prohibits its use in an organization >> that does not allow any medium-high to high CVEs. This represents >> hundreds of thousands of impacted users in my community alone. How >> does the CVE get applied if the latest cannot be built and the git >> team does not apply the CVE fixes to old versions. Personally, I do >> not care if git versions are different between work and home, or even >> between CI/CD and other platforms. I don't even care if I have to use >> JGit instead of git in some situations (which I see is a likely >> outcome of this discussion). Is there an official statement of what an LTS means? >In other projects LTS is typically, and formally by policy 5 years. >> From what others have said here, positions of 6 months, 3 years, and >> "apply it yourself if you want to continue to use git" have been made. >> >> The core problem of adding a breaking dependency is when a CVE comes >> out that prohibits git from being used at all. If the git team is not >> going to provide a clear statement, one way or another, if how CVEs >> (at whatever severity level) will not have a commitment of any kind, >> then distributors are essentially cast adrift and on our own. It would >> be helpful of those of us who donate our time, for no compensation, >> are able to plan for this in a meaningful way. Please remember that we >> have to justify our participation to our management teams to be >> allowed to continue to participate. Nothing is free from this end and >> if fixing (not just applying fixes) CVEs are now 100% our >> responsibility, if would be critical to know that when we build our >> business cases to our bosses, who I am fairly certain will say an >> emphatic no. >> >> Also remember that without support from the git team, the code base is >> no longer the same, meaning the auditors will not necessarily accept >> fixes from third-party sources. This particular point enabled adoption >> on some platforms, particularly NonStop. >> Adoption was at 1-2 customers when we had a divergent code based >> because some platform fixes being different from the standard >> code-base and could not be certified as valid. Once the code-base >> because common, adoption was rapid and enthusiastic. >> If this goes away, I suspect that adoption rates will go negative. >> I am aware that that particular discussion is actually happening in >> some organizations in my community right now, with companies looking >> for alternatives to git based on this discussion thread. >> >> With over a decade of respect and participation, Randall > >Thanks, Randall, this is useful information. In regards to one point not fully covered >by Phillip: > >> Also remember that without support from the git team, the code base is >> no longer the same, meaning the auditors will not necessarily accept >> fixes from third-party sources. > >Why does it need to be "third-party" sources? Linus years ago blessed having >someone else be in charge of providing updates for stable releases of Linux. Junio >could do the same with Git and similarly mark an individual or group of people as >the maintainers for the last Rust-optional version of Git, and those individuals could >make official releases of Git with extended security fix support. Then it's not every >platform repeating the backporting work that needs to be done, but rather >individuals from the affected platform(s) collaborating on that work and then >making official first-party releases. Linux has one set of rules, and other platforms have others. I do not define the audit requirements for PCI, SWIFT, or HIPPA compliance (and other rules outside of North America), which apply one way or another to most of my community. The audit teams, which are both internal to the companies and at governmental regulatory levels, do this. It is 100% out of my control but is a reality. Fixes to any code involved in managing financial and health instruments must be done by authorized and recognized sources. I am not one of them.

On the Git mailing list, wrote (reply to this):

On September 8, 2025 11:10 AM, Ezekiel Newren wrote: >On Sun, Sep 7, 2025 at 10:10 AM <[email protected]> wrote: >> >> On September 7, 2025 12:10 AM, Elijah Newren wrote: >> >Sorry for the delay; life outside of work is challenging at the moment... >> > >> >> I am going to address the critical point mentioned below and snip the rest for >brevity. >> >> >I still don't see why distributors _must_ ship the latest version of >> >Git and why folks on some platforms are considered broken if they are using a >slightly older version. >> >Let me ask again: has anyone answered why this is considered >> >mandatory? If they have, I've missed it, but I've asked multiple >> >times. Even if you want to lump "distributors cannot build a newer >> >version" under the umbrella of "breaking changes", I argue it's a >> >much different kind of break and one which merits different timelines for >handling than e.g. lumping it in with 3.0. >> >> I do not see that distributors _must_ ship the latest version. Suppose >> we are on >> 2.51.0 and a CVE comes out that prohibits its use in an organization >> that does not allow any medium-high to high CVEs. This represents >> hundreds of thousands of impacted users in my community alone. How >> does the CVE get applied if the latest cannot be built and the git >> team does not apply the CVE fixes to old versions. Personally, I do >> not care if git versions are different between work and home, or even >> between CI/CD and other platforms. I don't even care ... > >Ok, that answers the question for NonStop, but that doesn't answer the question >for the plethora of other distributions. Most distributions don't ship the latest >version of Git in their package manager, and if an organization deems it critical to >have the latest they can build it themselves and ignore the Git version in the >package manager. So why does Windows, Mac, Linux, etc... _need_ the latest >version of Git in the package manager? > >If security updates are backported to NonStop, until that platform supports Rust, >then I don't see why using an older version of Git in Windows, Mac, Linux, etc... is a >catastrophe. Most existing distributions _can_ package the latest version of Git, but >they _don't_. > >I reiterate Elijah's question "Why _must_ distributors ship the latest version of >Git?". My emphatic answer is that they do not. There is no requirement from me or anyone I know to ship the latest version. What is crucial is that there be fixes for medium-high and above CVEs that are delivered in 30 days from initial fix availability (that would be in Rust, for this conversation and applied to C). If that were supported, I could live with as would my customers and their auditors for LTS releases. Please see my expectation of LTS defined elsewhere in this thread - essentially 5 years. Perhaps 3 at a bare minimum.

On the Git mailing list, Elijah Newren wrote (reply to this):

On Mon, Sep 8, 2025 at 8:37 AM <[email protected]> wrote: > > On September 8, 2025 11:31 AM, Elijah Newren wrote: > >On Sun, Sep 7, 2025 at 9:10 AM <[email protected]> wrote: > >> > >> On September 7, 2025 12:10 AM, Elijah Newren wrote: > >Thanks, Randall, this is useful information. In regards to one point not fully covered > >by Phillip: > > > >> Also remember that without support from the git team, the code base is > >> no longer the same, meaning the auditors will not necessarily accept > >> fixes from third-party sources. > > > >Why does it need to be "third-party" sources? Linus years ago blessed having > >someone else be in charge of providing updates for stable releases of Linux. Junio > >could do the same with Git and similarly mark an individual or group of people as > >the maintainers for the last Rust-optional version of Git, and those individuals could > >make official releases of Git with extended security fix support. Then it's not every > >platform repeating the backporting work that needs to be done, but rather > >individuals from the affected platform(s) collaborating on that work and then > >making official first-party releases. > > Linux has one set of rules, and other platforms have others. I do not define the > audit requirements for PCI, SWIFT, or HIPPA compliance (and other rules outside > of North America), which apply one way or another to most of my community. > The audit teams, which are both internal to the companies and at > governmental regulatory levels, do this. It is 100% out of my control but is a > reality. Fixes to any code involved in managing financial and health instruments > must be done by authorized and recognized sources. I am not one of them. Perhaps I wasn't clear? Let me try to summarize what I've understood of the conversation: Randall: We need to have official git releases for the last Rust-optional release. Elijah: Great! Let's enable interested folks to make official git releases for the last Rust-optional release. Randall: We need to have official git releases for the last Rust-optional release. Which makes me just want to repeat what I said last time -- let's enable some folks to do that.

On the Git mailing list, wrote (reply to this):

On September 8, 2025 12:13 PM, Elijah Newren wrote: >On Mon, Sep 8, 2025 at 8:37 AM <[email protected]> wrote: >> >> On September 8, 2025 11:31 AM, Elijah Newren wrote: >> >On Sun, Sep 7, 2025 at 9:10 AM <[email protected]> wrote: >> >> >> >> On September 7, 2025 12:10 AM, Elijah Newren wrote: > >> >Thanks, Randall, this is useful information. In regards to one point not fully >covered >> >by Phillip: >> > >> >> Also remember that without support from the git team, the code base is >> >> no longer the same, meaning the auditors will not necessarily accept >> >> fixes from third-party sources. >> > >> >Why does it need to be "third-party" sources? Linus years ago blessed having >> >someone else be in charge of providing updates for stable releases of Linux. >Junio >> >could do the same with Git and similarly mark an individual or group of people as >> >the maintainers for the last Rust-optional version of Git, and those individuals >could >> >make official releases of Git with extended security fix support. Then it's not >every >> >platform repeating the backporting work that needs to be done, but rather >> >individuals from the affected platform(s) collaborating on that work and then >> >making official first-party releases. >> >> Linux has one set of rules, and other platforms have others. I do not define the >> audit requirements for PCI, SWIFT, or HIPPA compliance (and other rules outside >> of North America), which apply one way or another to most of my community. >> The audit teams, which are both internal to the companies and at >> governmental regulatory levels, do this. It is 100% out of my control but is a >> reality. Fixes to any code involved in managing financial and health instruments >> must be done by authorized and recognized sources. I am not one of them. > >Perhaps I wasn't clear? Let me try to summarize what I've understood >of the conversation: > >Randall: We need to have official git releases for the last >Rust-optional release. >Elijah: Great! Let's enable interested folks to make official git >releases for the last Rust-optional release. >Randall: We need to have official git releases for the last >Rust-optional release. > >Which makes me just want to repeat what I said last time -- let's >enable some folks to do that. Ok, what does that look like. When and how? Will I get added to the list of committers for this? How many people? Starting when, ending when? It would be very nice to have some kind of project plan for this with dependencies and milestones - I have been asking. If someone sends me a list, I can start officially tracking it.

gitgitgadget-git · 2025-08-23T13:47:59Z

User <[email protected]> has been added to the cc: list.

gitgitgadget-git · 2025-08-23T16:17:48Z

Makefile


 # Guard against environment variables
 BUILTIN_OBJS =
 BUILT_INS =


On the Git mailing list, Junio C Hamano wrote (reply to this):

"Ezekiel Newren via GitGitGadget" <[email protected]> writes: > diff --git a/rust/xdiff/src/lib.rs b/rust/xdiff/src/lib.rs > index e69de29bb2d1..8b137891791f 100644 > --- a/rust/xdiff/src/lib.rs > +++ b/rust/xdiff/src/lib.rs > @@ -0,0 +1 @@ > + This triggers an "new blank line at EOF" whitespace error while applying. Intended?

On the Git mailing list, Ezekiel Newren wrote (reply to this):

On Sat, Aug 23, 2025 at 10:14 AM Junio C Hamano <[email protected]> wrote: > > "Ezekiel Newren via GitGitGadget" <[email protected]> writes: > > > diff --git a/rust/xdiff/src/lib.rs b/rust/xdiff/src/lib.rs > > index e69de29bb2d1..8b137891791f 100644 > > --- a/rust/xdiff/src/lib.rs > > +++ b/rust/xdiff/src/lib.rs > > @@ -0,0 +1 @@ > > + > > This triggers an "new blank line at EOF" whitespace error while > applying. Intended? "new blank line at EOF" is intentional, but it is showing up in the wrong place in this patch series. Cargo format automatically creates a blank line for empty files. These warnings should have shown up on the "xdiff: introduce rust" commit. I will fix this.

gitgitgadget-git · 2025-08-23T18:09:04Z

Makefile


 # Guard against environment variables
 BUILTIN_OBJS =
 BUILT_INS =


On the Git mailing list, Junio C Hamano wrote (reply to this):

"Ezekiel Newren via GitGitGadget" <[email protected]> writes: > From: Ezekiel Newren <[email protected]> > > Trying to use Rust's Vec in C, or git's ALLOC_GROW() macros (via > wrapper functions) in Rust is painful because: > > * C doing vector things the Rust way would require wrapper functions, > and Rust doing vector things the C way would require wrapper > ... > * Currently, Rust defines its own 'Vec' type that is generic, but its > memory allocator and struct layout weren't designed for > interoperability with C (or any language for that matter), meaning > ... > * Similarly, git defines ALLOC_GROW() and related macros in > git-compat-util.h. While we could add functions allowing Rust to > ... All the good reasons any C (or any non-Rust language for that matter) projects would want to have an interoperability Shim for their dynamically allocated and grown array-like things. > To address these issue, introduce a new type, ivec -- short for > interoperable vector. (We refer to it as 'ivec' generally, though on > the Rust side the struct is called IVec to match Rust style.) I however was hoping by now Rust getting used more widely, somebody has already created a generic "this is how you make C-array and Rust vectors interoperate" wrapper that latecomer projects like us can use without inventing our own. > +INTEROP_OBJS += interop/ivec.o > +.PHONY: interop-objs > +interop-objs: $(INTEROP_OBJS) What is this phony target used for? No other targets seem to depend on this one (I am wondering if we need the latter two lines). > diff --git a/interop/ivec.c b/interop/ivec.c > new file mode 100644 > index 000000000000..9bc2258c04ad > --- /dev/null > +++ b/interop/ivec.c I am wondering if this needs a new hierarchy "interop"; shouldn't the existing "compat" be a good fit enough? I dunno. Even though this is a shim to somebody else's code, it still is a part of our codebase, so our CodingGuidelines for C programs should apply. > @@ -0,0 +1,151 @@ > +#include "ivec.h" > + > +static void ivec_set_capacity(void* self, usize new_capacity) { > + struct rawivec *this = self; - Asterisk sticks to the variable, not type. - The opening and closing {braces} for the function body are written at the leftmost column on its own line. - There should be a blank line between the declarations and the first statement. > + if (new_capacity == 0) > + FREE_AND_NULL(this->ptr); > + else > + this->ptr = xrealloc(this->ptr, new_capacity * this->element_size); > + this->capacity = new_capacity; > +} > + > +void ivec_init(void* self, usize element_size) { > + struct rawivec *this = self; > + this->ptr = NULL; > + this->length = 0; > + this->capacity = 0; > + this->element_size = element_size; > +} I notice that this reintroduces a variable named "this", which was eradicated in 585c0e2e (diff: rename 'this' variables, 2018-02-14). I do not think those who want to use C++ compilers on our C code would not mind "self", so how about doing something like... void ivec_init(void *self_, usize element_size) { struct rawivec *self = self; self->ptr = NULL; self->len = 0; self->capacity = 0; self->element_size = element_size; } ... perhaps? > diff --git a/interop/ivec.h b/interop/ivec.h > new file mode 100644 > index 000000000000..98be4bbeb54a > --- /dev/null > +++ b/interop/ivec.h > @@ -0,0 +1,52 @@ > +#ifndef IVEC_H > +#define IVEC_H > + > +#include "../git-compat-util.h" As we use -I. on the command line, there is no need to add "../" here; just writing #include <git-compat-util.h> should be enough. Also, if this file does not depend on the services compat-util header provides (and I do not think it does from a brief look at its contents), it is better not to include it. Instead, the sources (like ivec.c next door we just saw) should begin themselves with #include of git-compat-util.h header before including ivec.h. > diff --git a/rust/xdiff/src/lib.rs b/rust/xdiff/src/lib.rs > index e69de29bb2d1..8b137891791f 100644 > --- a/rust/xdiff/src/lib.rs > +++ b/rust/xdiff/src/lib.rs > @@ -0,0 +1 @@ > + If this empty line in an otherwise empty file is absolutely necessary to make Rust work, then please arrange .gitattributes to tell git that this file is excempt from the usual blank-at-eof whitespace rule we use. If not, remove that unnecessary empty line. Or perhaps remove the file altogether if nobody looks at it??? In any case, given that our top-level .gitattributes file starts with * whitespace=!indent,trail,space *.[ch] whitespace=indent,trail,space diff=cpp *.sh whitespace=indent,trail,space text eol=lf ... *.bat text eol=crlf CODE_OF_CONDUCT.md -whitespace ... I think a new rule to cover "*.rs" and perhaps *.toml files right before rules for each specific file begin would be in order. Thanks.

On the Git mailing list, Ezekiel Newren wrote (reply to this):

On Sat, Aug 23, 2025 at 12:05 PM Junio C Hamano <[email protected]> wrote: > > To address these issue, introduce a new type, ivec -- short for > > interoperable vector. (We refer to it as 'ivec' generally, though on > > the Rust side the struct is called IVec to match Rust style.) > > I however was hoping by now Rust getting used more widely, somebody > has already created a generic "this is how you make C-array and Rust > vectors interoperate" wrapper that latecomer projects like us can > use without inventing our own. I've looked for code that will do what Git needs, but as far as I know nothing can do everything that my ivec can. > > +INTEROP_OBJS += interop/ivec.o > > +.PHONY: interop-objs > > +interop-objs: $(INTEROP_OBJS) > > What is this phony target used for? No other targets seem to depend > on this one (I am wondering if we need the latter two lines). You are correct. I will remove those lines. > > diff --git a/interop/ivec.c b/interop/ivec.c > > new file mode 100644 > > index 000000000000..9bc2258c04ad > > --- /dev/null > > +++ b/interop/ivec.c > > I am wondering if this needs a new hierarchy "interop"; shouldn't > the existing "compat" be a good fit enough? I dunno. I had considered compat/, but I thought it didn’t fit. I thought it meant that the same API would exist everywhere, as opposed to “We speak different languages, but we’ve agreed on a translator or common protocol”. In particular, an example from ivec, a function on both the C and Rust sides: void ivec_extend_from_slice(void *_self, void const *ptr, usize size); pub fn extend_from_slice(&mut self, slice: &[T]) where T: Clone, The Rust side uses a slice or “fat” pointer, where the C side uses two arguments (a pointer and a size) in its place. The API is different, even if semantically they are the same and they are interoperable. Was I reading too much into the meaning of compat/? Do folks object to using interop/? > Even though this is a shim to somebody else's code, it still is a > part of our codebase, so our CodingGuidelines for C programs should > apply. Sorry I missed those. I will fix them up. > > @@ -0,0 +1,151 @@ > > +#include "ivec.h" > > + > > +static void ivec_set_capacity(void* self, usize new_capacity) { > > + struct rawivec *this = self; > > - Asterisk sticks to the variable, not type. > > - The opening and closing {braces} for the function body are > written at the leftmost column on its own line. > > - There should be a blank line between the declarations and the > first statement. I will make those changes. > > + if (new_capacity == 0) > > + FREE_AND_NULL(this->ptr); > > + else > > + this->ptr = xrealloc(this->ptr, new_capacity * this->element_size); > > + this->capacity = new_capacity; > > +} > > + > > +void ivec_init(void* self, usize element_size) { > > + struct rawivec *this = self; > > + this->ptr = NULL; > > + this->length = 0; > > + this->capacity = 0; > > + this->element_size = element_size; > > +} > > I notice that this reintroduces a variable named "this", which was > eradicated in 585c0e2e (diff: rename 'this' variables, 2018-02-14). > > I do not think those who want to use C++ compilers on our C code > would not mind "self", so how about doing something like... > > > void ivec_init(void *self_, usize element_size) > { > struct rawivec *self = self; > > self->ptr = NULL; > self->len = 0; > self->capacity = 0; > self->element_size = element_size; > } > > ... perhaps? That sounds good. I will make those changes. > > diff --git a/interop/ivec.h b/interop/ivec.h > > new file mode 100644 > > index 000000000000..98be4bbeb54a > > --- /dev/null > > +++ b/interop/ivec.h > > @@ -0,0 +1,52 @@ > > +#ifndef IVEC_H > > +#define IVEC_H > > + > > +#include "../git-compat-util.h" > > As we use -I. on the command line, there is no need to add "../" > here; just writing > > #include <git-compat-util.h> > > should be enough. Also, if this file does not depend on the > services compat-util header provides (and I do not think it does > from a brief look at its contents), it is better not to include it. This file actually does depend on git-compat-util.h, particularly the Rust primitive definitions (e.g. usize, u64, etc...). I'll use the include style you mentioned. > > diff --git a/rust/xdiff/src/lib.rs b/rust/xdiff/src/lib.rs > > index e69de29bb2d1..8b137891791f 100644 > > --- a/rust/xdiff/src/lib.rs > > +++ b/rust/xdiff/src/lib.rs > > @@ -0,0 +1 @@ > > + > > If this empty line in an otherwise empty file is absolutely > necessary to make Rust work, then please arrange .gitattributes to > tell git that this file is excempt from the usual blank-at-eof > whitespace rule we use. If not, remove that unnecessary empty line. > > Or perhaps remove the file altogether if nobody looks at it??? > > In any case, given that our top-level .gitattributes file starts > with > > * whitespace=!indent,trail,space > *.[ch] whitespace=indent,trail,space diff=cpp > *.sh whitespace=indent,trail,space text eol=lf > ... > *.bat text eol=crlf > CODE_OF_CONDUCT.md -whitespace > ... > > I think a new rule to cover "*.rs" and perhaps *.toml files right > before rules for each specific file begin would be in order. This is only a problem for empty files, and we only have empty .rs files to setup the basic Rust build in this commit. It's not a problem later in this series and shouldn't be a problem in the future. I will remove those blank lines from those commits.

On the Git mailing list, Elijah Newren wrote (reply to this):

On Sat, Aug 23, 2025 at 11:05 AM Junio C Hamano <[email protected]> wrote: > > diff --git a/interop/ivec.c b/interop/ivec.c > > new file mode 100644 > > index 000000000000..9bc2258c04ad > > --- /dev/null > > +++ b/interop/ivec.c > > Even though this is a shim to somebody else's code, it still is a > part of our codebase, so our CodingGuidelines for C programs should > apply. Sorry, I should have caught these in my preliminary review before he sent this off to the list. One question, though... > > diff --git a/interop/ivec.h b/interop/ivec.h > > new file mode 100644 > > index 000000000000..98be4bbeb54a > > --- /dev/null > > +++ b/interop/ivec.h > > @@ -0,0 +1,52 @@ > > +#ifndef IVEC_H > > +#define IVEC_H > > + > > +#include "../git-compat-util.h" > > As we use -I. on the command line, there is no need to add "../" > here; just writing > > #include <git-compat-util.h> > > should be enough. Also, if this file does not depend on the > services compat-util header provides (and I do not think it does > from a brief look at its contents), it is better not to include it. Should this rather be #include "git-compat-util.h" with quotes rather than angle brackets? In particular: $ git grep include.*git-compat-util -- '*.[ch]' | wc -l 362 $ git grep include.*git-compat-util -- '*/*.[ch]' | wc -l 125 So, we have 362 includes of git-compat-util.h in our codebase, 125 from subdirectories. Of those: $ git grep include.*git-compat-util -- '*.[ch]' | grep '"' | wc -l 361 $ git grep include.*git-compat-util -- '*.[ch]' | grep '<' | wc -l 1 Only one of these include statements uses angle brackets -- the compiler-tricks/not-constant.c file (which appears to be a temporary hack that we'll eventually delete). I had always assumed <> were for system includes and "" for project includes, but a quick Google search shows the actual situation is quite a bit murkier than I'd realized. Still, our current project practice appears to be double quotes; is that fine here or are you suggesting you'd like the current project practice to be changed?

On the Git mailing list, Junio C Hamano wrote (reply to this):

Elijah Newren <[email protected]> writes: >> > +#include "../git-compat-util.h" >> >> As we use -I. on the command line, there is no need to add "../" >> here; just writing >> >> #include <git-compat-util.h> >> >> should be enough. Also, if this file does not depend on the >> services compat-util header provides (and I do not think it does >> from a brief look at its contents), it is better not to include it. > > Should this rather be > > #include "git-compat-util.h" I meant <>; when "" included header is not found, it falls back as if it were <> included, IIRC, so writing <> when you specify exactly where your headers are with -I. avoids such unnecessary fallback in theory, but as both <> and "" search for implementation-defined places, the distinction does not make much practical difference. > Still, our current project practice appears to be double quotes; is > that fine here or are you suggesting you'd like the current project > practice to be changed? It would be nice if we could do so, but I do not think it is worth the patch churn.

gitgitgadget-git · 2025-08-24T13:33:41Z

Makefile


 # Guard against environment variables
 BUILTIN_OBJS =
 BUILT_INS =


On the Git mailing list, Ben Knoble wrote (reply to this):

> Le 22 août 2025 à 23:56, Ezekiel Newren via GitGitGadget <[email protected]> a écrit : > > From: Ezekiel Newren <[email protected]> > > Trying to use Rust's Vec in C, or git's ALLOC_GROW() macros (via > wrapper functions) in Rust is painful because: > > * C doing vector things the Rust way would require wrapper functions, > and Rust doing vector things the C way would require wrapper > functions, so ivec was created to ensure a consistent contract > between the 2 languages for how to manipulate a vector. > * Currently, Rust defines its own 'Vec' type that is generic, but its > memory allocator and struct layout weren't designed for > interoperability with C (or any language for that matter), meaning > that the C side cannot push to or expand a 'Vec' without defining > wrapper functions in Rust that C can call. Without special care, > the two languages might use different allocators (malloc/free on > the C side, and possibly something else in Rust), which would make > it difficult for a function in one language to free elements > allocated by a call from a function in the other language. > * Similarly, git defines ALLOC_GROW() and related macros in > git-compat-util.h. While we could add functions allowing Rust to > invoke something similar to those macros, passing three variables > (pointer, length, allocated_size) instead of a single variable > (vector) across the language boundary requires more cognitive > overhead for readers to keep track of and makes it easier to make > mistakes. Further, for low-level components that we want to > eventually convert to pure Rust, such triplets would feel very out > of place. I’m mildly surprised Vec isn’t a good fit: isn’t it a pointer, length, capacity triple? But it sounds like the main issue is allocator interop… which I would also have thought was supported? At least the current version is documented as being generic against an Allocator, too. > > To address these issue, introduce a new type, ivec -- short for > interoperable vector. (We refer to it as 'ivec' generally, though on > the Rust side the struct is called IVec to match Rust style.) This new > type is specifically designed for FFI purposes, so that both languages > handle the vector in the same way, though it could be used on either > side independently. This type is designed such that it can easily be > replaced by a standard Rust 'Vec' once interoperability is no longer a > concern. Am I reading the patch correctly that the ivec implementation is primarily C? I’m not familiar with too many FFI projects in Rust, but I might have hoped we could write parts in Rust to gain any benefits from that, too. Is that a fool’s errand I’m thinking of?

On the Git mailing list, Ezekiel Newren wrote (reply to this):

On Sun, Aug 24, 2025 at 7:31 AM Ben Knoble <[email protected]> wrote: > I’m mildly surprised Vec isn’t a good fit: isn’t it a pointer, length, capacity triple? But it sounds like the main issue is allocator interop… which I would also have thought was supported? At least the current version is documented as being generic against an Allocator, too. Conceptually yes, semantically and syntactically no. On top of Vec<T> not being defined with #[repr(C)] (which ensures field order, C ABI layout, padding, etc...) the struct definition for Vec isn't constant between Rust versions. I'd be open to suggestions for an alternative to my ivec type. === Rust version 1.61.0 === from: https://doc.rust-lang.org/1.61.0/src/alloc/vec/mod.rs.html#400 #[stable(feature = "rust1", since = "1.0.0")] #[cfg_attr(not(test), rustc_diagnostic_item = "Vec")] #[rustc_insignificant_dtor] pub struct Vec<T, #[unstable(feature = "allocator_api", issue = "32838")] A: Allocator = Global> { buf: RawVec<T, A>, len: usize, } from: https://doc.rust-lang.org/1.61.0/src/alloc/raw_vec.rs.html#52 #[allow(missing_debug_implementations)] pub(crate) struct RawVec<T, A: Allocator = Global> { ptr: Unique<T>, cap: usize, alloc: A, } === Rust version 1.89.0 === from: https://doc.rust-lang.org/1.89.0/src/alloc/vec/mod.rs.html#414 #[stable(feature = "rust1", since = "1.0.0")] #[rustc_diagnostic_item = "Vec"] #[rustc_insignificant_dtor] pub struct Vec<T, #[unstable(feature = "allocator_api", issue = "32838")] A: Allocator = Global> { buf: RawVec<T, A>, len: usize, } from: https://doc.rust-lang.org/1.89.0/src/alloc/raw_vec/mod.rs.html#74 #[allow(missing_debug_implementations)] pub(crate) struct RawVec<T, A: Allocator = Global> { inner: RawVecInner<A>, _marker: PhantomData<T>, } from: https://doc.rust-lang.org/1.89.0/src/alloc/raw_vec/mod.rs.html#86 #[allow(missing_debug_implementations)] struct RawVecInner<A: Allocator = Global> { ptr: Unique<u8>, /// Never used for ZSTs; it's `capacity()`'s responsibility to return usize::MAX in that case. /// /// # Safety /// /// `cap` must be in the `0..=isize::MAX` range. cap: Cap, alloc: A, } > Am I reading the patch correctly that the ivec implementation is primarily C? I’m not familiar with too many FFI projects in Rust, but I might have hoped we could write parts in Rust to gain any benefits from that, too. Is that a fool’s errand I’m thinking of? The ivec type is defined and implemented in C (interop/ivec.[ch]) and Rust (rust/interop/src/ivec.rs). When I started writing the ivec type I didn't know if the Git community would accept a hard dependency on Rust, so I made ivec usable in C without needing Rust.

On the Git mailing list, "D. Ben Knoble" wrote (reply to this):

On Mon, Aug 25, 2025 at 4:40 PM Ezekiel Newren <[email protected]> wrote: > > On Sun, Aug 24, 2025 at 7:31 AM Ben Knoble <[email protected]> wrote: > > I’m mildly surprised Vec isn’t a good fit: isn’t it a pointer, length, capacity triple? But it sounds like the main issue is allocator interop… which I would also have thought was supported? At least the current version is documented as being generic against an Allocator, too. > > Conceptually yes, semantically and syntactically no. On top of Vec<T> > not being defined with #[repr(C)] (which ensures field order, C ABI > layout, padding, etc...) the struct definition for Vec isn't constant > between Rust versions. I'd be open to suggestions for an alternative > to my ivec type. Ah, thanks—I had forgotten about the #[repr(C)] needs and changes. Makes sense. > > Am I reading the patch correctly that the ivec implementation is primarily C? I’m not familiar with too many FFI projects in Rust, but I might have hoped we could write parts in Rust to gain any benefits from that, too. Is that a fool’s errand I’m thinking of? > > The ivec type is defined and implemented in C (interop/ivec.[ch]) and > Rust (rust/interop/src/ivec.rs). When I started writing the ivec type > I didn't know if the Git community would accept a hard dependency on > Rust, so I made ivec usable in C without needing Rust. Right—I saw both implementations, but it looked like C did most of the work, which was my main question. Re-reading, it looks like Rust does more work than I thought (with implementations of insert/push/etc.) That said, I think it's sensible to leave the type useable from just C unless/until Rust becomes required (and then we can move things over). Thanks! -- D. Ben Knoble

On the Git mailing list, Ezekiel Newren wrote (reply to this):

On Tue, Aug 26, 2025 at 7:30 AM D. Ben Knoble <[email protected]> wrote: > > > Am I reading the patch correctly that the ivec implementation is primarily C? I’m not familiar with too many FFI projects in Rust, but I might have hoped we could write parts in Rust to gain any benefits from that, too. Is that a fool’s errand I’m thinking of? > > > > The ivec type is defined and implemented in C (interop/ivec.[ch]) and > > Rust (rust/interop/src/ivec.rs). When I started writing the ivec type > > I didn't know if the Git community would accept a hard dependency on > > Rust, so I made ivec usable in C without needing Rust. > > Right—I saw both implementations, but it looked like C did most of the > work, which was my main question. Re-reading, it looks like Rust does > more work than I thought (with implementations of insert/push/etc.) > > That said, I think it's sensible to leave the type useable from just C > unless/until Rust becomes required (and then we can move things over). I like your idea of implementation consolidation. I just don't know what that would look like yet. It's not straightforward because C doesn't have generics. I'll use IVec as an example, but this applies to any generic type in Rust. For a function like push() in IVec<T> it will have N definitions if there are N IVec types. e.g. If your code uses IVec<u64>, IVec<u8>, IVec<i32> that would mean that pub fn push(&mut self) {} would compile to 3 functions. If you don't use #[no_mangle] you'd have to figure out the Rust compiler's exact behavior for function names when calling it from C, which isn't stable or easily predictable. If you do use #[no_mangle] then the Rust compiler can't generate a generic function for each type. Another problem is that the functions in ivec mostly deal with resizing the memory rather than controlling access to memory for the C side. Even if the C side used Rust defined functions, that wouldn't solve memory access issues to the pointer on the C side. We could enforce access to each element by requiring C to call a Rust defined function for each element, but that sounds very painful and slow. ivec is meant to be used as a scaffolding type to help transition C to Rust. Other projects that do use Rust's builtin Vec (or some other collection type) often Box it and write wrapper functions. This means that the C side sees an opaque void* instead of a transparent struct like ivec with ptr, length, capacity, and element_size. I'm curious if the community has more design feedback, or suggestions for an alternative to my ivec type.

On the Git mailing list, "brian m. carlson" wrote (reply to this):

On 2025-08-26 at 18:47:06, Ezekiel Newren wrote: > I'm curious if the community has more design feedback, or suggestions > for an alternative to my ivec type. I think this is a fine approach. We used a similar Vec<u8>-like structure when porting a service from C to Rust at $DAYJOB, but we've also used the boxing approach to good success. Ultimately, I think boxing is the right choice when we don't need access from C. For instance, if I were to convert the loose object mapping code to Rust, I would store the data as a pair of Box<HashMap<_, _>> and then attach them to struct repository as `void *`. But it's less definitive when you need access from C. Because of the way we intimately mess with the internals of our data structures in this codebase, the ivec is probably the right choice here, so I'd keep that. -- brian m. carlson (they/them) Toronto, Ontario, CA

gitgitgadget-git · 2025-08-24T16:54:31Z

On the Git mailing list, Patrick Steinhardt wrote (reply to this):

On Mon, Aug 18, 2025 at 07:00:16PM -0700, Elijah Newren wrote:
> On Fri, Aug 15, 2025 at 8:10 AM Ramsay Jones
> <[email protected]> wrote:
> >
> > On 15/08/2025 02:22, Ezekiel Newren via GitGitGadget wrote:
> > > Changes in this second round of this RFC:
> > >
> > >  * Now builds and passes tests on all platforms (example run:
> > >    https://github.com/ezekielnewren/git/actions/runs/16974821401). Special
> > >    thanks to Johannes Schindelin for patches to things for Windows and
> > >    linux32.
> >
> > Hmm, builds on *all* platforms may be a bit optimistic (it doesn't on
> > cygwin, for instance), so I'm guessing you mean all platforms which
> > have CI defined. Perhaps you could mention the platforms which you
> > have tested on. :)
> 
> Ezekiel says this email didn't show up in his inbox (no idea why), but
> yes what was meant was all platforms where gitgitgadget CI runs.  If
> you follow the github.com link in the text that you quoted, you can
> see all those platforms (various windows flavors, various osx builds,
> musl, sparse, static analysis, etc.).

I do have some patches sitting around for a long while already that
implements CI via MSYS2 in different environments. It works with both
MSYS and MinGW, where I think they are somewhat related to Cygwin? If it
would prove useful I could maybe polish this patch series and send it
upstream.

Patrick

ezekielnewren · 2025-08-29T19:41:00Z

Closed in favor of #2043.

gitgitgadget-git · 2025-09-03T22:50:17Z

User Josh Steadmon <[email protected]> has been added to the cc: list.

gitgitgadget-git · 2025-09-23T17:52:41Z

User Jeff King <[email protected]> has been added to the cc: list.

modnmdiu · 2025-09-26T10:20:13Z

ج

gitgitgadget-git · 2025-10-05T05:42:05Z

User Yee Cheng Chin <[email protected]> has been added to the cc: list.

gitgitgadget-git bot added the new user label May 30, 2025

newren force-pushed the xdiff_rust_speedup branch 2 times, most recently from 8b3cc99 to 39211e9 Compare May 30, 2025 21:12

newren force-pushed the xdiff_rust_speedup branch from 39211e9 to 94ccfab Compare May 30, 2025 22:07

newren force-pushed the xdiff_rust_speedup branch from 94ccfab to d1acb81 Compare May 30, 2025 23:39

ezekielnewren force-pushed the xdiff_rust_speedup branch from d1acb81 to 9af7ba0 Compare May 31, 2025 01:22

ezekielnewren added 11 commits August 22, 2025 19:10

github workflows: upload Cargo.lock

e65488a

Make each ci workflow upload its Cargo.lock file as a build artifact so that we can audit build dependencies. Signed-off-by: Ezekiel Newren <[email protected]>

xdiff/xprepare: remove superfluous forward declarations

d4bed95

Move xdl_prepare_env() later in the file to avoid the need for forward declarations. Signed-off-by: Ezekiel Newren <[email protected]>

xdiff: delete unnecessary fields from xrecord_t and xdfile_t

7c68ce5

xrecord_t.next, xdfile_t.hbits, xdfile_t.rhash are initialized, but never used for anything by the code. Remove them. Signed-off-by: Ezekiel Newren <[email protected]>

xdiff: use one definition for freeing xdfile_t

21bfb9f

Simplify xdl_prepare_ctx() by using xdl_free_ctx() instead of using local variables with hand rolled memory management. Signed-off-by: Ezekiel Newren <[email protected]>

xdiff: delete nrec field from xdfile_t

0cfc6cf

Because of the data structure cleanup in the previous commit, the nrec field is no longer necessary. Use record.length in place of nrec. This commit is best viewed with --color-words. Signed-off-by: Ezekiel Newren <[email protected]>

xdiff: make xdfile_t more rust friendly

ea69913

Convert the remaining ambiguous fields in xdfile_t from C types to Rust types for interoperability. Signed-off-by: Ezekiel Newren <[email protected]>

xdiff: implement xdl_trim_ends() in Rust

b18544b

Replace the C implementation of xdl_trim_ends() with a Rust implementation. Signed-off-by: Ezekiel Newren <[email protected]>

ezekielnewren force-pushed the xdiff_rust_speedup branch from 395609a to b18544b Compare August 23, 2025 02:42

ezekielnewren changed the title ~~RFC: Accelerate xdiff and begin its rustification~~ RFC: Cleanup xdiff and begin its rustification Aug 23, 2025

gitgitgadget-git bot reviewed Aug 23, 2025

View reviewed changes

gitgitgadget-git bot reviewed Aug 24, 2025

View reviewed changes

ezekielnewren closed this Aug 29, 2025

ezekielnewren deleted the xdiff_rust_speedup branch September 23, 2025 22:17

RFC: Cleanup xdiff and begin its rustification #1980

RFC: Cleanup xdiff and begin its rustification #1980

Uh oh!

Conversation

ezekielnewren commented May 30, 2025 • edited by gitgitgadget-git bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gitgitgadget-git bot commented May 30, 2025

Welcome to GitGitGadget

Contributing the patches

Need help?

Uh oh!

gitgitgadget-git bot commented May 30, 2025

Uh oh!

gitgitgadget-git bot commented May 30, 2025

Uh oh!

gitgitgadget-git bot commented May 30, 2025

Uh oh!

gitgitgadget-git bot commented May 30, 2025

Uh oh!

gitgitgadget-git bot commented May 30, 2025

Uh oh!

gitgitgadget-git bot commented May 30, 2025

Uh oh!

gitgitgadget-git bot commented May 31, 2025

Uh oh!

dscho commented Jun 1, 2025

Uh oh!

gitgitgadget-git bot commented Jun 1, 2025

Uh oh!

gitgitgadget-git bot commented Jun 3, 2025

Uh oh!

gitgitgadget-git bot commented Jun 3, 2025

Uh oh!

gitgitgadget-git bot commented Jun 3, 2025

Uh oh!

gitgitgadget-git bot commented Jun 3, 2025

Uh oh!

gitgitgadget-git bot commented Jun 3, 2025

Uh oh!

gitgitgadget-git bot commented Jun 3, 2025

Uh oh!

gitgitgadget-git bot commented Jun 3, 2025

Uh oh!

gitgitgadget-git bot commented Jun 3, 2025

Uh oh!

gitgitgadget-git bot commented Jun 3, 2025

Uh oh!

gitgitgadget-git bot commented Jun 5, 2025

Uh oh!

gitgitgadget-git bot commented Jun 5, 2025

Uh oh!

gitgitgadget-git bot commented Jun 5, 2025

Uh oh!

gitgitgadget-git bot commented Jun 5, 2025

Uh oh!

gitgitgadget-git bot commented Jun 5, 2025

Uh oh!

ezekielnewren commented Aug 23, 2025

Uh oh!

gitgitgadget-git bot commented Aug 23, 2025

Uh oh!

gitgitgadget-git bot Aug 23, 2025

Choose a reason for hiding this comment

Uh oh!

gitgitgadget-git bot Aug 23, 2025

Choose a reason for hiding this comment

Uh oh!

gitgitgadget-git bot commented Aug 23, 2025

Uh oh!

gitgitgadget-git bot Aug 23, 2025

Choose a reason for hiding this comment

Uh oh!

gitgitgadget-git bot Aug 23, 2025

Choose a reason for hiding this comment

Uh oh!

gitgitgadget-git bot Aug 23, 2025

Choose a reason for hiding this comment

Uh oh!

gitgitgadget-git bot Aug 23, 2025

ezekielnewren commented May 30, 2025 •

edited by gitgitgadget-git bot

Loading