*: Initial support for opening sha256 repositories using `ObjectID` #1527

pjbgf · 2025-04-23T10:34:03Z

Introduces a new plumbing.ObjectID which supports either SHA1 and SHA256 object formats, removing the need of using build tags for the sha256 support.

Follow-up PRs will expand support for other git features. Please note that the sha256 is currently broken, as the creation of sha256 is yet to be implemented.

Relates to #899 #706.

aymanbagabas · 2025-04-23T12:30:18Z

plumbing/hash/hash.go

The plumbing/hash package feels redundant and can be replaced with Go's crypto and hash packages.

The only reason why plumbing.hash needs to exist currently is to support hash.RegisterHash within the explicit context of go-git. If we were to remove that, and lean on upstream's instead, then we would need to rethink on the UX.

I'm not sure we should force users to replace their sha1 implementation across their app, if they only want to use it in the context of go-git.
IIRC, upstream had a recent performance optimisations whereby they use both hashes: sha1cd when interacting with remote repositories, and sha1 for all local operations.

This feels like an orthogonal topic which could be dealt with as a separate issue/PR.

Sounds good

Actually, don't we already do that when sha1cd gets imported? It replaces upstream's sha1 in the registry.

You are absolutely right. That will need to change upstream depending on the approach we take.

I'll make a few tests around this, which may result in either the removal of plumbing/hash or the landing of the hybrid approach (if the performance gains justifies it).

Removal of the upstream behaviour will be handled separately, the same applies to the changes in go-git.

Perhaps we can remove plumbing/hash and combine the implementations of sha1 and sha256 hasher and object id. Also, instead of using interfaces for ObjectHasher, we can use a concrete type *ObjectHasher.

I've opened pjbgf#27 against your branch 🙂

Removal of the upstream behaviour will be handled separately, the same applies to the changes in go-git.

I think that's fine and is expected. Ideally, go-git shouldn't directly import crypto hashers to avoid overriding the user's registered hashers. Users should be able to use custom hashers using the crypto register and interface, while go-git could provide a default import for sha1cd and crypto/sha256.

Added your commits into the PR, PTAL.

There are some additional changes required around hashing and potentially the removal of our plumbing/hash, but I'd rather do that in a follow-up PR.

aymanbagabas · 2025-06-03T16:54:55Z

plumbing/hash_string.go

+
+// TODO: Compare and CompareBytes
+// Compare compares the hash's sum with a slice of bytes.
+func (s StringHash) Compare(b []byte) int {


Does this expect the hash bytes or the hash hex string?

aymanbagabas · 2025-06-03T20:58:19Z

plumbing/hash/hash.go

+const (
+	SHA1Size      = 20
+	SHA1HexSize   = SHA1Size * 2
+	SHA256Size    = 32
+	SHA256HexSize = SHA256Size * 2
+)


These are redefined here. We could use format.SHA1Size etc

This was part of the sha256 conditional build, which is no longer needed. All this references have now been removed.

Signed-off-by: Paulo Gomes <[email protected]>

The support for SHA256 will require the mapping between ObjectFormat and the correct hash.Hash to be used. Signed-off-by: Paulo Gomes <[email protected]>

The introduction of both new types are a stepping stone to enable SHA256 support concurrently - without the need for a build tag. ImmutableHash provides a way to represent varied sized hashes (for SHA1 or SHA256) with a single type, while keeping similar performance of the existing plumbing.Hash. The names were picked in order to provide a clearer distinction on what they do. ImmutableHash is the result of a hash operation and can't be changed once calculated. ObjectHasher, adds the Git object header before a normal hash operation that hash.Hash or plumbing/hash.Hash do. The performance results shows that SHA1 operations could be slower, however for SHA256 it can be over 3 times faster: ~Hasher/hasher-sha1-100B-16 2202226 574.7 ns/op 174.00 MB/s 272 B/op 7 allocs/op ~Hasher/objecthash-sha1-100B-16 1511851 772.6 ns/op 129.43 MB/s 272 B/op 9 allocs/op ~Hasher/objecthash-sha256-100B-16 5057584 247.4 ns/op 404.21 MB/s 96 B/op 7 allocs/op ~Hasher/hasher-sha1-5000B-16 96476 12523 ns/op 399.25 MB/s 5536 B/op 7 allocs/op ~Hasher/objecthash-sha1-5000B-16 105376 10983 ns/op 455.27 MB/s 272 B/op 9 allocs/op ~Hasher/objecthash-sha256-5000B-16 420741 2828 ns/op 1767.77 MB/s 96 B/op 7 allocs/op ~HashFromHex/hasher-parse-sha1-valid-16 23243548 64.65 ns/op 618.69 MB/s 48 B/op 1 allocs/op ~HashFromHex/objecthash-fromhex-sha1-valid-16 18120699 79.62 ns/op 502.36 MB/s 72 B/op 2 allocs/op ~HashFromHex/objecthash-fromhex-sha256-valid-16 8969871 124.2 ns/op 515.22 MB/s 96 B/op 2 allocs/op Signed-off-by: Paulo Gomes <[email protected]>

Signed-off-by: Paulo Gomes <[email protected]>

The binary.ReadHash is now replaced with ReadFrom on the ObjectID struct. Signed-off-by: Paulo Gomes <[email protected]>

With the addition of multi-hash support, there is no longer need to have a sha256 specific test that relies on build tags. Signed-off-by: Paulo Gomes <[email protected]>

The new ObjectID replaces plumbing.Hash and supports both SHA1 and SHA256. The backing array is sized after SHA256, resulting in a larger memory footprint on a per hash basis. Signed-off-by: Paulo Gomes <[email protected]>

Signed-off-by: Paulo Gomes <[email protected]>

The previous example of Blame, started from a clone, instead of opening an existing repository. Signed-off-by: Paulo Gomes <[email protected]>

Signed-off-by: Paulo Gomes <[email protected]>

ObjectID differs from the previous Hash as it has a new format field. For the purposes for equality checking, both values 'sha1' and '' should be perceived as the same, so to avoid cases where the equality check is not being done internally, the value 'sha1' is no longer used. The field format should only be populated when its value is 'sha256'. Signed-off-by: Paulo Gomes <[email protected]>

Signed-off-by: Paulo Gomes <[email protected]>

This commit changes the ObjectFormat type from a string to an int. This would make a default ObjectFormat declaration default to SHA1. A string function is defined to return the string representation of the ObjectFormat.

This combines the ObjectHasher implementation to be hash function agnostic. The SHA1 and SHA256 implementations are now combined into a single implementation that uses the given format to determine the hash function to use.

Signed-off-by: Paulo Gomes <[email protected]>

Users can opt-out of sha1cd by adding an empty import for the upstream implementation: _ "crypto/sha1" Signed-off-by: Paulo Gomes <[email protected]>

Signed-off-by: Paulo Gomes <[email protected]>

aymanbagabas · 2025-06-03T23:12:39Z

plumbing/format/commitgraph/encoder.go

 // NewEncoder returns a new stream encoder that writes to w.
 func NewEncoder(w io.Writer) *Encoder {
-	h := hash.New(hash.CryptoType)
+	h := hash.New(crypto.SHA1)


Let's put a todo here to add support for format object.

Suggested change

h := hash.New(crypto.SHA1)

// TODO: Support passing an ObjectFormat instead of hardcoding SHA1

h := hash.New(crypto.SHA1)

aymanbagabas · 2025-06-03T23:13:38Z

plumbing/format/commitgraph/encoder.go

 	if _, err = e.Write(commitFileSignature); err == nil {
 		version := byte(1)
-		if hash.CryptoType == crypto.SHA256 {
+		if crypto.Hash(e.hash.Size()) == crypto.Hash(crypto.SHA256.Size()) {


Suggested change

if crypto.Hash(e.hash.Size()) == crypto.Hash(crypto.SHA256.Size()) {

if e.hash.Size() == crypto.SHA256.Size() {

aymanbagabas · 2025-06-03T23:15:45Z

plumbing/format/idxfile/encoder.go

 // NewEncoder returns a new stream encoder that writes to w.
 func NewEncoder(w io.Writer) *Encoder {
-	h := hash.New(hash.CryptoType)
+	h := hash.New(crypto.SHA1)


Suggested change

h := hash.New(crypto.SHA1)

// TODO: Support passing ObjectFormat instead of hardcoding SHA1

h := hash.New(crypto.SHA1)

aymanbagabas · 2025-06-03T23:17:25Z

plumbing/format/index/decoder.go

 // NewDecoder returns a new decoder that reads from r.
 func NewDecoder(r io.Reader) *Decoder {
-	h := hash.New(hash.CryptoType)
+	h := hash.New(crypto.SHA1)


Suggested change

h := hash.New(crypto.SHA1)

// TODO: Support passing ObjectFormat instead of hardcoding SHA1

h := hash.New(crypto.SHA1)

aymanbagabas · 2025-06-03T23:18:16Z

plumbing/format/index/encoder.go

 // NewEncoder returns a new encoder that writes to w.
 func NewEncoder(w io.Writer) *Encoder {
-	h := hash.New(hash.CryptoType)
+	h := hash.New(crypto.SHA1)


Suggested change

h := hash.New(crypto.SHA1)

// TODO: Support passing ObjectFormat instead of hardcoding SHA1

h := hash.New(crypto.SHA1)

aymanbagabas · 2025-06-03T23:19:04Z

plumbing/format/packfile/encoder.go

 func NewEncoder(w io.Writer, s storer.EncodedObjectStorer, useRefDeltas bool) *Encoder {
 	h := plumbing.Hasher{
-		Hash: hash.New(hash.CryptoType),
+		Hash: hash.New(crypto.SHA1),


Suggested change

Hash: hash.New(crypto.SHA1),

// TODO: Support SHA256

Hash: hash.New(crypto.SHA1),

aymanbagabas · 2025-06-03T23:20:19Z

plumbing/format/packfile/scanner.go

 	dict := make([]byte, 16*1024)
 	crc := crc32.NewIEEE()
-	packhash := gogithash.New(gogithash.CryptoType)
+	packhash := gogithash.New(crypto.SHA1)


Suggested change

packhash := gogithash.New(crypto.SHA1)

// TODO: Support SHA256

packhash := gogithash.New(crypto.SHA1)

Signed-off-by: Paulo Gomes <[email protected]>

pjbgf added this to the v6.0.0 milestone Apr 23, 2025

aymanbagabas reviewed Apr 23, 2025

View reviewed changes

pjbgf marked this pull request as ready for review April 26, 2025 12:59

pjbgf force-pushed the sha256-open branch from ee62990 to 0f463ba Compare April 27, 2025 06:48

pjbgf requested a review from aymanbagabas April 27, 2025 07:06

pjbgf force-pushed the sha256-open branch from 0f463ba to 3149478 Compare June 3, 2025 16:47

aymanbagabas reviewed Jun 3, 2025

View reviewed changes

pjbgf force-pushed the sha256-open branch from 3149478 to 19fa376 Compare June 3, 2025 20:26

aymanbagabas reviewed Jun 3, 2025

View reviewed changes

This was referenced Jun 3, 2025

Support for SHA256 #706

Open

Support SHA-1 **and** SHA-256 at the same time #899

Closed

pjbgf and others added 19 commits June 3, 2025 23:56

plumbing: format/config, Add ErrInvalidObjectFormat

a4f298e

Signed-off-by: Paulo Gomes <[email protected]>

plumbing: hash, Add SHA1 and SHA256 constants

a28c4d4

Signed-off-by: Paulo Gomes <[email protected]>

plumbing: hash, Add ErrUnsupportedHashFunction

3ccc874

Signed-off-by: Paulo Gomes <[email protected]>

plumbing: hash, Add hash.FromObjectFormat

f3411df

The support for SHA256 will require the mapping between ObjectFormat and the correct hash.Hash to be used. Signed-off-by: Paulo Gomes <[email protected]>

plumbing: Rename hash constants

f7bcbfa

Signed-off-by: Paulo Gomes <[email protected]>

plumbing: hash, Move immutable hash to plumbing

ce37c5b

Signed-off-by: Paulo Gomes <[email protected]>

utils: Support new Hash format

27e3866

Signed-off-by: Paulo Gomes <[email protected]>

plumbing: protocol, Support new Hash format

5e3685e

Signed-off-by: Paulo Gomes <[email protected]>

storage: filesystem, Support new Hash format

500bbf1

Signed-off-by: Paulo Gomes <[email protected]>

plumbing: format/packfile, Support new Hash format

7de3e20

Signed-off-by: Paulo Gomes <[email protected]>

utils: binary, Remove ReadHash

94880d2

The binary.ReadHash is now replaced with ReadFrom on the ObjectID struct. Signed-off-by: Paulo Gomes <[email protected]>

build: Remove sha256-specific test

ee7e345

With the addition of multi-hash support, there is no longer need to have a sha256 specific test that relies on build tags. Signed-off-by: Paulo Gomes <[email protected]>

plumbing: Add ObjectID to replace Hash

ed15d0b

The new ObjectID replaces plumbing.Hash and supports both SHA1 and SHA256. The backing array is sized after SHA256, resulting in a larger memory footprint on a per hash basis. Signed-off-by: Paulo Gomes <[email protected]>

plumbing: format, Support new Hash format

cc609e4

Signed-off-by: Paulo Gomes <[email protected]>

plumbing: object, Support new Hash format

fcad8fb

Signed-off-by: Paulo Gomes <[email protected]>

plumbing: protocol, Support new Hash format

fa3fb77

Signed-off-by: Paulo Gomes <[email protected]>

plumbing: hash, Bump module to v6

c1790f5

Signed-off-by: Paulo Gomes <[email protected]>

_examples: Change blame example to open repo

665119d

The previous example of Blame, started from a clone, instead of opening an existing repository. Signed-off-by: Paulo Gomes <[email protected]>

pjbgf and others added 14 commits June 3, 2025 23:56

plumbing: transport/http, Set object size for NewMemoryIndex()

96a80f6

Signed-off-by: Paulo Gomes <[email protected]>

git: Support new Hash format

619cabe

Signed-off-by: Paulo Gomes <[email protected]>

*: Simplify ObjectID creation

93d9045

Signed-off-by: Paulo Gomes <[email protected]>

plumbing: format/commitgraph, Fix issues with ObjectID

cbb802c

Signed-off-by: Paulo Gomes <[email protected]>

plumbing: config, use an int for ObjectFormat

e9cea3b

This commit changes the ObjectFormat type from a string to an int. This would make a default ObjectFormat declaration default to SHA1. A string function is defined to return the string representation of the ObjectFormat.

plumbing: combine hasher implementation based on object format

db63721

This combines the ObjectHasher implementation to be hash function agnostic. The SHA1 and SHA256 implementations are now combined into a single implementation that uses the given format to determine the hash function to use.

plumbing: remove unused HexSize

49ace15

git: use standard library hashes for client side crypto

d677269

plumbing: format/packfile, Align hash formats

67b3c10

Signed-off-by: Paulo Gomes <[email protected]>

*: Set sha1cd as default implementation

6bc2df1

Users can opt-out of sha1cd by adding an empty import for the upstream implementation: _ "crypto/sha1" Signed-off-by: Paulo Gomes <[email protected]>

plumbing: format/commitgraph, Refactor tests

af89cec

Signed-off-by: Paulo Gomes <[email protected]>

plumbing: Improve ObjectID documentation

7dfd841

Signed-off-by: Paulo Gomes <[email protected]>

plumbing: hash, Remove references to sha256 conditional build

8ed10b1

Signed-off-by: Paulo Gomes <[email protected]>

pjbgf force-pushed the sha256-open branch from 57af247 to 8ed10b1 Compare June 3, 2025 23:04

aymanbagabas reviewed Jun 3, 2025

View reviewed changes

plumbing: format, Highlight API changes needed for sha256 support

6d619b3

Signed-off-by: Paulo Gomes <[email protected]>

aymanbagabas approved these changes Jun 4, 2025

View reviewed changes

pjbgf merged commit 856aa92 into go-git:v6-transport Jun 4, 2025
14 checks passed

pjbgf deleted the sha256-open branch June 4, 2025 18:05

	h := hash.New(crypto.SHA1)
	// TODO: Support passing an ObjectFormat instead of hardcoding SHA1
	h := hash.New(crypto.SHA1)

	if crypto.Hash(e.hash.Size()) == crypto.Hash(crypto.SHA256.Size()) {
	if e.hash.Size() == crypto.SHA256.Size() {

	h := hash.New(crypto.SHA1)
	// TODO: Support passing ObjectFormat instead of hardcoding SHA1
	h := hash.New(crypto.SHA1)

	Hash: hash.New(crypto.SHA1),
	// TODO: Support SHA256
	Hash: hash.New(crypto.SHA1),

	packhash := gogithash.New(crypto.SHA1)
	// TODO: Support SHA256
	packhash := gogithash.New(crypto.SHA1)

*: Initial support for opening sha256 repositories using ObjectID #1527

*: Initial support for opening sha256 repositories using ObjectID #1527

Uh oh!

Conversation

pjbgf commented Apr 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aymanbagabas Apr 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

*: Initial support for opening sha256 repositories using `ObjectID` #1527

*: Initial support for opening sha256 repositories using `ObjectID` #1527

pjbgf commented Apr 23, 2025 •

edited

Loading

aymanbagabas Apr 23, 2025 •

edited

Loading