Skip to content

Commit 3ef8327

Browse files
ken2812221ken2812221
committed
Use python instead of slow shell script on verify-commits
1 parent e24bf1c commit 3ef8327

File tree

7 files changed

+145
-161
lines changed

7 files changed

+145
-161
lines changed

‎.travis.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -104,5 +104,5 @@ jobs:
104104
- test/lint/lint-all.sh
105105
- if [ "$TRAVIS_REPO_SLUG" = "bitcoin/bitcoin" -a "$TRAVIS_EVENT_TYPE" = "cron" ]; then
106106
while read LINE; do travis_retry gpg --keyserver hkp://subset.pool.sks-keyservers.net --recv-keys $LINE; done < contrib/verify-commits/trusted-keys &&
107-
travis_wait 30 contrib/verify-commits/verify-commits.sh;
107+
travis_wait 30 contrib/verify-commits/verify-commits.py;
108108
fi

‎contrib/verify-commits/README.md‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -7,18 +7,18 @@ are PGP signed (nearly always merge commits), as well as a script to verify
77
commits against a trusted keys list.
88

99

10-
Using verify-commits.sh safely
10+
Using verify-commits.py safely
1111
------------------------------
1212

1313
Remember that you can't use an untrusted script to verify itself. This means
14-
that checking out code, then running `verify-commits.sh` against `HEAD` is
15-
_not_ safe, because the version of `verify-commits.sh` that you just ran could
14+
that checking out code, then running `verify-commits.py` against `HEAD` is
15+
_not_ safe, because the version of `verify-commits.py` that you just ran could
1616
be backdoored. Instead, you need to use a trusted version of verify-commits
1717
prior to checkout to make sure you're checking out only code signed by trusted
1818
keys:
1919

2020
git fetch origin && \
21-
./contrib/verify-commits/verify-commits.sh origin/master && \
21+
./contrib/verify-commits/verify-commits.py origin/master && \
2222
git checkout origin/master
2323

2424
Note that the above isn't a good UI/UX yet, and needs significant improvements
@@ -42,6 +42,6 @@ said key. In order to avoid bumping the root-of-trust `trusted-git-root`
4242
file, individual commits which were signed by such a key can be added to the
4343
`allow-revsig-commits` file. That way, the PGP signatures are still verified
4444
but no new commits can be signed by any expired/revoked key. To easily build a
45-
list of commits which need to be added, verify-commits.sh can be edited to test
45+
list of commits which need to be added, verify-commits.py can be edited to test
4646
each commit with BITCOIN_VERIFY_COMMITS_ALLOW_REVSIG set to both 1 and 0, and
4747
those which need it set to 1 printed.

‎contrib/verify-commits/allow-incorrect-sha512-commits‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
f8feaa4636260b599294c7285bcf1c8b7737f74e
2+
8040ae6fc576e9504186f2ae3ff2c8125de1095c

‎contrib/verify-commits/allow-unclean-merge-commits‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
6052d509105790a26b3ad5df43dd61e7f1b24a12
2+
3798e5de334c3deb5f71302b782f6b8fbd5087f1
3+
326ffed09bfcc209a2efd6a2ebc69edf6bd200b5
4+
97d83739db0631be5d4ba86af3616014652c00ec

‎contrib/verify-commits/pre-push-hook.sh‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -12,9 +12,9 @@ while read LINE; do
1212
if [ "$4" != "refs/heads/master" ]; then
1313
continue
1414
fi
15-
if ! ./contrib/verify-commits/verify-commits.sh $3 > /dev/null 2>&1; then
15+
if ! ./contrib/verify-commits/verify-commits.py $3 > /dev/null 2>&1; then
1616
echo "ERROR: A commit is not signed, can't push"
17-
./contrib/verify-commits/verify-commits.sh
17+
./contrib/verify-commits/verify-commits.py
1818
exit 1
1919
fi
2020
done < /dev/stdin

‎contrib/verify-commits/verify-commits.py‎

Lines changed: 131 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,131 @@
1+
#!/usr/bin/env python3
2+
import argparse
3+
import hashlib
4+
import os
5+
import subprocess
6+
import sys
7+
import time
8+
from subprocess import PIPE
9+
from sys import stderr
10+
11+
GIT = os.getenv('GIT','git')
12+
13+
def tree_sha512sum(commit='HEAD'):
14+
# request metadata for entire tree, recursively
15+
files = []
16+
blob_by_name = {}
17+
for line in subprocess.check_output([GIT, 'ls-tree', '--full-tree', '-r', commit]).splitlines():
18+
name_sep = line.index(b'\t')
19+
metadata = line[:name_sep].split() # perms, 'blob', blobid
20+
assert(metadata[1] == b'blob')
21+
name = line[name_sep+1:]
22+
files.append(name)
23+
blob_by_name[name] = metadata[2]
24+
25+
files.sort()
26+
# open connection to git-cat-file in batch mode to request data for all blobs
27+
# this is much faster than launching it per file
28+
p = subprocess.Popen([GIT, 'cat-file', '--batch'], stdout=PIPE, stdin=PIPE)
29+
overall = hashlib.sha512()
30+
for f in files:
31+
blob = blob_by_name[f]
32+
# request blob
33+
p.stdin.write(blob + b'\n')
34+
p.stdin.flush()
35+
# read header: blob, "blob", size
36+
reply = p.stdout.readline().split()
37+
assert(reply[0] == blob and reply[1] == b'blob')
38+
size = int(reply[2])
39+
# hash the blob data
40+
intern = hashlib.sha512()
41+
ptr = 0
42+
while ptr < size:
43+
bs = min(65536, size - ptr)
44+
piece = p.stdout.read(bs)
45+
if len(piece) == bs:
46+
intern.update(piece)
47+
else:
48+
raise IOError('Premature EOF reading git cat-file output')
49+
ptr += bs
50+
dig = intern.hexdigest()
51+
assert(p.stdout.read(1) == b'\n') # ignore LF that follows blob data
52+
# update overall hash with file hash
53+
overall.update(dig.encode("utf-8"))
54+
overall.update(" ".encode("utf-8"))
55+
overall.update(f)
56+
overall.update("\n".encode("utf-8"))
57+
p.stdin.close()
58+
if p.wait():
59+
raise IOError('Non-zero return value executing git cat-file')
60+
return overall.hexdigest()
61+
62+
def main():
63+
# get directory of this program
64+
dirname = os.path.dirname(os.path.abspath(__file__))
65+
parser = argparse.ArgumentParser(usage='%(prog)s [options] [commit id]')
66+
parser.add_argument('--disable-tree-check', action='store_false', dest='verify_tree', help='disable SHA-512 tree check')
67+
parser.add_argument('--clean-merge', type=float, dest='clean_merge', default=float('inf'), help='Only check clean merge after <NUMBER> days ago (default: %(default)s)', metavar='NUMBER')
68+
(args, commit) = parser.parse_known_args()
69+
print("Using verify-commits data from " + dirname)
70+
verified_root = open(dirname + "/trusted-git-root", "r").read().splitlines()[0]
71+
verified_sha512_root = open(dirname + "/trusted-sha512-root-commit", "r").read().splitlines()[0]
72+
revsig_allowed = open(dirname + "/allow-revsig-commits", "r").read().splitlines()
73+
unclean_merge_allowed = open(dirname + "/allow-unclean-merge-commits", "r").read().splitlines()
74+
incorrect_sha512_allowed = open(dirname + "/allow-incorrect-sha512-commits", "r").read().splitlines()
75+
current_commit = "HEAD" if len(commit) == 0 else commit[0]
76+
if ' ' in current_commit:
77+
print("Commit must not contain spaces?", file=sys.stderr)
78+
exit(1)
79+
verify_tree = args.verify_tree
80+
no_sha1 = True
81+
prev_commit = ""
82+
initial_commit = current_commit
83+
branch = subprocess.check_output([GIT,'show','-s','--format=%H',initial_commit], universal_newlines=True).splitlines()[0]
84+
while True:
85+
if current_commit == verified_root:
86+
print('There is a valid path from "' + initial_commit + '" to ' + verified_root + ' where all commits are signed!')
87+
exit(0)
88+
if current_commit == verified_sha512_root:
89+
if verify_tree:
90+
print("All Tree-SHA512s matched up to " + verified_sha512_root, file=stderr)
91+
verify_tree = False
92+
no_sha1 = False
93+
os.environ['BITCOIN_VERIFY_COMMITS_ALLOW_SHA1'] = "0" if no_sha1 else "1"
94+
os.environ['BITCOIN_VERIFY_COMMITS_ALLOW_REVSIG'] = "1" if current_commit in revsig_allowed else "0"
95+
if subprocess.call([GIT, '-c', 'gpg.program=' + dirname + '/gpg.sh', 'verify-commit', current_commit], stdout=subprocess.DEVNULL):
96+
if prev_commit != "":
97+
print("No parent of " + prev_commit + " was signed with a trusted key!", file=sys.stderr)
98+
print("Parents are:", file=sys.stderr)
99+
parents = subprocess.check_output([GIT, 'show', '-s', '--format=format:%P', prev_commit], universal_newlines=True).splitlines()[0].split(' ')
100+
for parent in parents:
101+
subprocess.call([GIT, 'show', '-s', parent], stdout=stderr)
102+
else:
103+
print(current_commit + " was not signed with a trusted key!", file=stderr)
104+
exit(1)
105+
if (verify_tree or prev_commit == "") and current_commit not in incorrect_sha512_allowed:
106+
tree_hash = tree_sha512sum(current_commit)
107+
if ("Tree-SHA512: "+tree_hash) not in subprocess.check_output([GIT, 'show', '-s', '--format=format:%B', current_commit], universal_newlines=True).splitlines():
108+
print("Tree-SHA512 did not match for commit " + current_commit, file=stderr)
109+
exit(1)
110+
parents = subprocess.check_output([GIT, 'show', '-s', '--format=format:%P', current_commit], universal_newlines=True).splitlines()[0].split(' ')
111+
commit_time = int(subprocess.check_output([GIT, 'show', '-s', '--format=format:%ct', current_commit], universal_newlines=True).splitlines()[0])
112+
check_merge = commit_time > time.time() - args.clean_merge * 24 * 60 * 60 # Only check commits in clean_merge days
113+
allow_unclean = current_commit in unclean_merge_allowed
114+
if len(parents) > 2:
115+
print("Commit " + current_commit + "is an octopus merge", file=stderr)
116+
exit(1)
117+
if len(parents) == 2 and check_merge and not allow_unclean:
118+
CURRENT_TREE=subprocess.check_output([GIT, 'show', '--format=%T', current_commit], universal_newlines=True).splitlines()[0]
119+
subprocess.call([GIT, 'checkout', '--force', '--quiet', parents[0]])
120+
subprocess.call([GIT, 'merge', '--no-ff', '--quiet', parents[1]], stdout=subprocess.DEVNULL)
121+
RECREATED_TREE = subprocess.check_output([GIT, 'show', '--format=format:%T', 'HEAD'], universal_newlines=True).splitlines()[0]
122+
if CURRENT_TREE != RECREATED_TREE:
123+
print("Merge commit " + current_commit + " is not clean", file=stderr)
124+
subprocess.call([GIT, 'diff', current_commit])
125+
subprocess.call([GIT, 'checkout', '--force', '--quiet', branch])
126+
exit(1)
127+
subprocess.call([GIT, 'checkout', '--force', '--quiet', branch])
128+
prev_commit = current_commit
129+
current_commit = parents[0]
130+
if __name__ == '__main__':
131+
main()

‎contrib/verify-commits/verify-commits.sh‎

Lines changed: 0 additions & 153 deletions
This file was deleted.

0 commit comments

Comments
 (0)