The conformance testsuite currently relies on a GitHub Actions OIDC credential, obtained by asking for the id-token: write permission.
Unfortunately (but understandably), this permission can't be given to a repository's (untrusted) third-party forks: forks can only receive id-token: read, which doesn't allow access to the OIDC credential.
As such, the conformance testsuite currently fails when it's triggered by a PR coming from a third-party fork:
=================================== FAILURES ===================================
_______________________________ test_sign_verify _______________________________
client = <test.client.SigstoreClient object at 0x7f6153166310>
def test_sign_verify(client: SigstoreClient) -> None:
"""
A basic test that signs and verifies an artifact for a given Sigstore
client.
"""
artifact_path = Path("artifact.txt")
signature_path = Path("artifact.txt.sig")
certificate_path = Path("artifact.txt.crt")
assert artifact_path.exists()
assert not signature_path.exists()
assert not certificate_path.exists()
# Sign the artifact.
> client.sign(artifact_path, signature_path, certificate_path)
/home/runner/work/_actions/trailofbits/sigstore-conformance/4b6b3c2877f7fd629d33c654b7[51](https://github.com/sigstore/sigstore-python/actions/runs/3649487017/jobs/6164227325#step:5:54)43ecfcc68e2b/test/test_sign_verify.py:20:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/home/runner/work/_actions/trailofbits/sigstore-conformance/4b6b3c2877f7fd629d33c654b75143ecfcc68e2b/test/client.py:[53](https://github.com/sigstore/sigstore-python/actions/runs/3649487017/jobs/6164227325#step:5:56): in sign
self.run(
/home/runner/work/_actions/trailofbits/sigstore-conformance/4b6b3c2877f7fd629d33c6[54](https://github.com/sigstore/sigstore-python/actions/runs/3649487017/jobs/6164227325#step:5:57)b75143ecfcc[68](https://github.com/sigstore/sigstore-python/actions/runs/3649487017/jobs/6164227325#step:5:71)e2b/test/client.py:35: in run
subprocess.run(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
input = None, capture_output = False, timeout = None, check = True
popenargs = (['/home/runner/work/sigstore-python/sigstore-python/test/integration/sigstore-python-conformance', 'sign', '--signature', PosixPath('artifact.txt.sig'), '--certificate', PosixPath('artifact.txt.crt'), ...],)
kwargs = {'stderr': -2, 'stdout': -1, 'text': True}
process = <Popen: returncode: 1 args: ['/home/runner/work/sigstore-python/sigstore-pyt...>
stdout = "\nInsufficient permissions for GitHub Actions workflow.\n\nThe most common reason for this is incorrect\nconfiguratio...gnature', 'artifact.txt.sig', '--certificate', 'artifact.txt.crt', 'artifact.txt']' returned non-zero exit status 1.\n"
stderr = None, retcode = 1Logs: https://github.com/sigstore/sigstore-python/actions/runs/3649487017/jobs/6164227325
Consumers of the test suite can hack around this by disabling the testsuite on third-party forks, but that isn't ideal in the long term (since it means that third-party contributors are receiving less test coverage than first-party ones).
Instead, we should figure out a secure way to share an OIDC credential with the conformance testsuite, even when it's coming from an untrusted fork. One possible way to do that would be to leverage the fact that the conformance suite should always be pinned to a hash that we know ahead of time.
The conformance testsuite currently relies on a GitHub Actions OIDC credential, obtained by asking for the
id-token: writepermission.Unfortunately (but understandably), this permission can't be given to a repository's (untrusted) third-party forks: forks can only receive
id-token: read, which doesn't allow access to the OIDC credential.As such, the conformance testsuite currently fails when it's triggered by a PR coming from a third-party fork:
Logs: https://github.com/sigstore/sigstore-python/actions/runs/3649487017/jobs/6164227325
Consumers of the test suite can hack around this by disabling the testsuite on third-party forks, but that isn't ideal in the long term (since it means that third-party contributors are receiving less test coverage than first-party ones).
Instead, we should figure out a secure way to share an OIDC credential with the conformance testsuite, even when it's coming from an untrusted fork. One possible way to do that would be to leverage the fact that the conformance suite should always be pinned to a hash that we know ahead of time.