PROBLEM:

xbps signature files mix data from 2 different standards. As a result, standard tools (such as openssl) cannot be used to check the authenticity of packages or create signature files for custom packages.

STEPS TO REPRODUCE:

1-Retrieve xbps public key, convert to standard format:

grep -Pom1 '(?<=<data>).*(?=</data>)' /var/db/xbps/keys/<PLIST> \
  | base64 -d > public.pem

2-Check authenticity of package with openssl:

openssl dgst -verify public.pem -signature <PKG>.sig <PKG>
Verification Failure

PROBLEM ANALYSIS:

1-Examine ASN.1 data in signature file:

openssl pkeyutl -verifyrecover -in <PKG>.sig -inkey public.pem -pubin -asn1parse

    0:d=0  hl=2 l=  45 cons: SEQUENCE
    2:d=1  hl=2 l=   9 cons:  SEQUENCE
    4:d=2  hl=2 l=   5 prim:   OBJECT            :sha1
   11:d=2  hl=2 l=   0 prim:   NULL
   13:d=1  hl=2 l=  32 prim:  OCTET STRING
      0000 - 30 d6 f3 0d 04 43 45 0f-1a f8 0a 88 7f a6 4c b6   0....CE.......L.
      0010 - 74 61 3b a9 38 4f 38 d2-f0 dd 2b 4f 7f 40 5a 5b   ta;.8O8...+O.@Z[

Note the conflicting data: SHA1 hash per DigestInfo field, but a 32-byte hash (instead of a 20-byte hash for SHA1).

2-Dump signature data for a more detailed look:

openssl pkeyutl -verifyrecover -in <PKG>.sig -inkey public.pem -pubin -hexdump

0000 - 30 2d 30 09 06 05 2b 0e-03 02 1a 05 00 04 20 30   0-0...+....... 0
0010 - d6 f3 0d 04 43 45 0f 1a-f8 0a 88 7f a6 4c b6 74   ....CE.......L.t
0020 - 61 3b a9 38 4f 38 d2 f0-dd 2b 4f 7f 40 5a 5b      a;.8O8...+O.@Z[

The DigestInfo data is not even completely correct for SHA1. Specifically, the second byte and last byte of the 15-byte prefix do not match the PKCS#1 v1.5 signature standard for SHA1 (see Notes in RFC 8017, page 46). So not only is the DigestInfo for the wrong hash being used, the DigestInfo is not even standard.

In order to produce a standard signature, the 19-byte DigestInfo for SHA256 should be combined with the SHA256 hash.

CODE CHANGE NOTES:

1-One of the culprits is this line (currently 66) in the xbps/lib/verifysig.c source file:

rv = RSA_verify(NID_sha1, sha256, SHA256_DIGEST_LENGTH, sig, siglen, rsa);

Should be NID_sha256.

2-The program used to generate signature files would of course also need to be fixed.
3-Unknown if other code would be affected too.
4-An appropriate time to release the fix would be when a new set of keys are generated for xbps package signing.