MAYO signing function returns MAYO_OK after 256 failed sample_solution() attempts #2392

Description

Hi, the MAYO signing function (mayo_sign_signature) initializes ret = MAYO_OK at line 362, then enters a 256-iteration rejection sampling loop calling sample_solution(). If none of the 256 attempts find a valid solution, execution falls through to the signature assembly code which runs unconditionally:

int ret = MAYO_OK;                                    // line 362
// ...
for (int ctr = 0; ctr <= 255; ++ctr) {
    // ...
    if (sample_solution(p, A, y, r, x, ...)) {
        break;                                         // found valid solution
    } else {
        memset(Mtmp, 0, sizeof(Mtmp));
        memset(A, 0, sizeof(A));
    }
}
// No check whether the loop succeeded — falls through regardless

for (int i = 0; i <= param_k - 1; ++i) {             // line 477
    // assembles signature from x (garbage if loop failed)
    mat_mul(sk.O, x + i * param_o, Ox, ...);
    mat_add(vi, Ox, s + i * param_n, ...);
}
encode(s, sig, param_n * param_k);
// ...
return ret;                                            // still MAYO_OK

ret is never updated on loop exhaustion. The liboqs wrapper translates MAYO_OK to OQS_SUCCESS, so the caller has no way to know the signature is garbage.

This affects all 12 MAYO variants in src/sig/mayo/pqmayo_mayo-{1,2,3,5}_{opt,avx2,aarch64}/mayo.c. The upstream source is https://github.com/PQCMayo/MAYO-C.

Expected behaviour

When all 256 sample_solution() attempts fail, the function should return an error (MAYO_ERR) instead of assembling a signature from invalid data and returning MAYO_OK.

A simple fix:

int found = 0;
for (int ctr = 0; ctr <= 255; ++ctr) {
    if (sample_solution(...)) { found = 1; break; }
    // ...
}
if (!found) { ret = MAYO_ERR; goto err; }

liboqs version

0.15.0 (commit ab0e07c)

Environment

• Architecture: x86_64
• OS: Ubuntu 24.04 LTS
• Found by source code analysis, not runtime testing

Additional information

With valid keys, this is practically unreachable — MAYO targets < 2^{-128} failure probability per signing attempt. However, a cryptographic signing function should never silently return success with invalid output. With malformed keys, all 256 attempts can fail.

This should also be reported upstream to https://github.com/PQCMayo/MAYO-C since liboqs vendors this code.