PR #9435 (commit ddd16c2) made changes to DH_check() in the 1.1.1 branch. This seems to have caused Travis to fail on the pyca cryptography external tests.

After some digging I've managed to create a standalone C test program that replicates the python test and demonstrates the change in behaviour before and after ddd16c2.

#include <stdio.h>
#include <openssl/bn.h>
#include <openssl/dh.h>

int main(void)
{
    DH *dh = DH_new();
    BIGNUM *p, *g;
    int ret = 1;
    int errflags = 0;

    p = BN_new();
    g = BN_new();
    if (dh == NULL || p == NULL || g == NULL)
        goto err;

    if (!BN_set_word(p, 23) || !BN_set_word(g, 18))
        goto err;

    if (!DH_set0_pqg(dh, p, NULL, g))
        goto err;
    p = g = NULL;

    if (!DH_check(dh, &errflags))
        goto err;

    /* We expect errflags to be non-zero */
    if (errflags == 0)
        goto err;

    ret = 0;

 err:
    BN_free(p);
    BN_free(g);
    DH_free(dh);

    if (ret == 0)
        printf("Success\n");
    else
        printf("Failure\n");
    return ret;
}

Prior to the change this test program will print "Success", and after it, it prints "Failure".

It seems that before this change the errflags were being set to DH_UNABLE_TO_CHECK_GENERATOR, but are now not set to anything. The PR in question seems to have removed the generator checks completely. Why is this?

Possibly the pyca test needs to change because it assumes something about the internal implementation of DH_check() which is no longer true (i.e. that it is unable to check certain values of generator).