I guess this is perfectly fine for now.

I only want to raise the issue of thinking about a future when instead of only PureEdDSA OpenSSL might support also HashEdDSA: would this change have an unintended impact on configuration patterns or command line arguments that might be difficult to break in future versions?

I honestly don't know the answer, nor if such issues would concern the guarantees in minor or major releases.

I only want to raise the issue of thinking about a future when instead of only PureEdDSA OpenSSL might support also HashEdDSA: would this change have an unintended impact on configuration patterns or command line arguments that might be difficult to break in future versions?

I did consider this, but I don't think it is an issue. The way it has been implemented the ca app asks the EVP_PKEY object what its default hash is. Only if the default hash is undefined is it allowed to not have a hash specified. If we later implement hash EdDSA then the EVP_PKEY will return with a sensible, defined, hash.

Thanks for clarifying it @mattcaswell!

It seems to me that req.c be fixed the similar way to ca.c.

I'm not sure that code should ignore configuration settings. Correction for #6201 , i.e. null digest is fine with me.

I'm not sure that code should ignore configuration settings. Correction for #6201 , i.e. null digest is fine with me

The current solution is causing some significant usability problems, so it doesn't seem to be viable:

https://mta.openssl.org/pipermail/openssl-users/2018-August/008473.html

I added a new commit to make an equivalent change in the req app. Interestingly, while updating the documentation for that I discovered that we already ignore the digest configuration setting in req in some situations so this approach seems consistent with that.

@richsalz please reconfirm

reconfirm. good catch, @petrovr !

I think the catch was @beldmit...!

My point is that program code can not ignore invalid values:

1. null is valid value,
2. missing digest could mean use default , i.e. null
3. other values, for instance sha1, should break processing

I disagree. It is perfectly reasonable to want to set a default digest of (say) sha256 for most stuff, but not use it when doing EdDSA.

I have the required approval for this, and I still think this is the correct approach, but given @petrovr's objection I invite any other comment from @openssl/committers.

I don't see the parameter as forcing a digest value conceptually.

I think defining default_digest to mean the digest for algorithms that allow a settable digest is a reasonable approach (and the documentation should be updated to make that clear that this is the definition of the configuration parameter - not that some algorithms ignore it - that it only applies to algorithms that allow settable values). That way it is clearer what is going on here and looks less like a workaround.

Basically more clarity in the docs wording - but I agree with @mattcaswell in terms of the approach.

It is about how to change existing code :
...
if (md == NULL && (md = lookup_conf(conf, section, ENV_DEFAULT_MD)) == NULL)
goto end;
...
, i.e. parameter "default_md" has to be specified either on command line or in configuration section.
Above could be split as follow:
...
if (md == null) md = lookup_conf(...)
if (md == null) {
if md is required : goto end
else md = "default"
}
...
I expect above code to allow user to use default_md=default and with proposed modifications (correction in existing function and addition of new one) for ASN1_PKEY_CTRL_DEFAULT_MD_NID "null" digest to be used.

Also think about "default" to be set unconditionally in above code snippet.

I expect above code to allow user to use default_md=default and

That would require a user wishing to use EdDSA, along with other algorithms to only use the "default" setting. Setting it to (say) sha256 would rule out the use of EdDSA - which I think is incorrect.

Fix up commit pushed renaming the ecx_ctrl functions as suggested.

An explicit -md mumble will still fail when mumble is not compatible with the key. That's good enough.

Just to be clear an explicit -md _mumble_ will succeed with EdDSA after this PR, but will fail with other algorithms where _mumble_ is not compatible. I think that is what you were saying anyway, but just wanted to be sure.

@vdukhovni please can you reconfirm after the ctrl function name change commit?

I don't like to stop progress on this issue, i.e. approvals.

For protocol I did tests with following code snipped from apps/ca.c

if (md == NULL)
    md = lookup_conf(conf, section, ENV_DEFAULT_MD);

if (md == NULL)
    md = "default";

if (strcmp(md, "null") == 0) {
    dgst = EVP_md_null();
} else {
    if (strcmp(md, "default") == 0) {
        int def_nid;
        if (EVP_PKEY_get_default_digest_nid(pkey, &def_nid) <= 0) {
            BIO_puts(bio_err, "no default digest\n");
            goto end;
        }
        /* NID_undef if signing algorithm requires there to be no digest */
        dgst = (def_nid == NID_undef)
            ? EVP_md_null()
            : EVP_get_digestbynid(def_nid);
    }
}
if (dgst == NULL) {
    if (!opt_md(md, &dgst))
        goto end;
}

This was my idea. Let me know if someone found above generic approach useful. Is so I open an enhancement request.
In all cases go ahead with current status.

BTW, why not to specify a separate NID, not NID_undef for md_null? It can save us from separate processing NID_undef as dgst_nid here and may be in some other places of code.

BTW, why not to specify a separate NID, not NID_undef for md_null? It can save us from separate processing NID_undef as dgst_nid here and may be in some other places of code.

One could perhaps argue that the "Intrinsic" TLS HashAlgorithm defined by RFC 8422 (section 5.1.3) would argue for having such a dedicated new NID, but if we do add such a thing, we'd need to assign it some semantics for all the other places it could be used, and it's unclear that we really want to do that.

For reasons I don't entirely understand (or agree with), this PR seems to have been more controversial than I expected. However I still believe this is the correct approach, and it has all the relevant approvals and support from an OMC member, with no "-1" from any committer, so I have pushed. If people want to they can follow up with further PRs as appropriate.