Skip to content

constify *_dup() and *i2d_*() and related fns, add DECLARE_ASN1_DUP_FUNCTION#8029

Closed
DDvO wants to merge 28 commits intoopenssl:masterfrom
mpeylo:constify_dup
Closed

constify *_dup() and *i2d_*() and related fns, add DECLARE_ASN1_DUP_FUNCTION#8029
DDvO wants to merge 28 commits intoopenssl:masterfrom
mpeylo:constify_dup

Conversation

@DDvO
Copy link
Contributor

@DDvO DDvO commented Jan 16, 2019

This adds const to parameters of *_dup() and *i2d_*() and related functions as far as (likely) possible, introducing also DECLARE_ASN1_DUP_FUNCTION.

I'm just not fully sure if there are cases of deep (vs. shallow) copying, which would require extra care.

This PR has been extracted from #7646 as advised by @FdaSilvaYY and @mspncp.

Fixes #3742
partly.

  • documentation is added or updated

@DDvO DDvO mentioned this pull request Jan 16, 2019
@DDvO
Copy link
Contributor Author

DDvO commented Jan 21, 2019

I've just eliminated further type casts from evp.h as far as possible,
changed the single remaining (char *) to (void *),
and made the type given for EVP_PKEY_CTX_set_mac_key() in EVP_PKEY_CTX_ctrl.pod() consistent with its 'implementation' as a macro (which includes a cast) in evp.h.

@richsalz
Copy link
Contributor

richsalz commented Jan 22, 2019 via email

Copy link
Member

@levitte levitte left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm going to assume that you scripted an automatic replacement of every _dup declaration with the corresponding DECLARE_ASN1_DUP_FUNCTION line. That script obviously went rogue, as it affected non-ASN.1 types. Or, if it was intentional, I'm just gonna say "no", because this is just deeply confusing (I'll go as far as saying that this is an abuse of that macro)

UPDATE: turns out I was only half right. For some of the items I complained about, "no" still stands. For others, not.

@DDvO
Copy link
Contributor Author

DDvO commented Jan 23, 2019

Nice clean-up!

Pleased to hear!

It caused quite a lot of work for me, in particular since constifying some functions required constifying many further declarations. In some cases it was pretty hard to find out that although a cast to const was needed this is semantically justified. Occasionally I was able to do some code rearrangements that persuaded the compiler that adding const is fine.

@levitte
Copy link
Member

levitte commented Jan 23, 2019

@DDvO, sorry for all the unnecessary noise

@levitte
Copy link
Member

levitte commented Jan 24, 2019

FYI, Travis currently reports this:

../_srcdist/crypto/evp/pmeth_lib.c:253:15: error: conflicting types for 'EVP_PKEY_CTX_dup'
 EVP_PKEY_CTX *EVP_PKEY_CTX_dup(const EVP_PKEY_CTX *pctx)
               ^
In file included from ../_srcdist/include/openssl/pem.h:16:0,
                 from ../_srcdist/include/openssl/ui.h:19,
                 from ../_srcdist/include/openssl/engine.h:24,
                 from ../_srcdist/crypto/evp/pmeth_lib.c:13:
../_srcdist/include/openssl/evp.h:1399:15: note: previous declaration of 'EVP_PKEY_CTX_dup' was here
 EVP_PKEY_CTX *EVP_PKEY_CTX_dup(EVP_PKEY_CTX *ctx);

@DDvO
Copy link
Contributor Author

DDvO commented Jan 24, 2019

../_srcdist/crypto/evp/pmeth_lib.c:253:15: error: conflicting types for 'EVP_PKEY_CTX_dup'

Thanks for pointing this out; when reverting to the old format I forgot to add the 'const'. Fixed.

@DDvO
Copy link
Contributor Author

DDvO commented Jan 25, 2019

Meanwhile I've also been able to constify i2d_ECPrivateKey(), i2d_ECParameters(), i2d_X509_AUX(), and i2d_SSL_SESSION() (where for the latter a justified cast from const is needed for technical reasons).

Then I realized that all the _const variants of the various DECLARE_ASN1_ and IMPLEMENT_ASN1_ macros as well as of the I2D_OF, ASN1_i2d_bio_of, ASN1_i2d_fp_of, and ASN1_dup_of macros are not needed any more after constifying (most of) the i2d_ functions. So I eliminated all of them, since the 'normal' ones can now include the const.
This significantly reduces redundancy in asn1.h and asn1t.h and avoids two evil casts to remove const in IMPLEMENT_ASN1_ENCODE_FUNCTIONS_const_fname.

Moreover, I've folded various 'manual' declarations of i2d_, d2i_ and _it functions by the use of DECLARE_ASN1_FUNCTIONS and friends.

@DDvO
Copy link
Contributor Author

DDvO commented Jan 25, 2019

I've just updated ParseC.pm to handle the new DECLARE_ASN1_* macros
and improved the macro argument matching there to allow whitespace after ','.

@DDvO
Copy link
Contributor Author

DDvO commented Jan 25, 2019

I've also constified most of the i2d_ functions with BIO or FILE output
and added comments where this is not possible (due to CMS_stream() and PKCS7_stream()).

@DDvO
Copy link
Contributor Author

DDvO commented Jan 29, 2019

Yesterday I noticed that - for whatever reason - Travis CI had not been run again after my last updates of Jan 25th, such that some outdated build errors still showed :-(
This is not the first time I experience this CI misbehavior.

The most elegant workaround for this issue I found meanwhile is to rebase and force-push :-)

@levitte
Copy link
Member

levitte commented Jan 29, 2019

Travis refuses to run jobs on PRs that it thinks has merge problems with the branch the PR started off from.

@DDvO
Copy link
Contributor Author

DDvO commented Jan 29, 2019

Travis refuses to run jobs on PRs that it thinks has merge problems with the branch the PR started off from.

This would indeed be a good reason, but AFAICS the (only) conflicting commit that has been pushed onto the master in the meantime (e85d19c) was pushed on Jan 27, which was several days after Travis took a break on this PR.

@DDvO
Copy link
Contributor Author

DDvO commented Jan 30, 2019

Travis refuses to run jobs on PRs that it thinks has merge problems with the branch the PR started off from.

This would indeed be a good reason, but AFAICS the (only) conflicting commit that has been pushed onto the master in the meantime (e85d19c) was pushed on Jan 27, which was several days after Travis took a break on this PR.

This time Travis simply got stuck on two builds :-/ saying

No output has been received in the last 10m0s, this potentially indicates a stalled build or something wrong with the build itself.

@levitte
Copy link
Member

levitte commented Jan 31, 2019

Ive restarted those two jobs, let's hope that changes things

@DDvO
Copy link
Contributor Author

DDvO commented Jan 31, 2019

I've restarted those two jobs, let's hope that changes things

It did - thanks :)

@DDvO
Copy link
Contributor Author

DDvO commented Feb 7, 2019

The last comments on the contents of this PR have been answered a week ago.
Does anyone have further ones, or how to proceed now?

Copy link
Member

@mattcaswell mattcaswell left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not a full review. I just noticed a few things.

Copy link
Member

@mattcaswell mattcaswell left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few minor things. But, assuming those are fixed, I think this is ok. Still needs a second review.

@mattcaswell mattcaswell added the approval: review pending This pull request needs review by a committer label Feb 22, 2019
@DDvO
Copy link
Contributor Author

DDvO commented Feb 23, 2019

Thanks @mattcaswell for your review and first approval.

@levitte, since you also requested changes and gave many comments on this PR that lead to various major improvements, I think it would be very beneficial if you could do the second review.

Copy link
Contributor

@FdaSilvaYY FdaSilvaYY left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice and impressive work 👍
Just a few nits more.

Copy link
Member

@mattcaswell mattcaswell left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reconfirmed after styling nit update. Ping @levitte

@DDvO
Copy link
Contributor Author

DDvO commented Mar 4, 2019

@levitte, when do you think you can do the final review/approval?

@levitte
Copy link
Member

levitte commented Mar 6, 2019

Looking now. Sorry, was distracted by Other Stuff™

@levitte levitte added approval: done This pull request has the required number of approvals and removed approval: review pending This pull request needs review by a committer labels Mar 6, 2019
@DDvO
Copy link
Contributor Author

DDvO commented Mar 6, 2019

Thanks @levitte -
I'm very glad that this large and trick set of internal improvements is approved now!

@mattcaswell
Copy link
Member

I squashed the commits and pushed this. Thanks @DDvO for this significant contribution!

@mattcaswell mattcaswell closed this Mar 6, 2019
levitte pushed a commit that referenced this pull request Mar 6, 2019
…e, introducing DECLARE_ASN1_DUP_FUNCTION

Reviewed-by: Richard Levitte <[email protected]>
Reviewed-by: Matt Caswell <[email protected]>
(Merged from #8029)
@DDvO DDvO deleted the constify_dup branch March 20, 2019 08:56
mspncp pushed a commit to mspncp/openssl that referenced this pull request Jul 10, 2019
…e, introducing DECLARE_ASN1_DUP_FUNCTION

Reviewed-by: Richard Levitte <[email protected]>
Reviewed-by: Matt Caswell <[email protected]>
(Merged from openssl#8029)

(cherry picked from commit 9fdcc21)

 Conflicts:
	crypto/ess/ess_asn1.c    (not in 1.1.1)
	crypto/evp/mac_lib.c     (not in 1.1.1)
	crypto/evp/pkey_mac.c    (not in 1.1.1)
	include/openssl/ess.h    (not in 1.1.1)
	include/openssl/evp.h    (EVP_MAC, EVP_MAC_CTX removed)
	util/perl/OpenSSL/ParseC.pm (not in 1.1.1)
@mspncp
Copy link
Contributor

mspncp commented Jul 10, 2019

@DDvO it's a pity that this constification was not backported to 1.1.1! Im currently porting my company's code from 1.0.2 to 1.1.1 and I stumbled over constness problems which you already fixed on master. I tried a cherry-pick, all conflicts were trivially resolvable (new files which did not exist yet in 1.1.1, as well as the new EVP_MAC and EVP_MAC_CTX prototypes in evp.h).

Unfortunately, the cm_pmeth.c file was alredy deleted in master when your patch got merged, so this file was not constified by you.

crypto/cmac/cm_pmeth.c:134:5: error: initialization of \
'int (*)(EVP_PKEY_CTX *, const EVP_PKEY_CTX *)' {aka 'int (*)(struct evp_pkey_ctx_st *, const struct evp_pkey_ctx_st *)'}
        from incompatible pointer type 'int (*)(EVP_PKEY_CTX *, EVP_PKEY_CTX *)'
               {aka 'int (*)(struct evp_pkey_ctx_st *, struct evp_pkey_ctx_st *)'} [-Werror=incompatible-pointer-types]
     pkey_cmac_copy,
     ^~~~~~~~~~~~~~

Would it be a great effort for you to fix the constification for me and create a pull request for 1.1.1? Your help would be highly appreciated!

Ideally, you would call the pull request

 constify *_dup() and *i2d_*() and related fns, add DECLARE_ASN1_DUP_FUNCTION [1.1.1]

and refer to #8029 in the description. To give you a start, I pushed the cherry-picked commit to mspncp@6e36edb on my repository clone. (It's on the branch pr-8029-constify-dup-111).

To obtain a local copy of the branch, you can do the following

$ git fetch https://github.com/mspncp/openssl pr-8029-constify-dup-111 
From https://github.com/mspncp/openssl
 * branch                  pr-8029-constify-dup-111 -> FETCH_HEAD

$ git checkout -b pr-8029-constify-dup-111 FETCH_HEAD
Switched to a new branch 'pr-8029-constify-dup-111'

@DDvO
Copy link
Contributor Author

DDvO commented Jul 10, 2019

I'll give it a try..

@mspncp
Copy link
Contributor

mspncp commented Jul 11, 2019

I'll give it a try..

On the next morning, with more sleep, the problem turned out to be easy to fix. I'll open a pull request soon. Thanks again for volunteering and sorry for interrupting you.

mspncp added a commit to mspncp/openssl that referenced this pull request Jul 11, 2019
… possible, introducing DECLARE_ASN1_DUP_FUNCTION

Fix compiler errors for modules which were already removed when
pull request openssl#8029 was merged to master.
@mspncp
Copy link
Contributor

mspncp commented Jul 11, 2019

See #9347. Feedback welcome.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approval: done This pull request has the required number of approvals

Projects

None yet

Development

Successfully merging this pull request may close these issues.

more const

8 participants