BN_CONSTTIME_SIZE_LIMIT or BN_MONT_CONSTTIME_SIZE_LIMIT?

BN_SIZE_LIMIT feels right, as @davidben pointed out also BN_mod_exp and BN_mod_exp_mont will eventually call
bn_mul_mont_fixed_top and thus bn_mul_mont, so it is neither specific to consttime nor montgomery at all.
Actutally the whole issue starts to look a lot less harmless than in the beginning.

Then BN_MONT_SIZE_LIMIT?

Unless this limit applies to all BN operations it should not be called BN_SIZE_LIMIT.

It is more the upper bound on the stack allocation of any BN operation we are willing to allow.
It also affects BN_mod_exp and BN_mod_sqrt.
Note: there are many call sites of the affected functions, especially of BN_mod_exp,
but not all of them seem to sanity check the input parameters.
If the modulus is a large enough odd number, a stack overflow will probably happen unexpectedly

Having a function that's explicitly about treating the exponent as secret then fallback to a function which doesn't seems poor. It seems better to just make the operation fail at this point.

Additionally, both these operations actually have comparable memory requirements anyway, because they're both windowed exponentiations. It's just that this one happens to allocate the memory contiguously, while BN_mod_exp_mont happens to just make a bunch of BIGNUMs. (Well, slightly less memory usage because, by using a sliding window, the public exponent variant is able to halve the table size, since you only need odd powers.)

The fallback from bn_mul_mont is more defensible, though I'd personally still just declare this is the limit for Montgomery reduction so there aren't extra codepaths to test and such.

The rationale is this:
I dont think there is any valid cryptographic reason for doing computations with such large moduli,
RSA is limited 16384 bits, DH and DSA is limited to 10000 bits for a good reason.
It is much more likely that someone was tricked into calling this function with some bogus values
because doing this is likely to cause a stack overflow.
However, it is a clear ABI break to return an error when previously this function might have succeeded.
On the other hand, it is easy to configure with ./config -DBN_SIZE_LIMIT=1000000 if you are fine
with a stack allocation of 1MB or more.

Having a function that's explicitly about treating the exponent as secret

by the way, the number of bits in the exponent is not a secret at all.
I kind of assumed that the while loop down below uses the number of bits in the modulus,
but now i see it takes a short cut if the exponent is much smaller than the modulus.

I dont think there is any valid cryptographic reason for doing computations with such large moduli,

Well, one exception came just into my mind: I think I remember having read once a paper where researchers
did multiply the moduli of lots of publically available RSA certificates, forming huge composite numbers, and
doing GCD to find common divisors between them, And in deed that did result in some common primes
that were the result of poor random number generators.

However I doubt that they did consider using openssl for such computations, sinve there are better suited
math programs for math with such huge numbers. And moreover I doubt that I would call such use a
legitimate cryptographic use.

And doing this kind of math is not safe either, since currently all openssl versions might crash unexpectedly
with such computations, or will just be not optimized for use with such numbers if this PR gets merged.
But I wouldn't rule out the possibility that a future openssl version will turn this soft limit into a hard limit.

As this is not hard size limit I'd still like to see a different name. BN_STACK_SIZE_LIMIT?

I have extended the comment:
It is not recommended to do computations with numbers exceeding this limit,
since the result will be highly version dependent:
While the current OpenSSL version will use non-optimized, but safe code,
previous versions will use optimized code, that may crash due to unexpected
stack overflow, and future versions may very well turn this into a hard
limit.
Note however, that it is possible to override the size limit using
"./config -DBN_SITE_LIMIT=" if necessary, and the O/S specific
stack limit is known and taken into consideration.

I'd still like to see a different name for the macro. The current one is totally misleading.

Could we agree on BN_SOFT_LIMIT ?

That would be fine with me.