I'm having some issues making an SSL/TLS connection to redis for the Celery broker and backend.

I'm using:
Celery: 4.3.0rc2 (master)
Python: 3.6
OS: Mac OS X.
Related dependency versions: kombu (4.4.0), redis-py (3.2.0)

Celery works fine using a non-SSL connection to redis. My redis/SSL setup uses redis behind stunnel and I can successfully connect to and use this deployment via py-redis directly and via other libraries.

It looks like this is related to #3830, however that issue seemed to have been resolved by adding the broker_use_ssl option and adding this option doesn't appear to be having any effect here.

To setup SSL, I'm adding the broker_use_ssl attribute to configuration parameters as described in the docs. I'm also adding the same set of parameters for the redis_backend_use_ssl option. I notice there is some information here about adding the parameters in the query string but I've opted to stick with providing the dict of parameters.

broker_url = 'rediss://localhost:6380'
key_file = '/path/to/client.key' 
cert_file = '/path/to/client.crt' 
ca_file = '/path/to/CAcert.pem'
    
app = Celery('app', broker=broker_url, backend=broker_url,
             broker_use_ssl = {
                 'keyfile': key_file, 'certfile': cert_file, 
                 'ca_certs': ca_file,  
                 'cert_reqs': ssl.CERT_REQUIRED 
            }, 
             redis_backend_use_ssl = { 
                 'keyfile': key_file, 'certfile': cert_file,
                 'ca_certs': ca_file,  
                 'cert_reqs': ssl.CERT_REQUIRED
            })
app.connection().connect()
app.send_task('a_task')

When I attempt to make and use the connection as shown above, I get an error:

ValueError: A rediss:// URL must have parameter ssl_cert_reqs be CERT_REQUIRED, CERT_OPTIONAL, or CERT_NONE

It looks like the rediss URL scheme is being picked up but the other parameters are not. Apologies if I'm missing something in the documentation or elsewhere.

Some investigation suggests that the broker_use_ssl and redis_backend_use_ssl options are not being carried through to the connection settings and, indeed, looking at the __init__ function for the Celery object, I can't see that these are being used anywhere. Stepping through the code, they don't seem to be present by the time the connection function is reached and ssl in the input parameters to the connection function is set to None.

As a test, I've tried making a couple of additions to the Celery __init__ function to add the broker_use_ssl and redis_backed_use_ssl to the configuration:

        self.__autoset('broker_use_ssl', 
                       (kwargs['broker_use_ssl'] if 'broker_use_ssl' 
                        in kwargs else None))
        self.__autoset('redis_backend_use_ssl', 
                       (kwargs['redis_backend_use_ssl'] if  
                        'redis_backend_use_ssl' in kwargs else None))

This adds the provided options to the Celery conf. It flags up another issue which is that the SSL parameters used for the backend connection use different names to those used for the broker connection - the documentation says the values are the same as broker_use_ssl (although the correct values are shown in the example of using query string parameters). I assume this is one of the things being handled under issue #4812.

A further point that I'm unclear about is that in the _connection function, there is a call to get the broker_use_ssl parameters ssl=self.either('broker_use_ssl', ssl) and this returns None. However, if I manually create an app object and then call either, it returns the broker_use_ssl parameters correctly (parameters as per example above):

app = Celery('app', broker=broker_url, backend=broker_url,
             broker_use_ssl = {
                 'keyfile': key_file, 'certfile': cert_file, 
                 'ca_certs': ca_file,  
                 'cert_reqs': ssl.CERT_REQUIRED 
            }, 
             redis_backend_use_ssl = { 
                 'keyfile': key_file, 'certfile': cert_file,
                 'ca_certs': ca_file,  
                 'cert_reqs': ssl.CERT_REQUIRED
            })

app.either('broker_use_ssl', None)  

# Returns:
# {'keyfile': '/path/to/client.key',
#  'certfile': '/path/to/client.crt',
#  'ca_certs': '/path/to/CAcert.pem',
#  'cert_reqs': <VerifyMode.CERT_REQUIRED: 2>}

So, I'm unclear if I'm missing something and taking completely the wrong approach here or whether there are some issues with the SSL implementation in 4.3.0rc2.

Thanks.