This is one of those things that makes you question your sanity, but might just be a small bug related to a very specific edge case.

We're using the valueFrom syntax to configure S3 bucket credentials just like mentioned here: https://altinity.com/blog/clickhouse-confidential-using-kubernetes-secrets-with-the-altinity-operator

Everything worked fine in 2 testing environments, but when deploying the changes to our production cluster (using a differently named production bucket), ClickHouse just returned an error as if it had invalid credentials.

After some extended head scratching we came to the conclusion that if the bucket name has exactly the wrong number of characters in it, then the configuration gets ignored and both the env var and the config using the env var are not present in the kube cluster.

Example:

      s3/iamtwentyonecharslong/secret_access_key:
        valueFrom:
          secretKeyRef:
            key: secret
            name: hmac
      s3/thebucketnameof22chars/secret_access_key:
        valueFrom:
          secretKeyRef:
            key: secret
            name: hmac
      s3/thishasjust23characters/secret_access_key:
        valueFrom:
          secretKeyRef:
            key: secret
            name: hmac

Results in the following in the ClickHouse pod:

        - name: CONFIGURATION_SETTINGS_S3_IAMTWENTYONECHARSLONG_SECRET_ACCESS_K
          valueFrom:
            secretKeyRef:
              key: secret
              name: hmac
        - name: CONFIGURATION_SETTINGS_S3_THISHASJUST23CHARACTERS_SECRET_ACCESS
          valueFrom:
            secretKeyRef:
              key: secret
              name: hmac

So the configuration for the bucket with name thebucketnameof22chars is missing.

In the *-common-configd configmap it looks like this:

        <s3>
            <iamtwentyonecharslong>
                <secret_access_key from_env="CONFIGURATION_SETTINGS_S3_IAMTWENTYONECHARSLONG_SECRET_ACCESS_K"></secret_access_key>
            </iamtwentyonecharslong>
            <thebucketnameof22chars>
                <secret_access_key from_env=""></secret_access_key>
            </thebucketnameof22chars>
            <thishasjust23characters>
                <secret_access_key from_env="CONFIGURATION_SETTINGS_S3_THISHASJUST23CHARACTERS_SECRET_ACCESS"></secret_access_key>
            </thishasjust23characters>
        </s3>

I suspect this has something to do with the truncated env var name ending with a _ character, since the missing env var should be CONFIGURATION_SETTINGS_S3_THEBUCKETNAMEOF22CHARS_SECRET_ACCESS_

We are currently running clickhouse-operator:0.24.4. We have not yet tried updating to the latest version, but we will do that and report the results here.