Operator fails to enable authentication for Solr version 6.6.6, 6.6.6-slim #372
Description
Summary:
The operator documentation says that it works with all official Solr images. However, the operator is not properly configuring authentication for solrImageTag 6.6.6-slim or 6.6.6.
We are dependent on Solr version 6.6.6 since that is the most recent Solr version supported by CKAN, which is the basis for our application (data.gov).
What I observed:
When I enter the service URL in my browser, I see the Solr Admin UI.
What I expected:
When I enter the service URL in my browser, I am prompted for authentication.
How to reproduce:
$ kubectl create -f https://solr.apache.org/operator/downloads/crds/v0.4.0/all-with-dependencies.yaml
$ helm install --namespace kube-system --version 0.4.0 solr solr-operator
# [Wait until the operator is ready...]
$ helm install example-solr apache-solr/solr --version 0.4.0 \
--set image.tag="6.6.6-slim" \
--set solrOptions.javaMemory="-Xms300m -Xmx300m" \
--set solrOptions.security.authenticationType="Basic" \
--set addressability.external.method=Ingress \
--set addressability.external.domainName="ing.local.domain" \
--set addressability.external.useExternalAddress="true"
$ kubectl get solrclouds -w # Until the nodes are all ready
$ output=$(curl -s -o /dev/null -L -w "%{http_code}" "http://default-example-solrcloud.ing.local.domain/solr/#/~cloud?view=nodes")
$ [ $output -eq "401" ] || echo "Authentication is not enabled"
You will see "Authentication is not enabled" on the final line of output.
You get the same result if you use image.tag 6.6.6.
Observations
We don't know a lot about how the operator does its thing, but we tried querying the ZK instance for /security.json and got {} back. From inspection of the code, we think this might be an indicator that the setup-zk initContainer is not doing its job correctly.
Here's what we see in the solr-operator logs:
2021-11-09T23:55:14.703Z INFO controllers.SolrCloud Updating SolrCloud Status {"namespace": "default", "solrCloud": "solr-ae12422a8985749e", "status": {"solrNodes":[],"replicas":0,"podSelector":"solr-cloud=solr-ae12422a8985749e,technology=solr-cloud","readyReplicas":0,"upToDateNodes":0,"version":"6.6.6","internalCommonAddress":"http://solr-ae12422a8985749e-solrcloud-common.default","externalCommonAddress":"http://default-solr-ae12422a8985749e-solrcloud.ing.local.domain","zookeeperConnectionInfo":{"internalConnectionString":"solr-ae12422a8985749e-solrcloud-zookeeper-0.solr-ae12422a8985749e-solrcloud-zookeeper-headless.default.svc.cluster.local:2181,solr-ae12422a8985749e-solrcloud-zookeeper-1.solr-ae12422a8985749e-solrcloud-zookeeper-headless.default.svc.cluster.local:2181,solr-ae12422a8985749e-solrcloud-zookeeper-2.solr-ae12422a8985749e-solrcloud-zookeeper-headless.default.svc.cluster.local:2181","chroot":"/"},"backupRestoreReady":false}}
2021-11-09T23:55:14.721Z ERROR controller Reconciler error {"reconcilerGroup": "solr.apache.org", "reconcilerKind": "SolrCloud", "controller": "solrcloud", "name": "solr-ae12422a8985749e", "namespace": "default", "error": "Operation cannot be fulfilled on solrclouds.solr.apache.org \"solr-ae12422a8985749e\": the object has been modified; please apply your changes to the latest version and try again"}
github.com/go-logr/zapr.(*zapLogger).Error
/go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:132
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:246
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:218
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:197
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.Until
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90
2021-11-09T23:55:15.725Z INFO controllers.SolrCloud Updating SolrCloud Status {"namespace": "default", "solrCloud": "solr-ae12422a8985749e", "status": {"solrNodes":[{"name":"solr-ae12422a8985749e-solrcloud-0","nodeName":"datagov-broker-test-worker3","internalAddress":"http://solr-ae12422a8985749e-solrcloud-0.default","externalAddress":"http://default-solr-ae12422a8985749e-solrcloud-0.ing.local.domain","ready":false,"version":"6.6.6","specUpToDate":true}],"replicas":1,"podSelector":"solr-cloud=solr-ae12422a8985749e,technology=solr-cloud","readyReplicas":0,"upToDateNodes":1,"version":"6.6.6","internalCommonAddress":"http://solr-ae12422a8985749e-solrcloud-common.default","externalCommonAddress":"http://default-solr-ae12422a8985749e-solrcloud.ing.local.domain","zookeeperConnectionInfo":{"internalConnectionString":"solr-ae12422a8985749e-solrcloud-zookeeper-0.solr-ae12422a8985749e-solrcloud-zookeeper-headless.default.svc.cluster.local:2181,solr-ae12422a8985749e-solrcloud-zookeeper-1.solr-ae12422a8985749e-solrcloud-zookeeper-headless.default.svc.cluster.local:2181,solr-ae12422a8985749e-solrcloud-zookeeper-2.solr-ae12422a8985749e-solrcloud-zookeeper-headless.default.svc.cluster.local:2181","chroot":"/"},"backupRestoreReady":false}}
2021-11-09T23:55:15.754Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "solr.apache.org", "reconcilerKind": "SolrCloud", "controller": "solrcloud", "name": "solr-ae12422a8985749e", "namespace": "default"}
2021-11-09T23:55:15.755Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "solr.apache.org", "reconcilerKind": "SolrCloud", "controller": "solrcloud", "name": "solr-ae12422a8985749e", "namespace": "default"}
2021-11-09T23:55:15.811Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "solr.apache.org", "reconcilerKind": "SolrCloud", "controller": "solrcloud", "name": "solr-ae12422a8985749e", "namespace": "default"}
2021-11-09T23:55:15.855Z INFO controllers.SolrCloud Updating SolrCloud Status {"namespace": "default", "solrCloud": "solr-ae12422a8985749e", "status": {"solrNodes":[{"name":"solr-ae12422a8985749e-solrcloud-0","nodeName":"datagov-broker-test-worker3","internalAddress":"http://solr-ae12422a8985749e-solrcloud-0.default","externalAddress":"http://default-solr-ae12422a8985749e-solrcloud-0.ing.local.domain","ready":false,"version":"6.6.6","specUpToDate":true}],"replicas":1,"podSelector":"solr-cloud=solr-ae12422a8985749e,technology=solr-cloud","readyReplicas":0,"upToDateNodes":1,"version":"6.6.6","internalCommonAddress":"http://solr-ae12422a8985749e-solrcloud-common.default","externalCommonAddress":"http://default-solr-ae12422a8985749e-solrcloud.ing.local.domain","zookeeperConnectionInfo":{"internalConnectionString":"solr-ae12422a8985749e-solrcloud-zookeeper-0.solr-ae12422a8985749e-solrcloud-zookeeper-headless.default.svc.cluster.local:2181,solr-ae12422a8985749e-solrcloud-zookeeper-1.solr-ae12422a8985749e-solrcloud-zookeeper-headless.default.svc.cluster.local:2181,solr-ae12422a8985749e-solrcloud-zookeeper-2.solr-ae12422a8985749e-solrcloud-zookeeper-headless.default.svc.cluster.local:2181","externalConnectionString":"N/A","chroot":"/"},"backupRestoreReady":false}}