opensearch: securityconfig Job of an instance created before v1.7 keeps the cluster.local host after upgrade and is never recreated #5164

What is wrong?

On a cluster with the default networking.clusterDomain: cozy.local, an OpenSearch created on v1.6.5 or earlier stays not Ready after upgrade to v1.7. Its pods start now (#4152), but security is never initialized: bootstrap pod logs Failure no such index [.opendistro_security] and Not yet initialized (you may need to run securityadmin), node pod stays 0/1 and logs Not yet initialized too.

The <release>-securityconfig-update Job was created by the operator on v1.6.5, when it ran with DNS_BASE=cluster.local, and the upgrade keeps it with the old host:

until curl -k --silent https://opensearch-db-os-ext.tenant-a.svc.cluster.local:9200;
do
echo 'Waiting to connect to the cluster'; sleep 120;
done;

On my cluster it looped on that curl, then ended Failed (BackoffLimitExceeded, backoffLimit: 0) about 15 hours after the upgrade with no pod left, and nothing recreated it for more than four days after that, the upgrade to v1.7.0-rc.3 included.

#4185 moved the operator to the platform domain, but operator v2.8.0 deletes and recreates an existing Job only when its securityconfig/checksum annotation differs from the checksum of the securityconfig Secret (opensearch-operator/pkg/reconcilers/securityconfig.go:140-152), it doesn't look at the Job status. Host is not part of that checksum, so the DNS_BASE change never reaches the Job. v1.7.0-rc.3 ships the same operator (packages/system/opensearch-operator/images/opensearch-operator/Dockerfile:70), and no platform migration touches these Jobs.

I deleted the Job on that cluster after the upgrade to rc.3. Operator logged Starting securityconfig update job and created a new Job 3 seconds later with host opensearch-db-os-ext.tenant-a.svc.cozy.local. Its curl probe passed on the first try, securityadmin connected as CN=admin,OU=opensearch-db-os-ext, then waited for YELLOW cluster state and kept timing out for the 15 minutes I watched. That instance has a second fault of its own (bootstrap and node pods hold different cluster UUIDs, so the node never joins), so it doesn't show whether deleting the Job is enough for an instance to finish initializing.

Repro

1. On v1.6.5 with the default cluster domain create:

   apiVersion: apps.cozystack.io/v1alpha1
   kind: OpenSearch
   metadata:
     name: db-os-ext
     namespace: tenant-a
   spec:
     replicas: 1
     size: 5Gi
     external: true

   On v1.6.5 its node pod is refused by PodSecurity (OpenSearch fails to start in tenant namespaces: privileged init-sysctl violates default baseline PodSecurity #3022), the securityconfig Job is created anyway and loops on .svc.cluster.local.

2. Upgrade the platform to v1.7.0-rc.2.

3. Node pod opensearch-db-os-ext-nodes-0 is Running 0/1. Job opensearch-db-os-ext-securityconfig-update is still the one from step 1, with the .svc.cluster.local host.

Environment

Cozystack v1.6.5 -> v1.7.0-rc.2 -> v1.7.0-rc.3, isp-full variant, networking.clusterDomain: cozy.local.

How it should work?

Upgrade replaces such Jobs. #4185 left it out on purpose, because the fix has to delete Jobs in tenant namespaces. It can be a platform migration or a pre-upgrade hook that deletes *-securityconfig-update Jobs whose host is not on the cluster domain, Failed ones too, or our operator patch adds the host to what the reconciler compares.