Solr 8.9.0 has a new metrics endpoint being hit: /admin/zookeeper/status. The boostrapped security.json (created by the Solr operator when basic auth is enabled) doesn't wire-up access for this new endpoint for the k8s role.
ERROR - 2021-07-16 22:57:47.229; org.apache.solr.prometheus.collector.SchedulerMetricsCollector; Error occurred during metrics collection => org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://dev-solrcloud-0.dev-solrcloud-headless.dev:8983/solr: Forbidden
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://dev-solrcloud-0.dev-solrcloud-headless.dev:8983/solr: Forbidden
request: https://dev-solrcloud-0.dev-solrcloud-headless.dev:8983/solr/admin/zookeeper/status?wt=json&version=2.2
Work-around is to add the following mapping to your security.json file:
{
"name": "k8s-zk",
"role": "k8s",
"collection": null,
"path": "/admin/zookeeper/status"
},