monitor-agents failed to install after upgraded to v1.0.0-rc.1

Describe the bug
monitor-agents failed to install after upgraded to v1.0.0-rc.1.

HelmRelease event decoded:

Warning InstallFailed 5s (x6 over 53s) helm-controller Helm install failed for release cozy-monitoring/monitoring-agents with chart [email protected]+1b5894e823dc: template: cozy-monitoring-agents/charts/fluent-bit/templates/daemonset.yaml:37:28: executing "cozy-monitoring-agents/charts/fluent-bit/templates/daemonset.yaml" at <include (print $.Template.BasePath "/configmap.yaml")>: error calling include: template: cozy-monitoring-agents/charts/fluent-bit/templates/configmap.yaml:16:9: executing "cozy-monitoring-agents/charts/fluent-bit/templates/configmap.yaml" at <tpl .Values.config.outputs .>: error calling tpl: error during tpl function execution for "[OUTPUT]
    Name http
    Match kube.*
    Host vlogs-generic.{{ .Values.global.target }}.svc.{{ (index .Values._cluster \"cluster-domain\") | default \"cluster.local\" }}
    Port 9428
    compress gzip
    uri /insert/jsonline?_stream_fields=log_source,stream,kubernetes_pod_name,kubernetes_container_name,kubernetes_namespace_name&msg_field=log&time_field=date
    format json_lines
    json_date_format iso8601
    header AccountID 0
    header ProjectID 0
[OUTPUT]
    Name http
    Match events.*
    Host vlogs-generic.{{ .Values.global.target }}.svc.{{ (index .Values._cluster \"cluster-domain\") | default \"cluster.local\" }}
    Port 9428
    compress gzip
    uri /insert/jsonline?_stream_fields=log_source,reason,metadata_namespace,metadata_name&msg_field=message&time_field=date
    format json_lines
    json_date_format iso8601
    header AccountID 0
    header ProjectID 0
[OUTPUT]
    Name http
    Match audit.*
    Host vlogs-generic.{{ .Values.global.target }}.svc.{{ (index .Values._cluster \"cluster-domain\") | default \"cluster.local\" }}
    Port 9428
    compress gzip
    uri /insert/jsonline?_stream_fields=log_source,stage,user_username,verb,requestUri&msg_field=requestURI&time_field=date
    format json_lines
    json_date_format iso8601
    header AccountID 0
    header ProjectID 0
": template: gotpl:4:59: executing "gotpl" at <index .Values._cluster "cluster-domain">: error calling index: index of untyped nil

Environment

Cozystack version: v1.0.0-rc.1
Provider: k3s v1.34.3+k3s3

To Reproduce
Steps to reproduce the behavior:

Running cozystack v1.0.0-beta.6
Apply cozystack-operator-generic.yaml of v1.0.0-rc.1.

Expected behaviour
Upgrade succeeds.

Actual behaviour
monitor-agents failed to install

Logs

Paste any relevant logs here. Please redact tokens, passwords, private keys.

Screenshots
If applicable, add screenshots to help explain the problem.

Additional context
related to 0738fae

Checklist

I have checked the documentation
I have searched for similar issues
I have included all required information
I have provided clear steps to reproduce
I have included relevant logs

Robin Lee (@cheese)

Hi Robin Lee (@cheese)! I'm Dosu and I’m helping the cozystack team.

This error is caused by the _cluster values not being populated when the monitoring-agents chart is rendered. The commit you referenced (0738fae56d5610f9871b937b37699f11e21c3f5d) introduced FQDN resolution for tenant workload clusters, and while it uses a nil-safe pattern (index .Values._cluster "cluster-domain") | default "cluster.local", the _cluster values need to be present.

The _cluster values are populated via a Secret named cozystack-values in the cozy-system namespace, created by the platform package. During upgrade from v1.0.0-beta.6 to v1.0.0-rc.1, this could fail to populate due to:

Configuration migration issues - v1.0.0-rc.1 migrated from ConfigMap-based configuration to Platform Package CRD. Do not use --reuse-values when upgrading the Cozystack operator—pass custom values explicitly with --set.
Package dependency ordering - The monitoring-agents package now depends on cozystack.metrics-server, and if dependencies aren't properly ordered during upgrade, _cluster values might not be available yet.

Could you check if the cozystack-values Secret exists in the cozy-system namespace and verify it contains the _cluster configuration?

To reply, just mention @dosu.

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}

# (@dosu)

# (@dosu) Yes, the cozystack-values Secret exists in the cozy-system namespace and contains the _cluster configuration.

Thanks for confirming! The Secret exists in cozy-system, but that's actually only half the picture.

Cozystack uses a Secret replication mechanism - the cozystack-values Secret from cozy-system gets automatically replicated to namespaces labeled with cozystack.io/system=true. The monitoring-agents HelmRelease in cozy-monitoring references the Secret from its own namespace (not cross-namespace).

Could you check:

Does the cozystack-values Secret also exist in the cozy-monitoring namespace?
```
kubectl get secret cozystack-values -n cozy-monitoring
```

Does cozy-monitoring have the required label?

kubectl get ns cozy-monitoring --show-labels | grep cozystack.io/system

If the Secret isn't being replicated to cozy-monitoring, that would explain the "index of untyped nil" error - the HelmRelease can't access the _cluster values because the local copy of the Secret doesn't exist.

To reply, just mention @dosu.

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}

# (@dosu)

# (@dosu) Yes, the cozystack-values Secret also exist in the cozy-monitoring namespace, which has the required label.

This is interesting - commit 0738fae56d5610f9871b937b37699f11e21c3f5d from PR #2075 is confirmed to be included in v1.0.0-rc.1, and that PR introduced the safe nil-handling pattern (index .Values._cluster "cluster-domain") | default "cluster.local".

However, looking at the error message you posted, the template is using that exact pattern but still failing with "index of untyped nil". This suggests the _cluster key might be completely missing from the values (not just the nested cluster-domain key), which would cause index to fail before the default can kick in.

Could you check what's actually in the cozystack-values Secret in cozy-monitoring?

kubectl get secret cozystack-values -n cozy-monitoring -o jsonpath='{.data.values\.yaml}' | base64 -d | grep -A5 "_cluster"

If _cluster is missing or malformed in that Secret, it would explain why the nil-safe pattern is still failing. The replicator copies the Secret, but if the source Secret in cozy-system doesn't have _cluster properly structured, the replicated Secret won't either.

To reply, just mention @dosu.

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}

# (@dosu)

# (@dosu) Output of kubectl get secret cozystack-values -n cozy-monitoring -o jsonpath='{.data.values\.yaml}' | base64 -d | grep -A5 "_cluster" is

_cluster:
  root-host: "xxx.nip.io"
  bundle-name: "isp-full-generic"
  solver: "http01"
  issuer-name: "letsencrypt-prod"
  oidc-enabled: "false"

The template file packages/system/monitoring-agents/charts/fluent-bit/templates/configmap.yaml at line 16 actually trigger the error. Can nested helm chart access the _cluster config in cozystack-values secret?

Found it! The issue is that your _cluster config is missing the cluster-domain key. Your Secret contains:

_cluster:
  root-host: "xxx.nip.io"
  bundle-name: "isp-full-generic"
  solver: "http01"
  issuer-name: "letsencrypt-prod"
  oidc-enabled: "false"

But the fluent-bit outputs template expects _cluster.cluster-domain:

Host vlogs-generic.{{ .Values.global.target }}.svc.{{ (index .Values._cluster "cluster-domain") | default "cluster.local" }}

The tpl function at line 16 passes the root context ($), so the subchart should have access to _cluster. The problem is that when cluster-domain is missing, the index function returns nil correctly, but somewhere in the template chain it's not falling through to the default properly.

This looks like it could be a bug where the cozystack-values Secret in cozy-system (the source) doesn't include cluster-domain in the _cluster section. The platform package should be populating this from networking.clusterDomain.

Can you check what's in the source Secret in cozy-system?

kubectl get secret cozystack-values -n cozy-system -o jsonpath='{.data.values\.yaml}' | base64 -d | grep -A10 "_cluster"

If cluster-domain is also missing there, that's the root cause - the platform configuration isn't setting it properly during upgrade.

To reply, just mention @dosu.

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}

# (@dosu)

# (@dosu) The _cluster config contains the cluster-domain key. You told me to filter the full text with grep -A5 "_cluster", so what I pasted was only a partial content.

Apologies for the confusion with the grep command!

Since _cluster exists with cluster-domain in the Secret, this is puzzling. The error "index of untyped nil" indicates that .Values._cluster is nil at the time the tpl function executes, even though the Secret contains the data.

This could be a timing/injection issue where the valuesFrom injection from the Secret hasn't been properly applied to the HelmRelease. Can you check the actual HelmRelease to see if it has the valuesFrom configuration?

kubectl get helmrelease monitoring-agents -n cozy-monitoring -o yaml | grep -A10 "valuesFrom"

If valuesFrom is missing or not referencing cozystack-values, that would explain why the Secret data isn't making it into the template context - the HelmRelease would be rendering without the _cluster values even though the Secret exists.

Also, could you share the full HelmRelease status to see if there are any other clues?

kubectl get helmrelease monitoring-agents -n cozy-monitoring -o jsonpath='{.status.conditions}' | jq .

To reply, just mention @dosu.

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}

Robin Lee (@cheese)

Hey Robin Lee (@cheese) thanks for report, we'll fix this soon

fix(monitoring-agents): use global.clusterDomain for FQDN resolution

fix(monitoring-agents): revert _cluster FQDN and use cluster-domain i…

This is regression introduced by #2075 and should be fixed by #2086

fix(monitoring-agents): use global.clusterDomain for FQDN resolution (#…

Navigation Menu

Repository navigation

monitor-agents failed to install after upgraded to v1.0.0-rc.1 #2084

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

monitor-agents failed to install after upgraded to v1.0.0-rc.1 #2084

Description

Activity

dosubot commented on Feb 23, 2026

cheese commented on Feb 23, 2026

dosubot commented on Feb 23, 2026

cheese commented on Feb 23, 2026

dosubot commented on Feb 23, 2026

cheese commented on Feb 23, 2026

dosubot commented on Feb 23, 2026

cheese commented on Feb 23, 2026

dosubot commented on Feb 23, 2026

kvaps commented on Feb 23, 2026

kvaps commented on Feb 23, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions