Repository navigation
feat: add a ServiceMonitor for the llmisvc controller - #88
Merged
Merged
Conversation
…ind address kserve-llmisvc-resources ships a metrics Service and no ServiceMonitor at all, so nothing scrapes it even where Prometheus Operator CRDs are served. Its metrics also bind to 127.0.0.1 by default, which the Service can never reach regardless -- the same class of gap the kagent controller had before kagent-dev/kagent#2886.
Published Helm chart
|
| Chart | kserve-runtime-configs |
| Version | 0.4.3-dev.quentin-llm--vicemonitor.2026-09-22.18-34-43.h9478d7b |
| OCI reference | oci://gsoci.azurecr.io/charts/giantswarm/kserve-runtime-configs:0.4.3-dev.quentin-llm--vicemonitor.2026-09-22.18-34-43.h9478d7b |
| Digest | sha256:2315874b8e3f57930b5bdede4c103aa69e9805341c6eb267673b3d8d05a0300c |
| Registry | public — gsoci.azurecr.io |
Pull this chart
helm pull oci://gsoci.azurecr.io/charts/giantswarm/kserve-runtime-configs --version 0.4.3-dev.quentin-llm--vicemonitor.2026-09-22.18-34-43.h9478d7bPosted by architect-orb · build 1111 · commit 9478d7b · updated in place on every push
Published Helm chart
|
| Chart | kserve-llmisvc-resources |
| Version | 0.4.3-dev.quentin-llm--vicemonitor.2026-09-22.18-34-43.h9478d7b |
| OCI reference | oci://gsoci.azurecr.io/charts/giantswarm/kserve-llmisvc-resources:0.4.3-dev.quentin-llm--vicemonitor.2026-09-22.18-34-43.h9478d7b |
| Digest | sha256:2a6c246bf5f5008be5bee646504d47bbc4f90cff5db0263428990dc9b1b58b7e |
| Registry | public — gsoci.azurecr.io |
Pull this chart
helm pull oci://gsoci.azurecr.io/charts/giantswarm/kserve-llmisvc-resources --version 0.4.3-dev.quentin-llm--vicemonitor.2026-09-22.18-34-43.h9478d7bPosted by architect-orb · build 1108 · commit 9478d7b · updated in place on every push
Published Helm chart
|
| Chart | kserve-resources |
| Version | 0.4.3-dev.quentin-llm--vicemonitor.2026-09-22.18-34-43.h9478d7b |
| OCI reference | oci://gsoci.azurecr.io/charts/giantswarm/kserve-resources:0.4.3-dev.quentin-llm--vicemonitor.2026-09-22.18-34-43.h9478d7b |
| Digest | sha256:61f251e730714d147bb3c4c40757e21c42f69326f2bfd9eb3788eaeb064268a2 |
| Registry | public — gsoci.azurecr.io |
Pull this chart
helm pull oci://gsoci.azurecr.io/charts/giantswarm/kserve-resources --version 0.4.3-dev.quentin-llm--vicemonitor.2026-09-22.18-34-43.h9478d7bPosted by architect-orb · build 1107 · commit 9478d7b · updated in place on every push
Published Helm chart
|
| Chart | kserve-llmisvc-crd |
| Version | 0.4.3-dev.quentin-llm--vicemonitor.2026-09-22.18-34-43.h9478d7b |
| OCI reference | oci://gsoci.azurecr.io/charts/giantswarm/kserve-llmisvc-crd:0.4.3-dev.quentin-llm--vicemonitor.2026-09-22.18-34-43.h9478d7b |
| Digest | sha256:7b99bdb13e65a576f7b4e6659108c0b1d96240ea08b9cd284000ad286c61261e |
| Registry | public — gsoci.azurecr.io |
Pull this chart
helm pull oci://gsoci.azurecr.io/charts/giantswarm/kserve-llmisvc-crd --version 0.4.3-dev.quentin-llm--vicemonitor.2026-09-22.18-34-43.h9478d7bPosted by architect-orb · build 1110 · commit 9478d7b · updated in place on every push
Published Helm chart
|
| Chart | kserve-crd |
| Version | 0.4.3-dev.quentin-llm--vicemonitor.2026-09-22.18-34-43.h9478d7b |
| OCI reference | oci://gsoci.azurecr.io/charts/giantswarm/kserve-crd:0.4.3-dev.quentin-llm--vicemonitor.2026-09-22.18-34-43.h9478d7b |
| Digest | sha256:5be50ade8dbbd1dd3ecdcf71f8185f6d8823a3a5cea65732c1f21fc562cbb063 |
| Registry | public — gsoci.azurecr.io |
Pull this chart
helm pull oci://gsoci.azurecr.io/charts/giantswarm/kserve-crd --version 0.4.3-dev.quentin-llm--vicemonitor.2026-09-22.18-34-43.h9478d7bPosted by architect-orb · build 1109 · commit 9478d7b · updated in place on every push
QuentinBisson
changed the title
The schema generator closes every empty map under noAdditionalProperties, so the hand-edited open labels map was regenerated as additionalProperties: false and pre-commit failed. Annotate it in values.yaml instead.
…itor Upstream defaults --metrics-secure=true, so the plain-HTTP scrape never worked. metricsSecure now sets the flag and the ServiceMonitor scheme; the secure mode scrapes with the scraper's token and ships a metrics-reader ClusterRole with optional subjects. The ServiceMonitor selects the metrics Service by a dedicated label, because the webhook Service carries the same labels and port name.
QuentinBisson
marked this pull request as ready for review
September 22, 2026 18:50
QuentinBisson
added a commit
to giantswarm/agent-platform
that referenced
this pull request
Sep 22, 2026
* feat(observability): scrape the llm-d controller's metrics kserve-llmisvc-resources.kserve.llmisvc.controller.serviceMonitor.enabled follows the resolved global.observability.metrics.serviceMonitor.enabled like every other monitor of this platform, and .labels carries the tenant label. metricsSecure: false serves the metrics as plain HTTP, since the fleet's collector does not authenticate to the secure endpoint. The kserve components move to 0.5.x, the first line with these keys (giantswarm/kserve#88). Part of giantswarm/giantswarm#36711. * fix(connectivity): keep the llm-d controller policy's rendered comments unchanged verify-gpu-pool compares the render byte for byte against origin/main; only the template comment changes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
kserve-llmisvc-resourceshas a metrics Service (llmisvc-controller-manager-service, porthttps/8443 tometrics) and no ServiceMonitor, so nothing scrapes it (giantswarm/giantswarm#36711). The defaultmetricsBindAddressis127.0.0.1, which the Service can not reach. The same gap existed in the kagent controller before kagent-dev/kagent#2886.Change
kserve.llmisvc.controller.metricsBindAddresschanges from127.0.0.1to empty, so--metrics-addr=:8443binds every interface.serviceMonitor.enabled(off by default), takes.intervaland.labels. It selects the metrics Service by a newkserve.giantswarm.io/metrics: "true"label, because the webhook Service has the same labels and port name.metricsSecure(defaulttrue, the upstream default) sets--metrics-secure.truescrapes over HTTPS with the scraper's ServiceAccount token and renders aget /metricsClusterRole, bound toserviceMonitor.metricsReaderSubjectswhen set.falseserves and scrapes plain HTTP with no authentication.No installation runs this component yet (giantswarm/agent-platform#327 keeps it off by default), so the change has no effect until an installation turns it on.