[Bug]: No run metrics on kubernetes backend when privileged: true #3398
Description
Steps to reproduce
- Deploy a
minikubecluster:minikube start --driver=kvm2 --container-runtime=containerd1 - Start any run with
privileged: true - Check metrics:
dstack metrics <run-name>
Actual behaviour
NAME STATUS CPU MEMORY GPU
dev-environment running - - -
Expected behaviour
No response
dstack version
0.20.0
Server logs
WARNING dstack._internal.server.background.tasks.process_metrics:139 Failed to connect to job dev-environment-0-0 to collect metricsAdditional information
$ ssh <run-name>
# curl -w "%{http_code}\n" http://localhost:10999/api/metrics
could not read memory usage: open /sys/fs/cgroup/memory.current: no such file or directory
500
Footnotes
-
Cannot reproduce with
dockercontainer runtime ↩