Image garbage collection is not working on workers

Hi, my nodes are not able to run new images after images fs overfilled:

kubectl describe pod angular-app-5cbcdcc7db-xrcrb -n default
Events:
  Type    Reason   Age                    From     Message
  ----    ------   ----                   ----     -------
  Normal  BackOff  4m23s (x2385 over 9h)  kubelet  Back-off pulling image "registry.onayzakup.kz/frontend/angular-app:dbb507a20c2523dbf752c767f6341f0fc81cd539"

I found that Kamaji sets Kubelet configuration to disable garbage collection.

According to this line:

kamaji/internal/kubeadm/uploadconfig.go

Line 115 in 755cc5b

ImageGCHighThresholdPercent: pointer.To(int32(100)),

The garbage collection for images is not triggered
(see: kubernetes/kubernetes#79929)

Is there any specific reason, why is kamaji disables garbage collection?

Also all of these zeroDuration look weird to me? Is there any reason why are they configured like so?

kamaji/internal/kubeadm/uploadconfig.go

Lines 110 to 125 in 755cc5b

    
           EvictionPressureTransitionPeriod: zeroDuration, 
        
           FileCheckFrequency:               zeroDuration, 
        
           HealthzBindAddress:               "127.0.0.1", 
        
           HealthzPort:                      pointer.To(int32(10248)), 
        
           HTTPCheckFrequency:               zeroDuration, 
        
           ImageGCHighThresholdPercent:      pointer.To(int32(100)), 
        
           NodeStatusUpdateFrequency:        zeroDuration, 
        
           NodeStatusReportFrequency:        zeroDuration, 
        
           RotateCertificates:               true, 
        
           RuntimeRequestTimeout:            zeroDuration, 
        
           ShutdownGracePeriod:              zeroDuration, 
        
           ShutdownGracePeriodCriticalPods:  zeroDuration, 
        
           StaticPodPath:                    "/etc/kubernetes/manifests", 
        
           StreamingConnectionIdleTimeout:   zeroDuration, 
        
           SyncFrequency:                    zeroDuration, 
        
           VolumeStatsAggPeriod:             zeroDuration,

According to kubelet configuration, they should have default values:
https://kubernetes.io/docs/reference/config-api/kubelet-config.v1beta1/

fix kamaji garbage collection (#263)

Definitely a bad choice from our side, I guess it was just a basic scaffolding since most of the time nodes are joining with a dedicated kubeadm configuration file (e.g.: Cluster API).

Wondering if you're facing this issue even tho running with Cluster API, IIRC a local configuration has precedence over the one stored in the cluster.

fix kamaji garbage collection (#263)

Navigation Menu

Repository navigation

Image garbage collection is not working on workers #508

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	EvictionPressureTransitionPeriod: zeroDuration,
	FileCheckFrequency: zeroDuration,
	HealthzBindAddress: "127.0.0.1",
	HealthzPort: pointer.To(int32(10248)),
	HTTPCheckFrequency: zeroDuration,
	ImageGCHighThresholdPercent: pointer.To(int32(100)),
	NodeStatusUpdateFrequency: zeroDuration,
	NodeStatusReportFrequency: zeroDuration,
	RotateCertificates: true,
	RuntimeRequestTimeout: zeroDuration,
	ShutdownGracePeriod: zeroDuration,
	ShutdownGracePeriodCriticalPods: zeroDuration,
	StaticPodPath: "/etc/kubernetes/manifests",
	StreamingConnectionIdleTimeout: zeroDuration,
	SyncFrequency: zeroDuration,
	VolumeStatsAggPeriod: zeroDuration,

Image garbage collection is not working on workers #508

Description

Activity

kvaps commented on Aug 7, 2024

prometherion commented on Aug 7, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions