Registry crash-loops on an empty storage directory: distribution's PurgeUploads panic is fixed upstream but Harbor's 2.8.x fork predates the fix #23795

Re-filing per the bot's instruction on #21172, which was closed as stale on 2025-02-17 rather than fixed. There is no open issue for this now, and since then the underlying bug has been fixed upstream in distribution — so this is narrower and more actionable than it was in 2024.

Symptom

harbor-registry-registry (Harbor 2.15.1, registry reports v2.8.3-25-g086d3c33, go1.25.9) restarted 67 times in 30 hours, exit code 2, backed by a SeaweedFS S3 gateway.

panic: runtime error: index out of range [0] with length 0
  storage.getOutstandingUploads.func1   purgeuploads.go:73
  s3-aws.(*driver).doWalk.func1         s3.go
  ...
  storage.PurgeUploads                  purgeuploads.go:34
  handlers.startUploadPurger.func1      app.go:1130

Worth flagging because it makes this hard to notice: the pod reads 2/2 Running with both containers ready=true the whole time, serving 200s between crashes. Only restartCount shows it. KubePodCrashLooping never fires either, because the purge fires after a jitter drawn at start-up (we observed 1m, 15m, 32m), the container runs longer than the 10-minute kubelet backoff-reset window, and so it never enters CrashLoopBackOff.

Root cause

getOutstandingUploads does:

_, file := path.Split(filePath)
if file[0] == '_' {          // panics when file == ""

path.Split returns an empty file for a path ending in /. The S3 driver trims the trailing slash for CommonPrefixes (commonPrefix[:len(commonPrefix)-1]) but not for Contents (strings.Replace(*file.Key, d.s3Path(""), prefix, 1)), so a Contents key ending in / reaches the walk callback unmodified.

On AWS such a key cannot exist — a prefix with no objects under it is not returned. SeaweedFS's filer has real directory inodes, so deleting the last object leaves the directory behind, and its S3 gateway then reports that directory as a zero-byte Contents key ending in /. Reproduced directly:

$ aws s3api list-objects-v2 --bucket <b> --delimiter '/' \
    --prefix 'docker/registry/v2/repositories/<repo>/_uploads/<uuid>/hashstates/sha256/'
{"Contents":[{"Key":"docker/.../hashstates/sha256/","Size":0}],"CommonPrefixes":[]}

These directories are invisible to the obvious check: a full recursive listing (2,682 keys) returns zero keys ending in /, because the gateway only synthesises the directory entry under a delimited list. We found them by diffing the filer's directory listing against the object listing.

Why this is a Harbor issue rather than a distribution one

distribution already fixed it. distribution/distribution#4713 ("PurgeUploads panic", closed 2026-04-22) added the guard, and it is present in main and in v3.1.1:

// path.Split returns file == "" for paths that end in "/" ...
if len(file) > 0 && file[0] == '_' {

Harbor's registry still reports v2.8.3-25-g086d3c33 — distribution 2.8.x plus patches — and release/2.8 is unguarded. So current Harbor ships the panic even though upstream has shipped the fix.

Ask

Cherry-pick the one-line len(file) > 0 guard into the registry fork Harbor builds. Moving Harbor's registry to distribution v3.x is obviously a much larger discussion; this is just the guard.

Reproduction

Any Harbor on an S3 implementation that materialises directories will hit it. Minimal recipe against such a backend:

1. PUT an object at docker/registry/v2/repositories/<repo>/_uploads/<uuid>/hashstates/sha256/0
2. DELETE that object — the directory chain remains
3. Wait for the upload purger to run (or restart the registry and wait out the jitter)

In practice step 1–2 is just an aborted or interrupted blob push. We accumulated 15 such directories across 3 repositories over four months; 13 dated from the day the registry was first populated.

Impact and workaround

Registry availability was ~99% — each crash drops in-flight pushes/pulls and recovery is seconds — so this is disruptive rather than fatal. The workaround is deleting the empty directories, which must be done at the _uploads/<uuid> level: removing only the empty leaf empties its parent and recreates the identical panic one level up.

Notably there is no way to disable the purger as a stopgap: storage.maintenance.uploadpurging lives in the registry ConfigMap, which is Helm-owned and reverts on reconcile, and dryrun: true does not help because the panic happens during the walk, before any delete.