Skip to content

fix: remove stale COSI socket before binding - #8

Merged
chrislusf merged 1 commit into
seaweedfs:mainfrom
lexfrei:fix/remove-stale-cosi-socket
Jun 2, 2026
Merged

chrislusf merged 1 commit into
seaweedfs:mainfrom
lexfrei:fix/remove-stale-cosi-socket

Conversation

@lexfrei

@lexfrei lexfrei commented Jun 2, 2026 •

Copy link
Copy Markdown
Contributor

Problem

The COSI gRPC socket (unix:///var/lib/cosi/cosi.sock by default) is created on an emptyDir shared between the driver and the COSI sidecar. Go's net package only unlinks a unix socket on a graceful listener close, so a non-graceful exit (SIGKILL / OOM / panic) leaves the socket file behind. The emptyDir survives container restarts, so the next start fails to bind:

listen unix /var/lib/cosi/cosi.sock: bind: address already in use

A single transient crash then wedges the pod in a permanent CrashLoopBackOff, recoverable only by deleting the pod, and object storage stays unusable in the meantime.

This fix comes from Cozystack, a CNCF Sandbox project that ships seaweedfs-cosi-driver as the COSI provisioner for its built-in S3 offering. It hits real users in production, so we are carrying the patch as a vendored fix in Cozystack and upstreaming it here so the whole driver community benefits. Tracked downstream in cozystack/cozystack#2429 (72 driver restarts observed on a single pod).

Fix

Remove a stale socket before starting the provisioner server, so the driver self-heals on restart.

  • The endpoint is parsed with net/url — the same way provisioner.Run derives the bind path (url.Parse(addr).Path) — so the path removed is exactly the path that will be bound, including for non-default endpoints.
  • Removal is guarded: it deletes the path only if it exists, is a socket (os.ModeSocket), and has no live listener (a short net.DialTimeout probe). It never deletes a regular file/directory or a socket that is still being served.

Tests

Adds cmd/seaweedfs-cosi-driver/main_test.go covering: stale socket removed, live socket kept, absent path no-op, regular file kept, non-unix endpoint ignored, and the url.Path parsing contract.

go test, go vet, go build, gofmt, and golangci-lint all pass locally.

Summary by CodeRabbit

  • Bug Fixes

    • Improved service restart reliability by automatically cleaning up stale socket files, eliminating manual intervention requirements.
  • Tests

    • Added comprehensive test coverage for socket cleanup logic.

@coderabbitai

coderabbitai Bot commented Jun 2, 2026 •

Copy link
Copy Markdown

Review Change Stack

Warning

Review limit reached

@lexfrei, we couldn't start this review because you've reached your PR review rate limit.

More reviews will be available in 50 minutes and 27 seconds. Learn how PR review limits work.

Your organization has run out of usage credits. Purchase more in the billing tab.

⌛ How to resolve this issue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans include higher PR review limits than trial, open-source, and free plans. In all cases, reviews become available again over time. During sustained high-volume PR review activity, CodeRabbit may temporarily slow when the next review becomes available.

Please see our Fair Usage Limits Policy for further information.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 3662be53-51d6-49f9-9eaa-723416010e32

📥 Commits

Reviewing files that changed from the base of the PR and between f50a9d9 and 60f5672.

📒 Files selected for processing (3)
  • .gitignore
  • cmd/seaweedfs-cosi-driver/main.go
  • cmd/seaweedfs-cosi-driver/main_test.go
📝 Walkthrough

Walkthrough

The PR adds graceful handling of stale Unix domain sockets in the COSI driver startup. A new removeStaleSocket function parses the endpoint, detects dead sockets via a short connection probe, and removes them safely. Comprehensive tests verify correct behavior across stale sockets, live listeners, missing paths, regular files, and non-unix endpoints. The build artifact is added to .gitignore.

Changes

Stale Socket Cleanup

Layer / File(s) Summary
Socket cleanup implementation and integration
cmd/seaweedfs-cosi-driver/main.go, .gitignore
Adds removeStaleSocket function that parses unix:// endpoints, checks for socket file existence, probes for active listeners with a dial timeout, and removes only confirmed stale sockets while preserving live ones. Integrates into run before gRPC server startup. Updates imports for url parsing and networking. Replaces ioutil.ReadFile with os.ReadFile. Adds driver binary to .gitignore.
Socket cleanup test coverage
cmd/seaweedfs-cosi-driver/main_test.go
Six unit tests verify removeStaleSocket removes dead sockets, preserves live listeners, treats missing sockets as no-op, leaves regular files intact, ignores non-unix endpoints, and correctly parses absolute-path unix:// URLs.

🎯 2 (Simple) | ⏱️ ~10 minutes

🐰 A socket once stale, now swept away,
The driver starts fresh, come what may,
No lingering ghosts block the gRPC way,
Test by test, we're here to stay! ✨

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 77.78% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title directly and concisely describes the main change: removing a stale COSI socket before binding, which is the core fix in this changeset.
Description check ✅ Passed The description includes a clear Problem section, detailed Fix explanation, Tests section, and tool verification, but lacks Issue ticket number/link and Checklist.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a self-healing mechanism to clean up stale COSI Unix domain sockets on startup, preventing CrashLoopBackOff issues when the driver restarts after a non-graceful exit. It also replaces deprecated ioutil.ReadFile with os.ReadFile and adds comprehensive unit tests for the socket cleanup logic. The reviewer suggests improving the robustness of the socket cleanup function by separating the URL parsing error check from the scheme check and explicitly checking for an empty path before calling os.Stat.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment on lines +112 to +118
u, err := url.Parse(endpoint)
if err != nil || u.Scheme != "unix" {
return nil
}
path := u.Path

fi, err := os.Stat(path)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

For better readability, safety, and defensive programming, we should:

  1. Separate the error check from the scheme check to avoid any potential nil-pointer dereference concerns if the condition is refactored in the future.
  2. Add an explicit check for an empty path to avoid calling os.Stat("") unnecessarily, which would otherwise perform a redundant system call and rely on OS-specific behavior for empty paths.
Suggested change
u, err := url.Parse(endpoint)
if err != nil || u.Scheme != "unix" {
return nil
}
path := u.Path
fi, err := os.Stat(path)
u, err := url.Parse(endpoint)
if err != nil {
return nil
}
if u.Scheme != "unix" {
return nil
}
path := u.Path
if path == "" {
return nil
}
fi, err := os.Stat(path)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Applied in 60f5672 — split the url.Parse error check from the scheme check and added an explicit empty-path guard before os.Stat (skips a redundant os.Stat("")). Also added a unix:// no-path unit test. Thanks!

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (3)
cmd/seaweedfs-cosi-driver/main_test.go (1)

30-30: 💤 Low value

noctx: net.Listen in tests is flagged by golangci-lint.

These calls are appropriate for creating real unix sockets in tests, but the static analyzer reports noctx errors. If CI enforces this linter, switch to (&net.ListenConfig{}).Listen(ctx, "unix", sock) or exclude _test.go files from the noctx rule. See the .golangci config verification requested on main.go.

Also applies to: 55-55, 107-107

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@cmd/seaweedfs-cosi-driver/main_test.go` at line 30, Replace direct
net.Listen("unix", sock) calls in the test file with context-aware ListenConfig
to satisfy golangci-lint noctx: create a context (e.g., ctx :=
context.Background() or a cancellable/timeout context) and call
(&net.ListenConfig{}).Listen(ctx, "unix", sock) instead, and add the context
import; alternatively, if you prefer lint config changes, exclude _test.go from
the noctx rule in .golangci.yml. Update all occurrences of net.Listen in this
file (the three test call sites) and their imports accordingly.
cmd/seaweedfs-cosi-driver/main.go (2)

129-134: 💤 Low value

noctx: replace net.DialTimeout with context-based (*net.Dialer).DialContext

net.DialTimeout is still supported (not deprecated), but the noctx rule expects the context-aware dialing API.

♻️ Optional: context-aware dial
-	if c, err := net.DialTimeout("unix", path, 200*time.Millisecond); err == nil {
+	ctx, cancel := context.WithTimeout(context.Background(), 200*time.Millisecond)
+	defer cancel()
+	var d net.Dialer
+	if c, err := d.DialContext(ctx, "unix", path); err == nil {
 		// A live listener is still bound — leave it. The bind that follows will
 		// surface the real "address already in use" if another instance runs.
 		_ = c.Close()
 		return nil
 	}
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@cmd/seaweedfs-cosi-driver/main.go` around lines 129 - 134, Replace the
blocking call to net.DialTimeout with a context-aware dial: create a context
with timeout (e.g., ctx, cancel := context.WithTimeout(context.Background(),
200*time.Millisecond) and defer cancel()) and call
(&net.Dialer{}).DialContext(ctx, "unix", path) instead of net.DialTimeout; keep
the existing nil-check and _ = c.Close()/return nil logic unchanged so a live
listener still short-circuits the bind attempt. Ensure you import context if not
already present and preserve the 200*time.Millisecond timeout value.

111-116: 💤 Low value

removeStaleSocket: document the intentional nil on malformed unix endpoints (CI likely won’t flag nilerr)

url.Parse error being a no-op is reasonable, and the || preserves safety for u.Scheme. This repo’s CI runs golangci-lint via pre-commit (v2.10.1) without a .golangci.* config, and nilerr isn’t enabled by default in the standard preset—so the code should not surface as nilerr in this setup unless CI uses an explicit preset/config that enables it. For readability/resilience, split the parse-error vs non-unix checks and add a brief comment; avoid //nolint:nilerr unless nilerr is actually active.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@cmd/seaweedfs-cosi-driver/main.go` around lines 111 - 116, In
removeStaleSocket: split the combined check so we explicitly handle a parse
error and non-unix schemes separately and add a short comment explaining that
returning nil on a malformed or non-unix endpoint is intentional; e.g., call
url.Parse(endpoint), if err != nil { /* malformed endpoint or not relevant to
unix sockets: nothing to remove */ return nil }, then if u.Scheme != "unix" {
return nil }, then proceed to use u.Path — do not add a //nolint:nilerr unless
nilerr is enabled.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@cmd/seaweedfs-cosi-driver/main_test.go`:
- Line 30: Replace direct net.Listen("unix", sock) calls in the test file with
context-aware ListenConfig to satisfy golangci-lint noctx: create a context
(e.g., ctx := context.Background() or a cancellable/timeout context) and call
(&net.ListenConfig{}).Listen(ctx, "unix", sock) instead, and add the context
import; alternatively, if you prefer lint config changes, exclude _test.go from
the noctx rule in .golangci.yml. Update all occurrences of net.Listen in this
file (the three test call sites) and their imports accordingly.

In `@cmd/seaweedfs-cosi-driver/main.go`:
- Around line 129-134: Replace the blocking call to net.DialTimeout with a
context-aware dial: create a context with timeout (e.g., ctx, cancel :=
context.WithTimeout(context.Background(), 200*time.Millisecond) and defer
cancel()) and call (&net.Dialer{}).DialContext(ctx, "unix", path) instead of
net.DialTimeout; keep the existing nil-check and _ = c.Close()/return nil logic
unchanged so a live listener still short-circuits the bind attempt. Ensure you
import context if not already present and preserve the 200*time.Millisecond
timeout value.
- Around line 111-116: In removeStaleSocket: split the combined check so we
explicitly handle a parse error and non-unix schemes separately and add a short
comment explaining that returning nil on a malformed or non-unix endpoint is
intentional; e.g., call url.Parse(endpoint), if err != nil { /* malformed
endpoint or not relevant to unix sockets: nothing to remove */ return nil },
then if u.Scheme != "unix" { return nil }, then proceed to use u.Path — do not
add a //nolint:nilerr unless nilerr is enabled.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 0885ab31-4189-4249-901e-5da6108481ba

📥 Commits

Reviewing files that changed from the base of the PR and between 49c6808 and f50a9d9.

📒 Files selected for processing (3)
  • .gitignore
  • cmd/seaweedfs-cosi-driver/main.go
  • cmd/seaweedfs-cosi-driver/main_test.go

A non-graceful driver exit (SIGKILL/OOM/panic) leaves the COSI unix socket
file behind — Go's net package only unlinks it on a graceful listener close.
The socket lives on an emptyDir shared with the sidecar that survives container
restarts, so the next net.Listen("unix", …) fails with EADDRINUSE and wedges
the pod in CrashLoopBackOff ("bind: address already in use"), recoverable only
by deleting the pod.

Remove a stale socket before starting the provisioner server so the driver
self-heals on restart. The endpoint is parsed with net/url (matching the path
the provisioner sidecar binds), and removal is guarded: it deletes only a path
that exists, is a socket, and has no live listener — never a regular file or a
socket that is still being served.

Assisted-By: Claude <[email protected]>
Signed-off-by: Aleksei Sviridkin <[email protected]>
@lexfrei
lexfrei force-pushed the fix/remove-stale-cosi-socket branch from f50a9d9 to 60f5672 Compare June 2, 2026 18:11
@chrislusf
chrislusf merged commit badc925 into seaweedfs:main Jun 2, 2026
2 checks passed
lexfrei added a commit to cozystack/cozystack that referenced this pull request Jun 5, 2026
## What this PR does

Bumps the `seaweedfs-cosi-driver` image from `v0.3.0` to `v0.3.1`.

`v0.3.1` ships the stale-socket self-heal fix. After a non-graceful exit
(SIGKILL / OOM / panic) Go does not unlink the COSI unix socket, and
because it lives on an `emptyDir` that survives container restarts, the
next start failed with `bind: address already in use` and the
`objectstorage-provisioner` wedged in `CrashLoopBackOff`. While it was
down, `BucketClaim` reconciliation stopped and buckets were never
provisioned — the COSI `Bucket` could even report `bucketReady: true`
while no bucket existed. This is the CrashLoopBackOff reported in #2429.

The driver now removes a leftover socket before binding (only when it
exists, is a socket, and has no live listener), so the provisioner
self-heals on restart.

Upstream: seaweedfs/seaweedfs-cosi-driver#8 (merged), released in
v0.3.1. Points at the upstream published multi-arch image (amd64 +
arm64) — no vendoring needed.

### Release note

```release-note
fix(seaweedfs): bump seaweedfs-cosi-driver to v0.3.1, which self-heals the object storage COSI provisioner from a CrashLoopBackOff caused by a stale unix socket
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Chores**
  * Updated SeaweedFS cosi driver container image to a newer version.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants