Repository navigation
serve restic: DELETE replies 404 when the object lookup fails, so restic leaves its lock behind #10100
Description
Before you start
- I have searched the forum and the existing issues for this problem.
- I have tested with the latest beta or stable release and can still reproduce the problem.
- This is a reproducible bug, not a usage question (questions belong on the forum).
Associated forum post URL
No response
What is the problem you are having with rclone?
When rclone serve restic receives a DELETE request and the lookup of the object fails with an error other than "not found", it replies 404 Not Found. A GET request with the same lookup error replies 500 Internal Server Error.
restic treats a 404 on delete as final and does not retry. When this happens while restic removes its lock, the lock stays in the repository and the next exclusive operation (for example forget --prune) fails with "repository is already locked". The real error is logged only at debug level.
We hit this in production with a Backblaze B2 remote: a temporary error from B2 while restic removed its lock made the following forget --prune fail. The steps below use a local remote and a permission error, to get the same lookup failure deterministically.
In cmd/serve/restic/restic.go, deleteObject() replies 404 for any error returned by s.newObject(), while serveObject() replies 404 only for fs.ErrorObjectNotFound and 500 otherwise. Replying 500 for other errors in deleteObject() too would let restic retry.
Steps to reproduce (run as root with Podman; rclone runs as uid 1000, so an unreadable locks directory gives a lookup error that is not "not found"):
D=/tmp/rclone-repro; URL=http://127.0.0.1:8080/repo
mkdir -p $D/data && chown -R 1000:1000 $D
podman run -d --name rclone-repro --user 1000:1000 --network host -v $D/data:/data:Z \
docker.io/rclone/rclone:1.75.2 serve restic --addr 127.0.0.1:8080 --cache-objects=false -vv /data
restic() { podman run --rm -i --network host -e RESTIC_REPOSITORY=rest:$URL -e RESTIC_PASSWORD=test \
docker.io/restic/restic:0.19.1 "$@"; }
restic init -q
# 1. Same lookup error: GET replies 500, DELETE replies 404
id=$(printf 'a%.0s' {1..64})
curl -s -o /dev/null -w "POST %{http_code}\n" --data-binary test -X POST $URL/locks/$id
chmod 000 $D/data/repo/locks
curl -s -o /dev/null -w "GET %{http_code}\n" $URL/locks/$id
curl -s -o /dev/null -w "DELETE %{http_code}\n" -X DELETE $URL/locks/$id
chmod 755 $D/data/repo/locks; ls $D/data/repo/locks; rm $D/data/repo/locks/$id
# 2. restic loses its lock: make locks/ unreadable while a backup runs
( sleep 10; echo hello ) | restic backup -q --stdin --stdin-filename hello.txt &
sleep 5; chmod 000 $D/data/repo/locks; wait; chmod 755 $D/data/repo/locks
restic list locks
restic forget -q --keep-last 1 --pruneOutput:
POST 200
GET 500
DELETE 404
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
Remove(<lock/4a01776535>) failed: <lock/4a01776535> does not exist
error while unlocking: <lock/4a01776535> does not exist
4a01776535f330e29acc9f000545d882751de8e1ec243a6736ced257db95b05b
unable to create lock in backend: repository is already locked by PID 1 on rl2 by root (UID 0, GID 0)
lock was created at 2026-10-16 09:30:28 (10.715321392s ago)
storage ID 4a017765
the `unlock` command can be used to remove stale locks
Expected: DELETE replies 500 like GET, restic retries the removal, and no lock is left behind.
Actual: DELETE replies 404 while the file is still there, restic reports the lock as already gone ("does not exist") and the repository stays locked.
rclone version
rclone v1.75.2
- os/version: alpine 3.24.2 (64 bit)
- os/kernel: 5.14.0-687.49.1.el9_8.x86_64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.27.2
- go/linking: static
- go/tags: none
Operating system
Linux
Which cloud storage system are you using?
Backblaze B2 in production; local disk in the steps above
Output of rclone config redacted
; empty configThe command you were trying to run
rclone serve restic --addr 127.0.0.1:8080 --cache-objects=false -vv /dataA log from the command with the -vv flag
2026/10/16 09:30:25 DEBUG : rclone: Version "v1.75.2" starting with parameters ["rclone" "serve" "restic" "--addr" "127.0.0.1:8080" "--cache-objects=false" "-vv" "/data"]
2026/10/16 09:30:25 DEBUG : Creating backend with remote "/data"
2026/10/16 09:30:25 NOTICE: Config file "/data/.rclone.conf" not found - using defaults
2026/10/16 09:30:25 NOTICE: Local file system at /data: Serving restic REST API on [http://127.0.0.1:8080/]
2026/10/16 09:30:26 DEBUG : repo/config: HEAD request error: object not found
2026/10/16 09:30:26 INFO : repo: Creating repository
2026/10/16 09:30:26 DEBUG : repo/config: HEAD request error: object not found
2026/10/16 09:30:26 DEBUG : repo/keys: list request
2026/10/16 09:30:26 DEBUG : repo/snapshots: list request
2026/10/16 09:30:27 DEBUG : repo/keys/e87593295bb9acd0f0192857ff2f32f98f10ca6bafa799d5b83e8326e7d9e7a8: size = 440 OK
2026/10/16 09:30:27 DEBUG : repo/config: size = 155 OK
2026/10/16 09:30:27 DEBUG : repo/locks/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa: size = 4 OK
2026/10/16 09:30:27 DEBUG : repo/locks/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa: GET request error: permission denied
2026/10/16 09:30:27 DEBUG : repo/locks/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa: Delete request error: permission denied
2026/10/16 09:30:27 DEBUG : repo/keys: list request
2026/10/16 09:30:28 DEBUG : repo/locks: list request
2026/10/16 09:30:28 DEBUG : repo/locks/4a01776535f330e29acc9f000545d882751de8e1ec243a6736ced257db95b05b: size = 144 OK
2026/10/16 09:30:28 DEBUG : repo/locks: list request
2026/10/16 09:30:28 DEBUG : repo/index: list request
2026/10/16 09:30:37 DEBUG : repo/data/50/50e066f2d11adaf2dadd787d83d61d80f4bb872b4eac63fd13dfb621982b0a5d: size = 290 OK
2026/10/16 09:30:37 DEBUG : repo/data/c5/c5cbab08231ba67946df84a5849d7f838c21699b3799c24d7a79355ead74d473: size = 124 OK
2026/10/16 09:30:37 DEBUG : repo/index/6670202d2d709a5f8b718c2a85ed5957cd3f0ef603eee2247acb852333bb7fe0: size = 309 OK
2026/10/16 09:30:37 DEBUG : repo/snapshots/92c9f298223ca0ff9da1a1e31d9e7baf2a0ffb31a424c79bc30ddbe2ff99fd78: size = 347 OK
2026/10/16 09:30:37 DEBUG : repo/locks/4a01776535f330e29acc9f000545d882751de8e1ec243a6736ced257db95b05b: Delete request error: permission denied
2026/10/16 09:30:37 DEBUG : repo/keys: list request
2026/10/16 09:30:38 DEBUG : repo/locks: list request
2026/10/16 09:30:38 DEBUG : repo/keys: list request
2026/10/16 09:30:38 DEBUG : repo/locks: list request
2026/10/16 09:30:38 DEBUG : repo/locks: list requestAnything else?
Related: #8785 (locks through serve restic, but a different cause: eventual consistency of the Mega backend).
Activity
bunnysayzz commentedon Oct 17, 2026
Root cause:
deleteObject()in cmd/serve/restic/restic.go returned 404 for any error fromnewObject(), whileserveObject()in the same file only returns 404 forfs.ErrorObjectNotFoundand 500 for everything else. So a transient lookup failure on DELETE looked final to restic, it never retried, and the lock stayed behind.Fix mirrors the
serveObject()mapping indeleteObject(), with a regression test (TestDeleteErrors) covering both cases: generic lookup error -> 500, not-found -> 404. Verified the test fails on the old code (404) and passes with the fix.PR: #10109
ncw commentedon Oct 18, 2026
Thank you for the nice report @DavidePrincipi
I've merged @bunnysayzz fix for this to master now which means it will be in the latest beta in 15-30 minutes and released in v1.76
serve restic: fix temporary errors confusing restic into leaving lock…