Skip to content

Conversation

@awskii
Copy link
Member

@awskii awskii commented Oct 14, 2025

WARN [10-14|08:26:39.496] error reading inbox                      err="failed to get blobs: error fetching blobs in 9408714 l1 block: beacon client in blobSidecars got error or empty response fetching non-expired blobs in slo
t: 8724651, if using a Prysm endpoint, try --enable-experimental-backfill flag, err: response returned with status 404 Not Found, want 200 OK. body: {\"code\":404,\"message\":\"blobs are not supported after fulu fork\"}\n

nitro still tries to call /blobs/blob_sidecars which is disabled after fulu.

@awskii awskii requested review from Giulio2002 and domiwei October 14, 2025 15:21
@awskii
Copy link
Member Author

awskii commented Oct 14, 2025

@awskii
Copy link
Member Author

awskii commented Oct 14, 2025

going to check later and then close this pr

@awskii
Copy link
Member Author

awskii commented Oct 15, 2025

@awskii awskii closed this Oct 15, 2025
@awskii awskii deleted the arb-fulu-fix branch October 15, 2025 04:12
@awskii awskii restored the arb-fulu-fix branch October 15, 2025 04:27
@awskii
Copy link
Member Author

awskii commented Oct 15, 2025

after 20 minutes of correct operation see:

WARN [10-15|04:25:55.657] error reading inbox                      err="failed to get blobs: error fetching blobs for slot 8730642: beacon client in getBlobs got error fetching non-expired blobs in slot: 8730642, err: Get \"http://l1-erigon:9093/eth/v1/beacon/blobs/8730642?versioned_hashes=0x018cd0ddfb47915268e7140c0b369f44274c0aa6983da632ac1cc24492cebe6a&versioned_hashes=0x01e3d71faab36258c622f976de99eeeb7c61d69caf59f67041ad7ce832a9d008&versioned_hashes=0x01146
80c38dc93c0090ce2b55b07a57eb8f14ba36a15222fb9d05944453b4385&versioned_hashes=0x01f39a8bb86a0080d1d35fad5eb09b5260e54bc44cfc4800fa71e0c2865bb657&versioned_hashes=0x0120d99d3b95c8fcf25bd8dec36e5be7e48b1901cd692f16f918072874396035\": EOF"
l2-nitro  | ERROR[10-15|04:25:55.702] ExecuteNextMsg failed to readMessage     err="failed to fetch batch mentioned by batch posting report: accumulator not found: no metadata for batch 605287" msgIdxToExecute=204761999
l2-nitro  | ERROR[10-15|04:25:55.723] Error routing message from Primary Sequencer Feeds err="failed to fetch batch mentioned by batch posting report: accumulator not found: no metadata for batch 605287"
l2-nitro  | ERROR[10-15|04:25:55.803] ExecuteNextMsg failed to readMessage     err="failed to fetch batch mentioned by batch posting report: accumulator not found: no metadata for batch 605287" msgIdxToExecute=204761999

@awskii awskii reopened this Oct 15, 2025
@jonathanudd
Copy link

I had some issues with op-node when connecting to Caplin, getting this error

Oct 15 05:03:47 juju-8111ad-0 op-node[622705]: t=2025-10-15T05:03:47+0000 lvl=warn msg="Deriver system is resetting" err="derivation failed: reset: failed to fetch blobs: failed to get blob sidecars for L1BlockRef 0x581bc6ad3f156c04c77a6772cb04cec0ffd92fb909d772d1b1af638f06568e62:9408578: failed to fetch blob sidecars for slot 8724483: failed request with status 404: {\"code\":404,\"message\":\"blobs are not supported after fulu fork\"}\n: not found"

But if I build Erigon from this branch and connect the same op-node it works.
I would prefer not running a "non-release" version of Erigon though.

What should be the next steps here?
Should this be fixed and patched in Erigon or in op-node?
I have made an issue in the op-node repo as well ethereum-optimism/optimism#17871

@air3ijai
Copy link

air3ijai commented Oct 15, 2025

Docker build from that PR head branch

git clone https://github.com/erigontech/erigon.git
cd erigon

git switch -C arb-fulu-fix origin/arb-fulu-fix

docker build -t erigon-arb-fulu-fix-c550d653f4 -f Dockerfile .
Screenshot 2025-10-15 at 14 32 20
more logs
erigon-sepolia  | [INFO] [10-15|11:31:48.608] [Sentinel] Update ENR on subscription    subnet=37 subscribe=true type=attestation
erigon-sepolia  | [INFO] [10-15|11:31:48.609] [Sentinel] Update ENR on subscription    subnet=60 subscribe=true type=attestation
erigon-sepolia  | [INFO] [10-15|11:31:49.688] Started Ethereum 2.0 Gossip Service      app=caplin
erigon-sepolia  | [INFO] [10-15|11:31:49.689] Beacon API started                       addr=0.0.0.0:5555
erigon-sepolia  | [INFO] [10-15|11:31:49.689] [Caplin] starting clstages loop          app=caplin
erigon-sepolia  | [INFO] [10-15|11:31:49.689] Starting downloading History             app=caplin stage=DownloadHistoricalBlocks from=8732448
erigon-sepolia  | 2025/10/15 11:31:50 http: panic serving 172.18.0.3:45550: runtime error: invalid memory address or nil pointer dereference
erigon-sepolia  | goroutine 371513 [running]:
erigon-sepolia  | net/http.(*conn).serve.func1()
erigon-sepolia  | 	net/http/server.go:1947 +0xbe
erigon-sepolia  | panic({0x295ffa0?, 0x5194a70?})
erigon-sepolia  | 	runtime/panic.go:792 +0x132
erigon-sepolia  | github.com/erigontech/erigon/cl/beacon/handler.(*ApiHandler).GetEthV1BeaconBlobs(0xc08120b408, {0x0?, 0x5?}, 0xc08a6f9540)
erigon-sepolia  | 	github.com/erigontech/erigon/cl/beacon/handler/blobs.go:215 +0x888
erigon-sepolia  | github.com/erigontech/erigon/cl/beacon/beaconhttp.EndpointHandlerFunc[...].Handle(...)
erigon-sepolia  | 	github.com/erigontech/erigon/cl/beacon/beaconhttp/api.go:94
erigon-sepolia  | github.com/erigontech/erigon/cl/beacon/handler.(*ApiHandler).init.func2.1.3.HandleEndpointFunc[...].8(0xc08a6f9540)
erigon-sepolia  | 	github.com/erigontech/erigon/cl/beacon/beaconhttp/api.go:103 +0x58
erigon-sepolia  | net/http.HandlerFunc.ServeHTTP(0xc088393590?, {0x37e06c0?, 0xc0894f8c40?}, 0xc0868f12c0?)
erigon-sepolia  | 	net/http/server.go:2294 +0x29
erigon-sepolia  | github.com/go-chi/chi/v5.(*Mux).routeHTTP(0xc08a67f380, {0x37e06c0, 0xc0894f8c40}, 0xc08a6f9540)
erigon-sepolia  | 	github.com/go-chi/chi/[email protected]/mux.go:477 +0x31c
erigon-sepolia  | net/http.HandlerFunc.ServeHTTP(0xc08dbaff50?, {0x37e06c0?, 0xc0894f8c40?}, 0xc0a1029218?)
erigon-sepolia  | 	net/http/server.go:2294 +0x29
erigon-sepolia  | github.com/go-chi/chi/v5.(*Mux).ServeHTTP(0xc08a67f380, {0x37e06c0, 0xc0894f8c40}, 0xc08a6f9540)
erigon-sepolia  | 	github.com/go-chi/chi/[email protected]/mux.go:73 +0x32f
erigon-sepolia  | github.com/go-chi/chi/v5.(*Mux).Mount.func1({0x37e06c0, 0xc0894f8c40}, 0xc08a6f9540)
erigon-sepolia  | 	github.com/go-chi/chi/[email protected]/mux.go:321 +0x1bb
erigon-sepolia  | net/http.HandlerFunc.ServeHTTP(0xc088393590?, {0x37e06c0?, 0xc0894f8c40?}, 0xc08d145ad0?)
erigon-sepolia  | 	net/http/server.go:2294 +0x29
erigon-sepolia  | github.com/go-chi/chi/v5.(*Mux).routeHTTP(0xc08a67f260, {0x37e06c0, 0xc0894f8c40}, 0xc08a6f9540)
erigon-sepolia  | 	github.com/go-chi/chi/[email protected]/mux.go:477 +0x31c
erigon-sepolia  | net/http.HandlerFunc.ServeHTTP(0xc08dbaff50?, {0x37e06c0?, 0xc0894f8c40?}, 0xc0a10293e8?)
erigon-sepolia  | 	net/http/server.go:2294 +0x29
erigon-sepolia  | github.com/go-chi/chi/v5.(*Mux).ServeHTTP(0xc08a67f260, {0x37e06c0, 0xc0894f8c40}, 0xc08a6f9540)
erigon-sepolia  | 	github.com/go-chi/chi/[email protected]/mux.go:73 +0x32f
erigon-sepolia  | github.com/go-chi/chi/v5.(*Mux).Mount.func1({0x37e06c0, 0xc0894f8c40}, 0xc08a6f9540)
erigon-sepolia  | 	github.com/go-chi/chi/[email protected]/mux.go:321 +0x1bb
erigon-sepolia  | net/http.HandlerFunc.ServeHTTP(0xc088393590?, {0x37e06c0?, 0xc0894f8c40?}, 0xc08d145aa0?)
erigon-sepolia  | 	net/http/server.go:2294 +0x29
erigon-sepolia  | github.com/go-chi/chi/v5.(*Mux).routeHTTP(0xc08a67f200, {0x37e06c0, 0xc0894f8c40}, 0xc08a6f9540)
erigon-sepolia  | 	github.com/go-chi/chi/[email protected]/mux.go:477 +0x31c
erigon-sepolia  | net/http.HandlerFunc.ServeHTTP(0xc08dbaff50?, {0x37e06c0?, 0xc0894f8c40?}, 0xc0a10295b8?)
erigon-sepolia  | 	net/http/server.go:2294 +0x29
erigon-sepolia  | github.com/go-chi/chi/v5.(*Mux).ServeHTTP(0xc08a67f200, {0x37e06c0, 0xc0894f8c40}, 0xc08a6f9540)
erigon-sepolia  | 	github.com/go-chi/chi/[email protected]/mux.go:73 +0x32f
erigon-sepolia  | github.com/go-chi/chi/v5.(*Mux).Mount.func1({0x37e06c0, 0xc0894f8c40}, 0xc08a6f9540)
erigon-sepolia  | 	github.com/go-chi/chi/[email protected]/mux.go:321 +0x1bb
erigon-sepolia  | net/http.HandlerFunc.ServeHTTP(0xc088393590?, {0x37e06c0?, 0xc0894f8c40?}, 0xc084cafa44?)
erigon-sepolia  | 	net/http/server.go:2294 +0x29
erigon-sepolia  | github.com/go-chi/chi/v5.(*Mux).routeHTTP(0xc08a67f140, {0x37e06c0, 0xc0894f8c40}, 0xc08a6f9540)
erigon-sepolia  | 	github.com/go-chi/chi/[email protected]/mux.go:477 +0x31c
erigon-sepolia  | net/http.HandlerFunc.ServeHTTP(0xc08dbaff50?, {0x37e06c0?, 0xc0894f8c40?}, 0x7b19395bd108?)
erigon-sepolia  | 	net/http/server.go:2294 +0x29
erigon-sepolia  | github.com/go-chi/chi/v5.(*Mux).ServeHTTP(0xc08a67f140, {0x37e06c0, 0xc0894f8c40}, 0xc08a6f9540)
erigon-sepolia  | 	github.com/go-chi/chi/[email protected]/mux.go:73 +0x32f
erigon-sepolia  | github.com/erigontech/erigon/cl/beacon/handler.(*ApiHandler).ServeHTTP(0xc08120b408?, {0x37e06c0?, 0xc0894f8c40?}, 0xc08a6f9540?)
erigon-sepolia  | 	github.com/erigontech/erigon/cl/beacon/handler/handler.go:388 +0x6c
erigon-sepolia  | github.com/erigontech/erigon/cl/beacon.ListenAndServe.func1({0x37e06c0, 0xc0894f8c40}, 0xc08a6f9400)
erigon-sepolia  | 	github.com/erigontech/erigon/cl/beacon/router.go:59 +0x1dc
erigon-sepolia  | net/http.HandlerFunc.ServeHTTP(0xc0883934a0?, {0x37e06c0?, 0xc0894f8c40?}, 0xc084cafa44?)
erigon-sepolia  | 	net/http/server.go:2294 +0x29
erigon-sepolia  | github.com/go-chi/chi/v5.(*Mux).routeHTTP(0xc088134000, {0x37e06c0, 0xc0894f8c40}, 0xc08a6f9400)
erigon-sepolia  | 	github.com/go-chi/chi/[email protected]/mux.go:477 +0x31c
erigon-sepolia  | net/http.HandlerFunc.ServeHTTP(0xc089fde140?, {0x37e06c0?, 0xc0894f8c40?}, 0xc08a6f9400?)
erigon-sepolia  | 	net/http/server.go:2294 +0x29
erigon-sepolia  | github.com/go-chi/cors.(*Cors).Handler-fm.(*Cors).Handler.func1({0x37e06c0, 0xc0894f8c40}, 0xc08a6f9400)
erigon-sepolia  | 	github.com/go-chi/[email protected]/cors.go:228 +0x17e
erigon-sepolia  | net/http.HandlerFunc.ServeHTTP(0x37ea070?, {0x37e06c0?, 0xc0894f8c40?}, 0x519c4b0?)
erigon-sepolia  | 	net/http/server.go:2294 +0x29
erigon-sepolia  | github.com/go-chi/chi/v5.(*Mux).ServeHTTP(0xc088134000, {0x37e06c0, 0xc0894f8c40}, 0xc08a6f92c0)
erigon-sepolia  | 	github.com/go-chi/chi/[email protected]/mux.go:90 +0x2ee
erigon-sepolia  | net/http.serverHandler.ServeHTTP({0xc08dbafe00?}, {0x37e06c0?, 0xc0894f8c40?}, 0x1?)
erigon-sepolia  | 	net/http/server.go:3301 +0x8e
erigon-sepolia  | net/http.(*conn).serve(0xc08a870120, {0x37ea038, 0xc0800f0270})
erigon-sepolia  | 	net/http/server.go:2102 +0x625
erigon-sepolia  | created by net/http.(*Server).Serve in goroutine 371187
erigon-sepolia  | 	net/http/server.go:3454 +0x485
erigon-sepolia  | [INFO] [10-15|11:32:12.664] [Blobs-Downloader] Downloading blobs backwards app=caplin stage=DownloadHistoricalBlocks slot=8732449
erigon-sepolia  | [INFO] [10-15|11:32:14.692] Ready to insert history, waiting for sync cycle to finish app=caplin stage=DownloadHistoricalBlocks
erigon-sepolia  | [INFO] [10-15|11:32:14.692] [Caplin] Forward Sync                    app=caplin stage=ForwardSync from=8732448 to=8732861
erigon-sepolia  | [INFO] [10-15|11:32:17.771] Adding light client update               period=1065
erigon-sepolia  | [INFO] [10-15|11:32:47.246] [snapshots] Seeding                      peers=0 conns=0 upload=0B/s alloc=3.5GB sys=4.3GB
erigon-sepolia  | [INFO] [10-15|11:32:49.688] P2P                                      app=caplin peers=0
erigon-sepolia  | [INFO] [10-15|11:33:01.423] [Caplin] Forward Sync                    app=caplin stage=ForwardSync progress=8732651 distance-from-chain-tip=42m0s estimated-time-remaining=31s
erigon-sepolia  | [INFO] [10-15|11:33:02.591] Adding light client update               period=1066
erigon-sepolia  | [INFO] [10-15|11:33:17.243] [p2p] GoodPeers                          eth68=3 eth69=7

@air3ijai
Copy link

It solved the issue with Arbitrum Sepolia using v3.7.6 release - offchainlabs/nitro-node:v3.7.6-c0fe95e.

@awskii
Copy link
Member Author

awskii commented Oct 15, 2025

more sustainable solution in progress. until the release - can stick to this branch.

@awskii awskii closed this Oct 15, 2025
@gflow33
Copy link

gflow33 commented Oct 15, 2025

Upgraded to v3.7.6 and still getting a similar error

error reading inbox                      err="failed to get blobs: error fetching blobs for slot 8724651: beacon client in getBlobs got error fetching non-expired blobs in slot: 8724651, err: response returned with status 404 Not Found, want 200 OK. body: ady the case\",\"code\":404} "

@air3ijai
Copy link

air3ijai commented Oct 15, 2025

Upgraded to v3.7.6 and still getting a similar error

error reading inbox                      err="failed to get blobs: error fetching blobs for slot 8724651: beacon client in getBlobs got error fetching non-expired blobs in slot: 8724651, err: response returned with status 404 Not Found, want 200 OK. body: ady the case\",\"code\":404} "

This is how it works in our case on Arbitrum Sepolia.

erigon-sepolia-compose.yaml
services:
  erigon-sepolia:
    image: erigon-arb-fulu-fix-c550d653f4
    container_name: erigon-sepolia
    restart: unless-stopped
    user: 1000:1000
    ports:
      - 8545:8545       # JSON-RPC
      - 8546:8546       # WS-RPC
      - 5555:5555       # Beacon API
      - 30303:30303/tcp # eth/68 peering
      - 30303:30303/udp # eth/68 peering
      - 30304:30304/tcp # eth/67 peering
      - 30304:30304/udp # eth/67 peering
      - 4000:4000/udp   # Caplin DISCV5
      - 4001:4001/tcp   # Caplin DISCV5
      - 42069:42069/tcp # Snap sync (Bittorrent)
      - 42069:42069/udp # Snap sync (Bittorrent)
      - 6060:6060       # pprof
      - 6061:6061       # Metrics
    command:
      - --chain=sepolia
      - --datadir=/home/erigon/.local/share/erigon
      - --prune.mode=full
      - --txpool.disable=true
      - --http
      - --http.enabled=true
      - --http.addr=0.0.0.0
      - --http.port=8545
      - --http.vhosts=*
      - --http.corsdomain=*
      - --http.trace=false
      - --http.dbg.single=false
      - --http.api=eth,web3,net,debug,trace,txpool
      - --rpc.batch.limit=50
      - --ws=true
      - --ws.port=8546
      - --pprof
      - --pprof.addr=0.0.0.0
      - --pprof.port=6060
      - --metrics
      - --metrics.addr=0.0.0.0
      - --metrics.port=6061
      - --authrpc.addr=0.0.0.0
      - --authrpc.port=8551
      - --authrpc.jwtsecret=/home/erigon/.local/share/erigon/jwt.hex
      - --torrent.port=42069
      - --torrent.conns.perfile=10
      - --torrent.trackers.disable=true
      - --torrent.maxpeers=100
      - --torrent.download.rate=1024mb
      - --torrent.upload.rate=16mb
      - --downloader.verify=false
      - --nat=extip:x.x.x.x # replace x.x.x.x with your server's public IP
      - --port=30303
      - --maxpeers=100
      - --caplin.discovery.addr=0.0.0.0
      - --caplin.discovery.port=4000
      - --caplin.discovery.tcpport=4001
      - --caplin.adaptable-maximum-traffic-requirements=true
      - --caplin.blobs-no-pruning=true
      - --caplin.snapgen=false
      - --caplin.checkpoint-sync.disable=false
      - --caplin.blobs-immediate-backfill=true
      - --caplin.subscribe-all-topics
      - --beacon.api=beacon,builder,config,debug,events,node,validator,lighthouse
      - --beacon.api.addr=0.0.0.0
      - --beacon.api.port=5555
      - --beacon.api.cors.allow-methods=GET,POST,PUT,DELETE,OPTIONS
      - --beacon.api.cors.allow-origins=*
      - --beacon.api.cors.allow-credentials=false
      - --beacon.api.protocol=tcp
      - --db.pagesize=64kb
      - --verbosity=info
      - --log.console.verbosity=info
      - --log.dir.verbosity=debug
    environment:
      - ERIGON_SNAPSHOT_MADV_RND=false
    volumes:
      - /ethereum/erigon:/home/erigon/.local/share/erigon
    mem_limit: 40g
    mem_swappiness: 0
    logging:
      driver: json-file
      options:
        max-size: 100m
        max-file: 5
    stop_grace_period: 1800s
    networks:
      - common

networks:
  common:
    name: common
    driver: bridge
arbitrum-sepolia-compose.yaml
services:
  arbitrum-sepolia:
    image: offchainlabs/nitro-node:v3.7.6-c0fe95e
    container_name: arbitrum-sepolia
    restart: always
    user: 1000:1000
    ports:
      - 8547:8547 # HTTP
      - 8548:8548 # WS
      - 6070:6070 # Metrics
    command:
      - --node.staker.enable=false
      - --log-level=INFO
      - --parent-chain.connection.url=http://erigon-sepolia:8545
      - --parent-chain.blob-client.beacon-url=http://erigon-sepolia:5555
      - --chain.id=421614
      - --init.latest=pruned
      - --http.addr=0.0.0.0
      - --http.port=8547
      - --http.api=net,web3,eth,arb
      - --http.corsdomain=*
      - --http.vhosts=*
      - --ws.addr=0.0.0.0
      - --ws.port=8548
      - --ws.api=net,web3,eth,arb
      - --ws.origins=*
      - --metrics
      - --metrics-server.addr=0.0.0.0
      - --metrics-server.port=6070
    volumes:
      - /arbitrum/arbitrum:/home/user/.arbitrum
    logging:
      driver: json-file
      options:
        max-size: 100m
        max-file: 5
    stop_grace_period: 1800s
    networks:
      - common

networks:
  common:
    name: common
    driver: bridge
    external: true
  • Erigon was sync yesterday (after Fusaka fork) from scratch using latest release 3.2.0
  • Today we switched to Erigon Docker images which was built using the following steps - let arbitrum nitro work further (unless fixed in OCL/nitro) #17458 (comment).
  • Not sure if that is important but we have two options enabled and caplin.subscribe-all-topics was enabled based on today's recommendation in Arbitrum Discord
          - --caplin.blobs-immediate-backfill=true
          - --caplin.subscribe-all-topics

@awskii
Copy link
Member Author

awskii commented Oct 16, 2025

@air3ijai --caplin.subscribe-all-topics is enabled by default.

on this branch did not met any 404s from Nitro side on 3.8.0rc8+ and 3.7.6

@air3ijai
Copy link

Switched to a latest release v3.2.1 - so far so good.

Thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants