Skip to content

fix(operator): manage subscription to events without panic #1729

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
MauroToscano merged 16 commits into from
Jun 4, 2025
Merged

fix(operator): manage subscription to events without panic #1729

MauroToscano merged 16 commits into from
Jun 4, 2025

Conversation

@JuArce
Copy link
Collaborator

@JuArce JuArce commented Jan 9, 2025 •
edited by MarcosNicolau
Loading

Manage Operator's event subscription

Description

Previously, the Operator panicked if one connection (either main or fallback) failed.

The previous solution to this (#1692) was to iterate infinitely over both connections, this had the "silent failure" problem, as an operator could fail and retry forever without notice to its owner.

Now if one connection fails, the Operator will use the other one, while continiously trying to reconnect to the failed one; and if both connections fail, and the retryables are consumed, the Operator will exit, logging the errors of each connection RPC, so the owner/manager of the Operator server can fix it accordingly.

This PR also added the usage of fallback when operator calls DisableVerifiers

To test

  1. Start Ethereum Package:
make ethereum_package_start
  1. Start verification layer components:
make aggregator_start_ethereum_package ENVIRONMENT=devnet
make batcher_start_ethereum_package
make operator_register_and_start_ethereum_package ENVIRONMENT=devnet
  1. Stop and Start the containers ethereum package started for reth-2 and reth-3. reth-3 is set as the fallback of the operator and reth-2 as the main node:
docker stop <NODE_ID>
docker start <NODE_ID>
  • Verify the Operator behaves as described, and while submitting proofs take down 1 connection, take down the other connection, take down both connections.

Type of change

Please delete options that are not relevant.

  • New feature
  • Bug fix
  • Optimization
  • Refactor

Checklist

  • “Hotfix” to testnet, everything else to staging
  • Linked to Github Issue
  • This change depends on code or research by an external entity
    • Acknowledgements were updated to give credit
  • Unit tests added
  • This change requires new documentation.
    • Documentation has been added/updated.
  • This change is an Optimization
    • Benchmarks added/run
  • Has a known issue
  • If your PR changes the Operator compatibility (Ex: Upgrade prover versions)
    • This PR adds compatibility for operator for both versions and do not change batcher/docs/examples
    • This PR updates batcher and docs/examples to the newer version. This requires the operator are already updated to be compatible

@JuArce JuArce linked an issue Jan 9, 2025 that may be closed by this pull request
hotfix(operator): manage subscription to events without panic #1728
Closed
@PatStiles PatStiles mentioned this pull request Jan 9, 2025
Closed
14 tasks
@uri-99 uri-99 marked this pull request as ready for review January 13, 2025 14:36
uri-99
uri-99 approved these changes Jan 16, 2025
View reviewed changes
Copy link
Contributor

@uri-99 uri-99 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

biased self-approve

@Oppen Oppen added audit cantina Audit report from Cantina labels Jan 20, 2025
@JuArce JuArce removed audit cantina Audit report from Cantina labels Jan 20, 2025
JuArce
JuArce commented Jan 21, 2025
View reviewed changes
Copy link
Collaborator Author

@JuArce JuArce left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The same comments apply for V3

MarcosNicolau
MarcosNicolau reviewed May 20, 2025
View reviewed changes
@MarcosNicolau MarcosNicolau changed the base branch from testnet to staging May 20, 2025 17:42
@github-actions
Copy link

github-actions bot commented May 20, 2025

Changes to gas cost

Generated at commit: a33d8a04aa35a8606bcd45a8a35bc7dea911b4c8, compared to commit: b77e8fdf474834f0b9d018c28322b5c29512002e

🧾 Summary (10% most significant diffs)

Contract Method Avg (+/-) %
RegistryCoordinatorHarness blsApkRegistry
stakeRegistry		 +354 ❌
+354 ❌		 +57.47%
+51.91%
StakeRegistryHarness delegation +146 ❌ +22.19%
Full diff report 👇
Contract Deployment Cost (+/-) Method Min (+/-) % Avg (+/-) % Median (+/-) % Max (+/-) % # Calls (+/-)
Slasher 728,578 (-870,247) initialize 366 (-556) -60.30% 366 (-556) -60.30% 366 (-556) -60.30% 366 (-556) -60.30% 6 (0)
RegistryCoordinatorHarness 5,528,683 (-4,601,377) blsApkRegistry
initialize
stakeRegistry		 970 (+354)
55,078,420 (-1,203,055)
1,036 (+354)		 +57.47%
-2.14%
+51.91%		 970 (+354)
55,078,420 (-1,203,055)
1,036 (+354)		 +57.47%
-2.14%
+51.91%		 970 (+354)
55,078,420 (-1,203,055)
1,036 (+354)		 +57.47%
-2.14%
+51.91%		 970 (+354)
55,078,420 (-1,203,055)
1,036 (+354)		 +57.47%
-2.14%
+51.91%		 6 (0)
6 (0)
6 (0)
StakeRegistryHarness 2,590,172 (-2,549,338) delegation
initializeQuorum		 804 (+146)
144,420 (-1,780)		 +22.19%
-1.22%		 804 (+146)
164,216 (-1,780)		 +22.19%
-1.07%		 804 (+146)
164,320 (-1,780)		 +22.19%
-1.07%		 804 (+146)
164,320 (-1,780)		 +22.19%
-1.07%		 6 (0)
1,152 (0)
AlignedLayerServiceManager 4,398,072 (-3,921,136) batchesState
createNewTask
disableVerifier
disabledVerifiers
enableVerifier
isVerifierDisabled
receive
setDisabledVerifiers		 5,132 (-278)
56,970 (-824)
24,602 (+303)
3,030 (+516)
23,748 (-443)
2,406 (-450)
23,301 (-195)
24,243 (-7)		 -5.14%
-1.43%
+1.25%
+20.53%
-1.83%
-15.76%
-0.83%
-0.03%		 5,132 (-278)
76,868 (-920)
36,191 (+375)
3,030 (+516)
24,388 (-372)
2,406 (-450)
46,835 (-356)
35,262 (+86)		 -5.14%
-1.18%
+1.05%
+20.53%
-1.50%
-15.76%
-0.75%
+0.24%		 5,132 (-278)
77,026 (-920)
36,191 (+375)
3,030 (+516)
24,388 (-372)
2,406 (-450)
47,115 (-357)
35,262 (+86)		 -5.14%
-1.18%
+1.05%
+20.53%
-1.50%
-15.76%
-0.75%
+0.24%		 5,132 (-278)
77,812 (-956)
47,780 (+446)
3,030 (+516)
25,029 (-300)
2,406 (-450)
47,115 (-357)
46,282 (+179)		 -5.14%
-1.21%
+0.94%
+20.53%
-1.18%
-15.76%
-0.75%
+0.39%		 256 (0)
256 (0)
2 (0)
1 (0)
2 (0)
3 (0)
256 (0)
2 (0)
AVSDirectory 1,485,692 (-1,689,090) initialize 97,570 (-3,067) -3.05% 97,570 (-3,067) -3.05% 97,570 (-3,067) -3.05% 97,570 (-3,067) -3.05% 6 (0)
ServiceManagerMock 1,340,500 (-1,292,283) initialize 71,296 (-2,066) -2.82% 71,296 (-2,066) -2.82% 71,296 (-2,066) -2.82% 71,296 (-2,066) -2.82% 6 (0)
ProxyAdmin 412,495 (-345,151) upgrade
upgradeAndCall		 38,758 (-883)
55,403,611 (-1,228,407)		 -2.23%
-2.17%		 38,767 (-883)
55,403,611 (-1,228,407)		 -2.23%
-2.17%		 38,770 (-883)
55,403,611 (-1,228,407)		 -2.23%
-2.17%		 38,770 (-883)
55,403,611 (-1,228,407)		 -2.23%
-2.17%		 24 (0)
6 (0)
BLSApkRegistryHarness 1,676,721 (-1,447,516) initializeQuorum
setBLSPublicKey		 45,382 (-596)
89,551 (-556)		 -1.30%
-0.62%		 45,382 (-596)
89,551 (-556)		 -1.30%
-0.62%		 45,382 (-596)
89,551 (-556)		 -1.30%
-0.62%		 45,382 (-596)
89,551 (-556)		 -1.30%
-0.62%		 1,152 (0)
6 (0)
TransparentUpgradeableProxy 521,501 (-376,498) fallback 1,634 (+90) +5.83% 47,449 (-617) -1.28% 8,200 (+90) +1.11% 119,180 (-820) -0.68% 30 (0)
StrategyManagerMock 1,146,070 (-1,255,302) setAddresses 88,974 (-768) -0.86% 88,974 (-768) -0.86% 88,974 (-768) -0.86% 88,974 (-768) -0.86% 6 (0)
IndexRegistry 917,389 (-790,168) initializeQuorum 45,244 (-348) -0.76% 45,244 (-348) -0.76% 45,244 (-348) -0.76% 45,244 (-348) -0.76% 1,152 (0)

@JuArce JuArce changed the title hotfix(operator): manage subscription to events without panic fix(operator): manage subscription to events without panic May 20, 2025
MarcosNicolau
MarcosNicolau reviewed May 28, 2025
View reviewed changes
MarcosNicolau
MarcosNicolau approved these changes May 28, 2025
View reviewed changes
Copy link
Member

@MarcosNicolau MarcosNicolau left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Worked for me. I've updated the how-to-test section in the pr description to use ethereum-package. Not related to this pr I noticed that when deactivating the fallback of the Batcher, the Batcher stops listening to blocks, thus not creating new batches.

@MauroToscano MauroToscano added this pull request to the merge queue Jun 4, 2025
Merged via the queue into staging with commit ef6a851 Jun 4, 2025
3 checks passed
@MauroToscano MauroToscano deleted the 1728-hotfixoperator-manage-subscription-to-events-without-panic branch June 4, 2025 14:21
@JuArce JuArce mentioned this pull request Sep 9, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@MauroToscano MauroToscano MauroToscano approved these changes

@MarcosNicolau MarcosNicolau MarcosNicolau approved these changes

+1 more reviewer

@uri-99 uri-99 uri-99 approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

@Oppen Oppen

@uri-99 uri-99

@JuArce JuArce

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

hotfix(operator): manage subscription to events without panic

6 participants

@JuArce @MauroToscano @uri-99 @MarcosNicolau @Oppen