Fix: Operator Panic when reconnect fails after max Retries

Description

Closes #960.
SubscribeToNewTasksV2Retryable and SubscribeToNewTasksV3Retryable are called when the operator starts . SubscribeToNewTasksV2Retryable and SubscribeToNewTasksV3Retryable both spawn a go-routine to handle errors from the subscription to two rpc's a primary and a fallback. Upon erroring the go-routine attempts to subscribe again up to a retry limit.

When SubscribeToNewTasksV2Retryable and SubscribeToNewTasksV3Retryable exhaust the maximum number of retries the respective subscribe functions panics due to dereferencing a nil reference within the auto-generated contract bindings.

In the case that one of the subscriptions errors and then fails to re-connect the operator will panic while one subscription (the fallback subscription) is still available and the operator can continue to function as normal.

In the case that both connections fail error and fail to reconnect then the operator should shut down.

If the operator fails to subscribe initially the operator should shut down.

This pr addressed this issue by:
1.) Adding a defer() to the go routine the converts the panic into an error before the operator process exits.
2.) Changing the retry parameters for SubscribeToNewTasksV2Retryable and SubscribeToNewTasksV3Retryable to retry infinitely (up to retry intervals of 60 sec).

To Test:

• install and use nginx as a proxy to test the connection brew install nginx
• create a directory nginx with the file nginx.conf inside of it, with this content:

events { }

http {
    server {
        listen 8082;

        location /main/ {
            proxy_pass http://host.docker.internal:8545/;  # Note the trailing slashes
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "upgrade";
            proxy_connect_timeout 1d;
            proxy_send_timeout 1d;
            proxy_read_timeout 1d;
        }
    }
}

• create a docker-compose.yml with the following contents:

version: '3'
services:
  nginx:
    image: nginx:alpine
    container_name: nginx-anvil-proxy
    volumes:
      - ./nginx:/etc/nginx
    ports:
      - "8082:8082"

• Make sure the nginx.conf file is located within ./nignx folder and you docker-compose-file has the proper path to the ngix.conf file.
• change the respective Operator RPC url's in config-files within the aligned-layer repo to use :8082/main/ port and path.
• Start a local anvil node make anvil_start_with_block_time
• Start nginx in a docker container via docker compose -f nginx-docker-compose.yaml up -d
• Observe proof verification is successful.
• kill the connection to anvil via docker compose -f nginx-docker-compose.yaml stop
• observe operator blocks with the following message:

• Start the nginx proxy again docker compose -f nginx-docker-compose.yaml up -d && restart the aggregator make aggregator_start
• Observe the operator reconnects and start verifying batches

Type of change

Please delete options that are not relevant.

• Bug fix

Checklist

• “Hotfix” to testnet, everything else to staging
• Linked to Github Issue
• This change depends on code or research by an external entity
  • Acknowledgements were updated to give credit
• Unit tests added
• This change requires new documentation.
  • Documentation has been added/updated.
• This change is an Optimization
  • Benchmarks added/run
• Has a known issue
  • Link to the open issue addressing it
• If your PR changes the Operator compatibility (Ex: Upgrade prover versions)
  • This PR adds compatibility for operator for both versions and do not change batcher/docs/examples
  • This PR updates batcher and docs/examples to the newer version. This requires the operator are already updated to be compatible