fix(server): cool down recently failed similarity verification jobs - #3002

Summary

After a similarity verification job finishes with no_similar_match_found, an immediate request for the same chain and address currently starts another job and repeats the bytecode fetch and similarity search. This PR adds a configurable cooldown before that work starts and returns 429, similarity_recently_failed and the remaining wait in Retry-After.

Fixes #2939.

Why

Trigger and root cause

The contract page's automatic similarity request can be repeated by refreshing an unverified contract. The existing checkIfJobIsAlreadyRunning middleware only rejects incomplete jobs. Once a failed job has a completed_at value, another request passes that guard, and verifyFromSimilarityViaWorker immediately fetches the runtime bytecode and inserts another job. The background path then searches the same candidate set and, when candidates exist, recompiles them again.

The failure classification, completion timestamp, chain, address and verification endpoint are already persisted in verification_jobs; they were not consulted before another similarity attempt. This implements the existing request in #2939. I reproduced the behaviour on staging at 45acd3358358cc815d1cb6e9868c3b19704fbe99 (1 October 2026), rather than assuming it still existed from the issue description. That establishes the affected current revision; it does not identify the original introducing commit.

Reproduction and observed behaviour

Both regressions use the actual Express endpoint, the committed database schema and a locally deployed Hardhat contract:

1. Start with an unverified deployment whose runtime bytecode is longer than the 75-byte similarity prefix.
2. Submit POST /v2/verify/similarity/:chainId/:address and wait for the job to complete with no_similar_match_found.
3. Submit the same request again immediately.

One regression uses an empty candidate database. The other stores the existing Storage compilation and deploys a contract with the same indexed prefix but a changed SLOAD slot after that prefix. The real worker recompiles the candidate, finds neither a creation nor a runtime match, and persists the no-match outcome. The worker hook only controls when the real worker runs; its result is not stubbed.

The empty-candidate regression failed consistently in separate runs against the pristine base. Both regression cases also failed together on that base with expected 202 to equal 429; both pass with the patch.

Impact

Repeated completed failures can consume RPC, database and compilation work even while the deployment and available candidates remain unchanged. The reproduced defect is redundant verification work. I have not measured production traffic, resource exhaustion or a security severity, and this PR makes no such claim.

Changes

• Query the most recent completed similarity job classified as no_similar_match_found for the same chain and binary contract address, within the configured interval. The existing chain/address index supports this lookup; no schema migration is required.
• Check this history before getBytecode, job insertion and background work. Persisted history is shared by replicas using the same database and survives a server restart.
• Measure the interval from completion, round the remaining milliseconds up to whole seconds, and allow the request if the interval expires while the history query is running. Rejected requests do not insert a job or extend the cooldown.
• Add similarityFailureCooldownSeconds to the service options and server config: 600 by default, 0 to disable. Invalid negative or non-finite values are rejected at construction.
• Add the v2 error and Retry-After header, expose that header to cross-origin browser clients, and document the response and configuration in the existing OpenAPI and server README formats.
• Leave other persisted errors, other verification endpoints, other chains and other addresses outside this cooldown. Existing in-progress and already-verified guards continue to run first.

Notes

Validation evidence

The three focused suites passed with 38 tests, including 13 added cases: the two HTTP regressions; completion-time and remaining-header behaviour with a lowercase address; exact expiry; three transient-error controls; endpoint, chain and address isolation; custom duration; disabled cooldown; and expiry during the history lookup. Existing successful verification, duplicate-request, already-verified, storage and worker-error tests in those suites also passed.

The suite selection was:

cd services/server
npx mocha --exit \
  test/integration/apiv2/verification/verify.similarity.spec.ts \
  test/integration/storageServices/SourcifyDatabaseService.spec.ts \
  test/unit/VerificationService.spec.ts

The regression-only selection was run against a separate pristine checkout of the base revision with the new tests copied in. It produced two expected assertion failures, not setup or database errors:

npx mocha --exit test/integration/apiv2/verification/verify.similarity.spec.ts \
  --grep 'immediate retry after a completed|real candidate recompilation'

Dependency package builds (compilers-types, bytecode-utils, compilers, lib-sourcify) and the server build passed. tsc --noEmit --project services/server/tsconfig.lint.json, Prettier on every changed file, and git diff --check passed. ESLint over the server source and changed tests passed with zero errors and 15 existing unused-variable warnings; unrelated warnings were left alone.

The local test environment used Node 24.19.0, Hardhat 3.16.0 and PostgreSQL 18.3 compiled to WASM through PGlite 0.5.8 and its TCP socket adapter. The wrapper loaded the committed schema (including pg_trgm and pgcrypto) and ran the repository's unchanged fixtures and SQL through pg. Native PostgreSQL could not start in this sandbox, so the suite commands above were launched through that local wrapper. PGlite, the wrapper and environment setup are outside the proposed diff. I have not run the full monorepo suite or native PostgreSQL/CI here.

Bounded retry policy

This is a temporary chain/address cooldown, not a guarantee that a later attempt cannot succeed. A new verified candidate, a changed deployment or a different creation-transaction hint can become useful during the interval; callers must wait for expiry, and operators can shorten or disable the interval. Eligibility follows the existing persisted error code; worker error classification is not changed here. There is no forced-retry bypass or permanent failure cache.

The new lookup adds one indexed history query to enabled similarity attempts. Production query plans and latency have not been benchmarked. Concurrent admission races before any completed failure exists remain outside this change.