Skip to content

[HAMR-392] feat(monitors): per-type concurrency cap, overload backoff, error logging#603

Merged
riyazsh merged 4 commits into
mainfrom
riyaz/HAMR-392-monitor-throughput-and-error-visibility
Jul 6, 2026
Merged

[HAMR-392] feat(monitors): per-type concurrency cap, overload backoff, error logging#603
riyazsh merged 4 commits into
mainfrom
riyaz/HAMR-392-monitor-throughput-and-error-visibility

Conversation

@riyazsh

@riyazsh riyazsh commented Jul 5, 2026

Copy link
Copy Markdown
Contributor

Summary

Three related changes to make monitor sync more resilient when the destination monitor API is under load, and to make failures easier to diagnose from logs alone.

1. ResourceConfig.max_concurrent — general per-resource-type concurrency cap

New optional field on ResourceConfig. When set to a positive int N, installs an asyncio.Semaphore that limits _apply_resource_cb to at most N in-flight creates/updates of that resource type — independent of the global --max-workers. Left unset by default on every resource; callers opt in per-resource-type when they need tighter throttling than the global worker pool provides.

Motivating shape: at high --max-workers values, the monitor endpoint can be hit hard enough that the destination's edge/proxy layer starts terminating connections before the app finishes responding. A per-resource-type cap lets an operator throttle just monitors (or dashboards, or whichever endpoint they observe struggling) without slowing the whole sync.

2. HTTP _OVERLOAD_STATUSES retry backoff in custom_client.request_with_retry

For HTTP 502 / 503 / 504 / 512, retries now use a 30s / 60s / 120s schedule and honor Retry-After when the server sends one. Previously these codes went through the generic retry_count * 5s schedule (0 / 5 / 10 / 15s), which is too tight to let an overloaded upstream drain — retries refill the queue as fast as it clears.

429 keeps its existing dedicated x-ratelimit-reset path. The retry_timeout total budget still applies. CustomClient.overload_status_counter is exposed so callers can emit a metric on these retry classes if they want long-term dashboarding.

3. Structured error log on monitor create/update failures

monitors.py now emits a WARNING on any 4xx / 5xx create_resource / update_resource failure containing:

  • monitor id
  • monitor type (metric alert / rum alert / query alert / etc.)
  • outbound query string, truncated at 2000 chars with any @application.id: token preserved inline

Previously only the destination's response body was logged (e.g. 400 Bad Request - {"errors":["Invalid query: Check for invalid tags or facets in your query."]}), which doesn't identify WHICH monitor query the validator rejected. The new log line lets an operator jump straight from the failure to the offending query without cross-referencing source-org state.

No behavior change for callers that

  • don't set max_concurrent on any resource, AND
  • don't hit HTTP {502, 503, 504, 512}

Monitor error logging is additive (WARN-level, doesn't change return values or retry behavior).

Test plan

  • pytest tests/unit/ — 781 passing (757 pre-existing + 24 new).
  • New test files: test_max_concurrent_semaphore.py, test_overload_backoff.py, test_monitor_error_logging.py.
  • Integration cassette re-record if needed.

…ging

- ResourceConfig.max_concurrent: opt-in per-resource-type Semaphore for
  callers that need tighter throttling than --max-workers.
- request_with_retry: 30/60/120s backoff (honors Retry-After) for
  502/503/504/512, replacing the 0/5/10/15s generic-5xx schedule that
  refilled overloaded proxy queues. Adds CustomClient.overload_status_counter.
- monitors: log outbound id/type/query on 4xx/5xx create/update, truncated
  at 2000 chars with @application.id: preserved.

Refs: HAMR-392

Co-Authored-By: Claude Opus 4.7 <[email protected]>
@riyazsh
riyazsh requested a review from a team as a code owner July 5, 2026 02:01
riyazsh and others added 2 commits July 4, 2026 21:49
- resources_handler: track sem_acquired; only release if acquire returned.
- custom_client: fix overload guard so all backoff buckets are reachable.
- custom_client: honor fractional-second Retry-After.
- base_resource: init_async clears async_semaphore before re-install.
- monitors: cap @application.id token in summarizer; accept non-string query.
- tests: 788 passing (+7 new).

Co-Authored-By: Claude Opus 4.7 <[email protected]>
- monitors: fall back to resource.get("queries") when "query" is None,
  so formula / multi-query monitors log a meaningful payload on 4xx/5xx.
- custom_client: parse Retry-After HTTP-date form via
  email.utils.parsedate_to_datetime; past dates clamp to 0.
- tests: 791 passing (+3 new).

Co-Authored-By: Claude Opus 4.7 <[email protected]>
…ut test regression

Change 1: add a truncated, crash-safe server failure reason (reason=) to the
monitor HTTP error log line and correct the docstring to state the gate fires
on any status_code >= 400 (both 4xx and 5xx).

Change 2: pass a small --http-client-retry-timeout in the id-file subprocess
tests so the overload backoff gives up immediately instead of sleeping the
real 30/60/120s schedule past the 60s subprocess budget. Production backoff
is unchanged; the tests still exercise the 503 transient-failure path.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
@riyazsh
riyazsh merged commit 96959fd into main Jul 6, 2026
10 of 11 checks passed
@riyazsh
riyazsh deleted the riyaz/HAMR-392-monitor-throughput-and-error-visibility branch July 6, 2026 18:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants