[HAMR-392] feat(monitors): per-type concurrency cap, overload backoff, error logging#603
Merged
riyazsh merged 4 commits intoJul 6, 2026
Conversation
…ging - ResourceConfig.max_concurrent: opt-in per-resource-type Semaphore for callers that need tighter throttling than --max-workers. - request_with_retry: 30/60/120s backoff (honors Retry-After) for 502/503/504/512, replacing the 0/5/10/15s generic-5xx schedule that refilled overloaded proxy queues. Adds CustomClient.overload_status_counter. - monitors: log outbound id/type/query on 4xx/5xx create/update, truncated at 2000 chars with @application.id: preserved. Refs: HAMR-392 Co-Authored-By: Claude Opus 4.7 <[email protected]>
- resources_handler: track sem_acquired; only release if acquire returned. - custom_client: fix overload guard so all backoff buckets are reachable. - custom_client: honor fractional-second Retry-After. - base_resource: init_async clears async_semaphore before re-install. - monitors: cap @application.id token in summarizer; accept non-string query. - tests: 788 passing (+7 new). Co-Authored-By: Claude Opus 4.7 <[email protected]>
- monitors: fall back to resource.get("queries") when "query" is None,
so formula / multi-query monitors log a meaningful payload on 4xx/5xx.
- custom_client: parse Retry-After HTTP-date form via
email.utils.parsedate_to_datetime; past dates clamp to 0.
- tests: 791 passing (+3 new).
Co-Authored-By: Claude Opus 4.7 <[email protected]>
…ut test regression Change 1: add a truncated, crash-safe server failure reason (reason=) to the monitor HTTP error log line and correct the docstring to state the gate fires on any status_code >= 400 (both 4xx and 5xx). Change 2: pass a small --http-client-retry-timeout in the id-file subprocess tests so the overload backoff gives up immediately instead of sleeping the real 30/60/120s schedule past the 60s subprocess budget. Production backoff is unchanged; the tests still exercise the 503 transient-failure path. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
nathantournant
approved these changes
Jul 6, 2026
riyazsh
deleted the
riyaz/HAMR-392-monitor-throughput-and-error-visibility
branch
July 6, 2026 18:24
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Three related changes to make monitor sync more resilient when the destination monitor API is under load, and to make failures easier to diagnose from logs alone.
1.
ResourceConfig.max_concurrent— general per-resource-type concurrency capNew optional field on
ResourceConfig. When set to a positive int N, installs anasyncio.Semaphorethat limits_apply_resource_cbto at most N in-flight creates/updates of that resource type — independent of the global--max-workers. Left unset by default on every resource; callers opt in per-resource-type when they need tighter throttling than the global worker pool provides.Motivating shape: at high
--max-workersvalues, the monitor endpoint can be hit hard enough that the destination's edge/proxy layer starts terminating connections before the app finishes responding. A per-resource-type cap lets an operator throttle just monitors (or dashboards, or whichever endpoint they observe struggling) without slowing the whole sync.2. HTTP
_OVERLOAD_STATUSESretry backoff incustom_client.request_with_retryFor HTTP 502 / 503 / 504 / 512, retries now use a 30s / 60s / 120s schedule and honor
Retry-Afterwhen the server sends one. Previously these codes went through the genericretry_count * 5sschedule (0 / 5 / 10 / 15s), which is too tight to let an overloaded upstream drain — retries refill the queue as fast as it clears.429keeps its existing dedicatedx-ratelimit-resetpath. Theretry_timeouttotal budget still applies.CustomClient.overload_status_counteris exposed so callers can emit a metric on these retry classes if they want long-term dashboarding.3. Structured error log on monitor create/update failures
monitors.pynow emits aWARNINGon any 4xx / 5xxcreate_resource/update_resourcefailure containing:idtype(metric alert / rum alert / query alert / etc.)querystring, truncated at 2000 chars with any@application.id:token preserved inlinePreviously only the destination's response body was logged (e.g.
400 Bad Request - {"errors":["Invalid query: Check for invalid tags or facets in your query."]}), which doesn't identify WHICH monitor query the validator rejected. The new log line lets an operator jump straight from the failure to the offending query without cross-referencing source-org state.No behavior change for callers that
max_concurrenton any resource, AND{502, 503, 504, 512}Monitor error logging is additive (WARN-level, doesn't change return values or retry behavior).
Test plan
pytest tests/unit/— 781 passing (757 pre-existing + 24 new).test_max_concurrent_semaphore.py,test_overload_backoff.py,test_monitor_error_logging.py.