Skip to content

Commit 779c8e1

Browse files
authored
Untyped ValueError/AssertionError bug (#47091)
* fixing ValueError bug * fixing copilot comments * fixing tests * fixing test * fixing warninngs * fixing broken test * adding logging for stuck tests * fixing logic bug * fixing testing gap * fix: reentrant pkrange cache lock + emulator-safe regional setup in retry tests Two emulator-pipeline failures: 1) routing_map_provider.py / aio: change _shared_cache_lock from Lock to RLock. `__init__` holds this lock while indexing _shared_routing_map_cache; indexing can synchronously trigger GC of a prior PartitionKeyRangeCache on the same thread (e.g. MagicMock attribute access via _mock_set_magics, or any production scenario where GC sweeps a prior instance during __init__). The swept instance's __del__ -> release() re-acquires the same lock -> deadlock under non-reentrant Lock. Adds regression tests in test_shared_pk_range_cache{,_async}.py that hang under Lock and pass under RLock. 2) test_service_retry_policies{,_async}.py: when host points at the emulator (localhost/127.0.0.1), _setup_read_regions / _setup_write_regions now route every 'region' to the real emulator URL instead of synthesizing localhost-westus:8081 etc. After the perf change keyed the shared pkrange cache by endpoint URL and the testing-gap commit started calling update_location_cache(), the SDK actually routed pkranges/ReadFeed to the synthetic per-region URLs, which fail DNS resolution. The real ServiceRequestError surfaced before the mock could run, leaving mf.counter at 0 (sync) or drifting (async). Mapping all regions to the actual host keeps both the routing-map cache valid and the request reachable. Also tightens test_per_partition_circuit_breaker_mm_async to skip when the recovery-phase precondition isn't met instead of asserting a flaky timing window. * fixing testing gap * addressing pylint errors * adding new RLock specific tests * fixing two dormant issues which got explosed by rlock and the test infra issue on slit pipelines.
1 parent 203fcc5 commit 779c8e1

26 files changed

Lines changed: 2396 additions & 390 deletions

‎sdk/cosmos/azure-cosmos/CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@
1313
* Added an opt-in fallback for invalid UTF-8 in response bodies. Default behavior is unchanged (strict decode). Setting `AZURE_COSMOS_CHARSET_DECODER_ERROR_ACTION_ON_MALFORMED_INPUT` to `REPLACE` or `IGNORE` enables a permissive decode so reads, queries, and change-feed iteration can make progress past corrupt payloads. See [PR 47008](https://github.com/Azure/azure-sdk-for-python/pull/47008)
1414
* Fixed bug where `CosmosClient` construction with AAD credentials would crash at startup if the semantic reranking inference endpoint environment variable was not set, even when semantic reranking was not being used. The inference service is now lazily initialized on first use. See [PR 46243](https://github.com/Azure/azure-sdk-for-python/pull/46243)
1515
* Fixed bug where region names in `preferred_locations` and `excluded_locations` (client-level and per-request) were not matched tolerantly for differences in case, whitespace, hyphens, and underscores. See [PR 46937](https://github.com/Azure/azure-sdk-for-python/pull/46937)
16+
* Fixed bug where a `ValueError("Ranges overlap")` or an `AssertionError("code bug: returned overlapping ranges ... is empty")` from the partition key range cache could escape to the caller when the `/pkranges` response contained a transiently inconsistent snapshot (overlap or gap). See [PR 47091](https://github.com/Azure/azure-sdk-for-python/pull/47091)
1617

1718
#### Other Changes
1819
* Reduced per-client memory overhead when partition-level circuit breaker (PPCB) is enabled by sharing the partition key range routing map cache across CosmosClient instances connected to the same endpoint, and stripping unused fields from cached partition key ranges using compact PKRange namedtuples. See [PR 46297](https://github.com/Azure/azure-sdk-for-python/pull/46297)

‎sdk/cosmos/azure-cosmos/azure/cosmos/_retry_utility.py‎

Lines changed: 17 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -334,6 +334,14 @@ def _has_read_retryable_headers(request_headers):
334334
return True
335335
return False
336336

337+
def _is_read_retryable_request(request, request_params: Optional[RequestObject] = None):
338+
if request and _has_read_retryable_headers(request.headers):
339+
return True
340+
if request_params and _OperationType.IsReadOnlyOperation(request_params.operation_type):
341+
# Fallback for flows where operation headers are absent but request metadata is available.
342+
return True
343+
return False
344+
337345
def _has_database_account_header(request_headers):
338346
if request_headers.get(HttpHeaders.ThinClientProxyResourceType) == ResourceType.DatabaseAccount:
339347
return True
@@ -354,13 +362,17 @@ def _handle_service_request_retries(
354362
raise exception
355363

356364
def _handle_service_response_retries(request, client, response_retry_policy, exception, *args):
357-
if request and (_has_read_retryable_headers(request.headers) or (args and (is_write_retryable(args[0], client) or
358-
client._global_endpoint_manager.is_per_partition_automatic_failover_applicable(args[0])))):
365+
request_params = args[0] if args else None
366+
if request and (_is_read_retryable_request(request, request_params) or (request_params is not None and (
367+
is_write_retryable(request_params, client) or
368+
client._global_endpoint_manager.is_per_partition_automatic_failover_applicable(request_params)))):
359369
# we resolve the request endpoint to the next preferred region
360370
# once we are out of preferred regions we stop retrying
361371
retry_policy = response_retry_policy
362372
if not retry_policy.ShouldRetry():
363-
if args and args[0].should_clear_session_token_on_session_read_failure and client.session:
373+
if (request_params is not None
374+
and request_params.should_clear_session_token_on_session_read_failure
375+
and client.session):
364376
client.session.clear_session_token(client.last_response_headers)
365377
raise exception
366378
else:
@@ -444,7 +456,7 @@ def send(self, request):
444456
except ServiceResponseError as err:
445457
retry_error = err
446458
# Only read operations can be safely retried with ServiceResponseError
447-
if (not _has_read_retryable_headers(request.http_request.headers) or
459+
if (not _is_read_retryable_request(request.http_request, request_params) or
448460
_has_database_account_header(request.http_request.headers) or
449461
request_params.healthy_tentative_location):
450462
raise err
@@ -466,7 +478,7 @@ def send(self, request):
466478
if (_has_database_account_header(request.http_request.headers) or
467479
request_params.healthy_tentative_location):
468480
raise err
469-
if _has_read_retryable_headers(request.http_request.headers) and retry_settings['read'] > 0:
481+
if _is_read_retryable_request(request.http_request, request_params) and retry_settings['read'] > 0:
470482
_record_failure_if_request_not_cancelled(request_params, global_endpoint_manager, None)
471483
retry_active = self.increment(retry_settings, response=request, error=err)
472484
if retry_active:

‎sdk/cosmos/azure-cosmos/azure/cosmos/_routing/_routing_map_provider_common.py‎

Lines changed: 166 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -28,10 +28,19 @@
2828
"""
2929

3030
import logging
31+
import random
3132
from typing import Any, Dict, List, Optional, Tuple
3233

3334
from .. import _base, http_constants
34-
from .collection_routing_map import CollectionRoutingMap, _build_routing_map_from_ranges
35+
from ..exceptions import CosmosHttpResponseError
36+
# Re-exported here so provider modules and tests import these from one place
37+
# rather than reaching into ``collection_routing_map`` directly.
38+
from .collection_routing_map import ( # pylint: disable=unused-import
39+
CollectionRoutingMap,
40+
_build_routing_map_from_ranges,
41+
_OverlapDetected,
42+
_GapDetected,
43+
)
3544
from . import routing_range
3645
from .routing_range import (
3746
PKRange,
@@ -44,6 +53,144 @@
4453

4554
PAGE_SIZE_CHANGE_FEED = "-1" # Return all available changes
4655

56+
# Retry budget for transient ``/pkranges`` snapshot inconsistencies (overlap
57+
# or gap) before the caller surfaces a 503. Shared by sync and async providers.
58+
#
59+
# Total attempts the fetch loop will make before raising 503. With the
60+
# schedule below, 4 attempts means up to 3 sleeps: worst-case cumulative
61+
# blocking time is 1.4s (0.2 + 0.4 + 0.8), expected ~0.775s when all three
62+
# retries occur (sum of per-attempt midpoints of the floored uniform).
63+
_TRANSIENT_SNAPSHOT_RETRY_MAX_ATTEMPTS = 4
64+
65+
# Initial deterministic upper bound (seconds) for the first retry sleep.
66+
# Doubled each attempt and clamped at ``_TRANSIENT_SNAPSHOT_RETRY_MAX_BACKOFF_SECONDS``.
67+
# At 0.2s the median sleep on attempt 1 lands in the same window in which
68+
# /pkranges gateway-snapshot inconsistencies typically converge (tens to a
69+
# few hundred ms), so attempt 2 is much more likely to see fresh state.
70+
_TRANSIENT_SNAPSHOT_RETRY_INITIAL_BACKOFF_SECONDS = 0.2
71+
72+
# Hard cap on the deterministic upper bound for any single retry sleep.
73+
# Forward-protection: if ``_TRANSIENT_SNAPSHOT_RETRY_MAX_ATTEMPTS`` ever
74+
# grows, exponential growth alone cannot block the calling thread for more
75+
# than this many seconds inside a single sleep. Independent of the per-call
76+
# budget -- this caps *one* sleep, not the cumulative.
77+
_TRANSIENT_SNAPSHOT_RETRY_MAX_BACKOFF_SECONDS = 2.0
78+
79+
# Floor (seconds) for the jittered sleep. Below this, gateway /pkranges
80+
# state has not had time to begin converging, so a retry would burn an
81+
# attempt with no benefit. Applied as ``min(MIN, upper / 4)`` so the floor
82+
# never dominates the jitter range on small upper bounds (i.e. attempt 1).
83+
_TRANSIENT_SNAPSHOT_RETRY_MIN_BACKOFF_SECONDS = 0.05
84+
85+
86+
def _deterministic_backoff_for_attempt(attempt: int) -> float:
87+
"""Return the deterministic exponential upper bound for ``attempt``.
88+
89+
The schedule is ``INITIAL * 2^(attempt - 1)``, clamped at
90+
``_TRANSIENT_SNAPSHOT_RETRY_MAX_BACKOFF_SECONDS``. ``attempt`` is
91+
1-indexed (i.e. ``attempt=1`` is the first retry after the first
92+
failure).
93+
94+
Extracted as a single source of truth so the test suite can derive
95+
expected bounds from the same formula the production code uses rather
96+
than re-encoding the constants. A regression that changes either the
97+
base or the doubling factor now fails one test, not many.
98+
99+
:param int attempt: 1-indexed retry attempt number.
100+
:return: The deterministic upper bound (seconds) for this attempt's sleep.
101+
:rtype: float
102+
"""
103+
raw = _TRANSIENT_SNAPSHOT_RETRY_INITIAL_BACKOFF_SECONDS * (2 ** (attempt - 1))
104+
return min(raw, _TRANSIENT_SNAPSHOT_RETRY_MAX_BACKOFF_SECONDS)
105+
106+
107+
def _jittered_backoff(deterministic_upper: float) -> float:
108+
"""Return a floored-full-jitter sleep in ``[floor, deterministic_upper]``.
109+
110+
``floor = min(_TRANSIENT_SNAPSHOT_RETRY_MIN_BACKOFF_SECONDS, deterministic_upper / 4)``
111+
112+
This is the hybrid jitter strategy chosen for ``/pkranges`` snapshot
113+
retries:
114+
115+
* The **non-zero floor** eliminates the near-zero-sleep tail of pure
116+
full jitter. The failure mode here is state propagation on the
117+
gateway, not contention -- a retry that fires within a few ms of
118+
the previous one will see the same stale snapshot and burn an
119+
attempt for nothing.
120+
* The **uniform distribution over** ``[floor, upper]`` preserves the
121+
bulk of full jitter's fleet-wide herd dispersion. Using an additive
122+
form (``uniform(floor, upper)``) rather than ``max(floor, uniform(0,
123+
upper))`` avoids creating a probability spike at exactly ``floor``,
124+
which would itself form a micro-herd at scale.
125+
* The ``upper / 4`` clamp on the floor guarantees the jitter range is
126+
always at least 75% of the deterministic upper, so the floor never
127+
collapses the smallest attempts into a near-constant wait.
128+
129+
:param float deterministic_upper: Non-negative upper bound for the
130+
sleep (typically produced by :func:`_deterministic_backoff_for_attempt`).
131+
:return: A random sleep value in ``[floor, deterministic_upper]``, or
132+
``0.0`` when ``deterministic_upper`` is non-positive.
133+
:rtype: float
134+
"""
135+
if deterministic_upper <= 0:
136+
return 0.0
137+
floor = min(
138+
_TRANSIENT_SNAPSHOT_RETRY_MIN_BACKOFF_SECONDS,
139+
deterministic_upper / 4,
140+
)
141+
return random.uniform(floor, deterministic_upper)
142+
143+
144+
def _handle_transient_snapshot_retry_decision(
145+
*,
146+
retry_attempt_count: int,
147+
collection_link: str,
148+
logger: logging.Logger, # pylint: disable=redefined-outer-name
149+
) -> float:
150+
"""Return the next backoff to sleep, or raise 503 once the budget is exhausted.
151+
152+
Called after the routing-map builder reports a transient overlap or gap.
153+
The caller performs the actual sleep (``time.sleep`` vs ``await
154+
asyncio.sleep``) -- the only line that differs between sync and async.
155+
156+
:keyword int retry_attempt_count: Attempts so far, including the failed
157+
one. Pass ``1`` after the first failure.
158+
:keyword str collection_link: Used in log messages and the 503 body.
159+
:keyword logging.Logger logger: Caller's module-level logger.
160+
:return: Floored-full-jitter backoff seconds in
161+
``[floor, deterministic_upper_bound]``.
162+
:rtype: float
163+
:raises CosmosHttpResponseError: When the retry budget is exhausted.
164+
"""
165+
if retry_attempt_count >= _TRANSIENT_SNAPSHOT_RETRY_MAX_ATTEMPTS:
166+
logger.error(
167+
"Routing-map fetch for collection '%s' returned overlapping or "
168+
"gapped ranges on %d attempt(s). Surfacing as HTTP 503.",
169+
collection_link,
170+
retry_attempt_count,
171+
)
172+
raise CosmosHttpResponseError(
173+
status_code=http_constants.StatusCodes.SERVICE_UNAVAILABLE,
174+
message=(
175+
"Routing-map fetch for collection '{}' returned overlapping "
176+
"or gapped ranges on {} attempt(s)."
177+
).format(collection_link, retry_attempt_count),
178+
)
179+
180+
deterministic_backoff = _deterministic_backoff_for_attempt(retry_attempt_count)
181+
jittered_backoff = _jittered_backoff(deterministic_backoff)
182+
logger.warning(
183+
"Routing-map fetch for collection '%s' returned overlapping or "
184+
"gapped ranges (attempt %d/%d). Sleeping %.2fs and retrying.",
185+
collection_link,
186+
retry_attempt_count,
187+
_TRANSIENT_SNAPSHOT_RETRY_MAX_ATTEMPTS,
188+
jittered_backoff,
189+
)
190+
return jittered_backoff
191+
192+
193+
47194

48195
def is_cache_unchanged_since_previous(
49196
collection_routing_map_by_item: Dict[str, CollectionRoutingMap],
@@ -149,7 +296,7 @@ def _resolve_endpoint(client: Any) -> str:
149296

150297

151298
class _IncrementalMergeFailed(Exception):
152-
"""Sentinel raised by :func:`process_fetched_ranges` when the
299+
"""Private exception type raised by :func:`process_fetched_ranges` when the
153300
incremental update cannot resolve all partition key ranges.
154301
155302
The caller decides how to recover: retry the incremental fetch
@@ -162,7 +309,7 @@ def process_fetched_ranges(
162309
collection_id: str,
163310
collection_link: str,
164311
new_etag: Optional[str],
165-
) -> Optional[CollectionRoutingMap]:
312+
) -> CollectionRoutingMap:
166313
"""Turn raw PK-range results into a :class:`CollectionRoutingMap`.
167314
168315
Handles both initial-load (when *previous_routing_map* is ``None``)
@@ -177,10 +324,8 @@ def process_fetched_ranges(
177324
:param str collection_id: The ID of the collection.
178325
:param str collection_link: The link to the collection.
179326
:param str new_etag: The ETag from the change feed response, or ``None``.
180-
:return: The new/updated routing map, or ``None`` when an
181-
initial load yields no ranges.
327+
:return: The new/updated routing map.
182328
:rtype: ~azure.cosmos._routing.collection_routing_map.CollectionRoutingMap
183-
or None
184329
:raises _IncrementalMergeFailed: When the incremental path cannot
185330
resolve all ranges. The caller catches this and either retries
186331
the incremental fetch or falls back to a full refresh.
@@ -257,7 +402,21 @@ def process_fetched_ranges(
257402

258403
unresolved = next_unresolved
259404

260-
result = previous_routing_map.try_combine(range_tuples, effective_etag)
405+
try:
406+
result = previous_routing_map.try_combine(range_tuples, effective_etag)
407+
except ValueError as overlap_error:
408+
# Convert the overlap ``ValueError`` to ``_IncrementalMergeFailed`` so
409+
# the caller retries and falls back to a full refresh. Narrow the
410+
# match to the ``"Ranges overlap"`` prefix so any unrelated
411+
# ``ValueError`` still surfaces as a real bug.
412+
if not str(overlap_error).startswith("Ranges overlap"):
413+
raise
414+
logger.warning(
415+
"Incremental merge for collection '%s' produced overlapping ranges: %s. "
416+
"Falling back to a full refresh.",
417+
collection_link, str(overlap_error),
418+
)
419+
raise _IncrementalMergeFailed() from overlap_error
261420
if not result:
262421
logger.warning(
263422
"Incremental merge resulted in incomplete routing map for "

0 commit comments

Comments
 (0)