Skip to content

Fix: cold-start stale cache: hasFlags() now waits for persistence load#3282

Merged
gh-worker-dd-mergequeue-cf854d[bot] merged 7 commits into
developfrom
typo/fix-stale-cache-cold-start
Mar 25, 2026
Merged

Fix: cold-start stale cache: hasFlags() now waits for persistence load#3282
gh-worker-dd-mergequeue-cf854d[bot] merged 7 commits into
developfrom
typo/fix-stale-cache-cold-start

Conversation

@typotter

Copy link
Copy Markdown
Contributor

What does this PR do?

Fixes a bug where the flags module reported Error instead of Stale on cold start when a network failure occurred and valid cached flags existed in persistent storage.

Motivation

DefaultFlagsRepository.hasFlags() was reading atomicState directly without calling waitForPersistenceLoad(), unlike every other read method (getPrecomputedFlag, getFlagsSnapshot, getEvaluationContext, getPrecomputedFlagWithContext). On cold start, the async persistence callback had not yet fired when EvaluationsManager called hasFlags() to determine whether to fall back to cached Stale flags. The latch was not yet counted down, so atomicState was null, hasFlags() returned false, and the module transitioned to Error.

Changes

  • DefaultFlagsRepository.kt: hasFlags() now calls waitForPersistenceLoad() before reading atomicState, consistent with all other read methods. The wait is bounded by persistenceLoadTimeoutMs (default 100ms), only on the first call, only on the background flags-network executor thread — no deadlock risk, no main-thread impact.

  • DefaultFlagsRepositoryTest.kt: Removed the test that asserted hasFlags() does not block (it encoded the bug as a contract). Added three new tests covering: async callback with flags, async callback with no data, and timeout with no callback.

  • EvaluationsManagerTest.kt: Added an integration test using a real DefaultFlagsRepository that verifies the cold-start scenario ends in Stale (not Error) when cached flags exist, the context matches, and the network fails.

Additional Notes

Thread safety: the persistence latch is counted down by the datastore I/O thread (or by setFlagsAndContext() on a successful fetch), which is completely separate from the flags-network executor thread that calls hasFlags(). No deadlock is possible.

…tence load

On cold start, if the first network refresh fails, the flags module would
transition to Error instead of Stale even when valid cached flags existed.

Root cause: DefaultFlagsRepository.hasFlags() read atomicState directly
without calling waitForPersistenceLoad(), unlike every other read method.
The persistence latch had not been counted down yet at the moment hasFlags()
was called inside EvaluationsManager, so it returned false and the module
reported Error instead of falling back to cached Stale flags.

Fix: add waitForPersistenceLoad() call in hasFlags(), consistent with all
other read methods. This is bounded to at most persistenceLoadTimeoutMs
(default 100ms) on first call only, on a background executor thread.
@typotter typotter changed the title RUM-XXXX: Fix cold-start stale cache: hasFlags() now waits for persistence load Fix: cold-start stale cache: hasFlags() now waits for persistence load Mar 23, 2026
@typotter
typotter requested a review from Copilot March 23, 2026 18:56

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes a cold-start race where DefaultFlagsRepository.hasFlags() could read atomicState before persistence finished loading, causing EvaluationsManager to incorrectly transition to Error instead of Stale when cached flags existed and the network failed.

Changes:

  • Updated DefaultFlagsRepository.hasFlags() to call waitForPersistenceLoad() before checking cached flags.
  • Reworked DefaultFlagsRepositoryTest coverage for hasFlags() to validate behavior with async persistence load, empty persisted data, and timeout.
  • Added an EvaluationsManagerTest integration-style test using a real DefaultFlagsRepository to verify the cold-start stale-cache scenario.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 6 comments.

File Description
features/dd-sdk-android-flags/src/main/kotlin/com/datadog/android/flags/internal/repository/DefaultFlagsRepository.kt Makes hasFlags() consistent with other read APIs by awaiting persistence load before reading state.
features/dd-sdk-android-flags/src/test/kotlin/com/datadog/android/flags/internal/repository/DefaultFlagsRepositoryTest.kt Adds async/timeout tests for the new hasFlags() behavior; removes the previous “does not block” contract.
features/dd-sdk-android-flags/src/test/kotlin/com/datadog/android/flags/internal/evaluation/EvaluationsManagerTest.kt Adds an integration test validating Stale (not Error) on cold start with cached flags + network failure.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@datadog-datadog-prod-us1-2

This comment has been minimized.

- Switch all remaining version = any() stubs to anyOrNull() so the
  persistence callback is actually exercised rather than relying on
  timeout fallback
- Reduce persistenceLoadTimeoutMs from 5000ms to 500ms in async tests
  so regressions fail fast
- Remove flaky lower-bound elapsed-time assertion (currentTimeMillis
  resolution is too coarse on some JVMs)
- Assert capturedCallback non-null before invoking in integration test
  to surface misconfigured mocks immediately
- Assert awaitTermination returns true and call shutdownNow() in finally
  to avoid leaking threads in failing test runs

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Both tests previously fired the persistence callback before hasFlags()
was called, meaning they could pass even without the fix. Now:

- Repository test: runs hasFlags() on a background thread and polls
  Thread.State.TIMED_WAITING before firing the callback, ensuring the
  test fails if hasFlags() does not block on the persistence latch.

- Integration test: captures the executor thread via a custom
  ThreadFactory and applies the same TIMED_WAITING poll before firing
  the callback, guaranteeing the executor is blocked in hasFlags() when
  the persistence callback fires.
@codecov-commenter

codecov-commenter commented Mar 23, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 50.00000% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 71.60%. Comparing base (b3abae6) to head (e37cb73).
⚠️ Report is 62 commits behind head on develop.

Files with missing lines Patch % Lines
...lags/internal/repository/DefaultFlagsRepository.kt 50.00% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #3282      +/-   ##
===========================================
- Coverage    71.63%   71.60%   -0.03%     
===========================================
  Files          943      943              
  Lines        34815    34816       +1     
  Branches      5899     5899              
===========================================
- Hits         24937    24928       -9     
- Misses        8244     8248       +4     
- Partials      1634     1640       +6     
Files with missing lines Coverage Δ
...lags/internal/repository/DefaultFlagsRepository.kt 68.85% <50.00%> (+8.85%) ⬆️

... and 29 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Polling loops that wait for Thread.State.TIMED_WAITING would spin
forever if hasFlags() returned without blocking (regression), because
the thread would reach TERMINATED rather than TIMED_WAITING. Adding
TERMINATED as a break condition lets the loop exit immediately in the
regression case, after which the assertion fails fast with a clear
error rather than hanging the test suite.
@typotter
typotter marked this pull request as ready for review March 23, 2026 20:36
@typotter
typotter requested review from a team as code owners March 23, 2026 20:36
@typotter
typotter requested a review from Copilot March 23, 2026 21:02

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@kikoveiga

Copy link
Copy Markdown
Contributor

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 483fdf4ed4

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

- Assert capturedCallback non-null before invoking in both async
  repository tests, so stubbing mismatches produce an immediate
  clear failure rather than a silent timeout
- Apply TIMED_WAITING wait pattern to the "no data" async test,
  making it verify hasFlags() actually blocks rather than relying
  on a result that is coincidentally false either way
- Add bounded wait (5s) to the executor thread state poll loop in
  the integration test so a null executorThread produces a fast
  failure with a descriptive message instead of an infinite spin
newSingleThreadExecutor keeps its worker thread in WAITING when idle
between tasks, not TERMINATED. If hasFlags() returns without blocking
(regression), the thread finishes and sits in WAITING indefinitely.
The poll loop never exits until the 5s check fires. Adding WAITING as
a break condition makes regressions fail fast on the assertion rather
than spinning for 5 seconds.
The async threading in the two new repository tests and the integration
test was testing implementation details (that hasFlags() blocks) rather
than observable behavior. All that complexity also introduced multiple
rounds of review feedback about race conditions, infinite loops, and
WAITING vs TIMED_WAITING thread states.

Replace with synchronous datastore stubs that fire the callback during
DefaultFlagsRepository construction — the same pattern already used
throughout the test file. The integration test now uses mockExecutorService
(synchronous execution) instead of a real executor. The timeout test
(`persistence callback never fires within timeout`) remains and is the
only test that requires the latch to not be pre-counted-down.
@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot merged commit baf2d2f into develop Mar 25, 2026
27 checks passed
@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot deleted the typo/fix-stale-cache-cold-start branch March 25, 2026 00:24
@ncreated
ncreated restored the typo/fix-stale-cache-cold-start branch April 9, 2026 09:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants