Fix: cold-start stale cache: hasFlags() now waits for persistence load#3282
Conversation
…tence load On cold start, if the first network refresh fails, the flags module would transition to Error instead of Stale even when valid cached flags existed. Root cause: DefaultFlagsRepository.hasFlags() read atomicState directly without calling waitForPersistenceLoad(), unlike every other read method. The persistence latch had not been counted down yet at the moment hasFlags() was called inside EvaluationsManager, so it returned false and the module reported Error instead of falling back to cached Stale flags. Fix: add waitForPersistenceLoad() call in hasFlags(), consistent with all other read methods. This is bounded to at most persistenceLoadTimeoutMs (default 100ms) on first call only, on a background executor thread.
There was a problem hiding this comment.
Pull request overview
Fixes a cold-start race where DefaultFlagsRepository.hasFlags() could read atomicState before persistence finished loading, causing EvaluationsManager to incorrectly transition to Error instead of Stale when cached flags existed and the network failed.
Changes:
- Updated
DefaultFlagsRepository.hasFlags()to callwaitForPersistenceLoad()before checking cached flags. - Reworked
DefaultFlagsRepositoryTestcoverage forhasFlags()to validate behavior with async persistence load, empty persisted data, and timeout. - Added an
EvaluationsManagerTestintegration-style test using a realDefaultFlagsRepositoryto verify the cold-start stale-cache scenario.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 6 comments.
| File | Description |
|---|---|
| features/dd-sdk-android-flags/src/main/kotlin/com/datadog/android/flags/internal/repository/DefaultFlagsRepository.kt | Makes hasFlags() consistent with other read APIs by awaiting persistence load before reading state. |
| features/dd-sdk-android-flags/src/test/kotlin/com/datadog/android/flags/internal/repository/DefaultFlagsRepositoryTest.kt | Adds async/timeout tests for the new hasFlags() behavior; removes the previous “does not block” contract. |
| features/dd-sdk-android-flags/src/test/kotlin/com/datadog/android/flags/internal/evaluation/EvaluationsManagerTest.kt | Adds an integration test validating Stale (not Error) on cold start with cached flags + network failure. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
This comment has been minimized.
This comment has been minimized.
- Switch all remaining version = any() stubs to anyOrNull() so the persistence callback is actually exercised rather than relying on timeout fallback - Reduce persistenceLoadTimeoutMs from 5000ms to 500ms in async tests so regressions fail fast - Remove flaky lower-bound elapsed-time assertion (currentTimeMillis resolution is too coarse on some JVMs) - Assert capturedCallback non-null before invoking in integration test to surface misconfigured mocks immediately - Assert awaitTermination returns true and call shutdownNow() in finally to avoid leaking threads in failing test runs
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Both tests previously fired the persistence callback before hasFlags() was called, meaning they could pass even without the fix. Now: - Repository test: runs hasFlags() on a background thread and polls Thread.State.TIMED_WAITING before firing the callback, ensuring the test fails if hasFlags() does not block on the persistence latch. - Integration test: captures the executor thread via a custom ThreadFactory and applies the same TIMED_WAITING poll before firing the callback, guaranteeing the executor is blocked in hasFlags() when the persistence callback fires.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## develop #3282 +/- ##
===========================================
- Coverage 71.63% 71.60% -0.03%
===========================================
Files 943 943
Lines 34815 34816 +1
Branches 5899 5899
===========================================
- Hits 24937 24928 -9
- Misses 8244 8248 +4
- Partials 1634 1640 +6
🚀 New features to boost your workflow:
|
Polling loops that wait for Thread.State.TIMED_WAITING would spin forever if hasFlags() returned without blocking (regression), because the thread would reach TERMINATED rather than TIMED_WAITING. Adding TERMINATED as a break condition lets the loop exit immediately in the regression case, after which the assertion fails fast with a clear error rather than hanging the test suite.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 483fdf4ed4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
- Assert capturedCallback non-null before invoking in both async repository tests, so stubbing mismatches produce an immediate clear failure rather than a silent timeout - Apply TIMED_WAITING wait pattern to the "no data" async test, making it verify hasFlags() actually blocks rather than relying on a result that is coincidentally false either way - Add bounded wait (5s) to the executor thread state poll loop in the integration test so a null executorThread produces a fast failure with a descriptive message instead of an infinite spin
newSingleThreadExecutor keeps its worker thread in WAITING when idle between tasks, not TERMINATED. If hasFlags() returns without blocking (regression), the thread finishes and sits in WAITING indefinitely. The poll loop never exits until the 5s check fires. Adding WAITING as a break condition makes regressions fail fast on the assertion rather than spinning for 5 seconds.
The async threading in the two new repository tests and the integration test was testing implementation details (that hasFlags() blocks) rather than observable behavior. All that complexity also introduced multiple rounds of review feedback about race conditions, infinite loops, and WAITING vs TIMED_WAITING thread states. Replace with synchronous datastore stubs that fire the callback during DefaultFlagsRepository construction — the same pattern already used throughout the test file. The integration test now uses mockExecutorService (synchronous execution) instead of a real executor. The timeout test (`persistence callback never fires within timeout`) remains and is the only test that requires the latch to not be pre-counted-down.
What does this PR do?
Fixes a bug where the flags module reported
Errorinstead ofStaleon cold start when a network failure occurred and valid cached flags existed in persistent storage.Motivation
DefaultFlagsRepository.hasFlags()was readingatomicStatedirectly without callingwaitForPersistenceLoad(), unlike every other read method (getPrecomputedFlag,getFlagsSnapshot,getEvaluationContext,getPrecomputedFlagWithContext). On cold start, the async persistence callback had not yet fired whenEvaluationsManagercalledhasFlags()to determine whether to fall back to cachedStaleflags. The latch was not yet counted down, soatomicStatewas null,hasFlags()returnedfalse, and the module transitioned toError.Changes
DefaultFlagsRepository.kt:hasFlags()now callswaitForPersistenceLoad()before readingatomicState, consistent with all other read methods. The wait is bounded bypersistenceLoadTimeoutMs(default 100ms), only on the first call, only on the backgroundflags-networkexecutor thread — no deadlock risk, no main-thread impact.DefaultFlagsRepositoryTest.kt: Removed the test that assertedhasFlags()does not block (it encoded the bug as a contract). Added three new tests covering: async callback with flags, async callback with no data, and timeout with no callback.EvaluationsManagerTest.kt: Added an integration test using a realDefaultFlagsRepositorythat verifies the cold-start scenario ends inStale(notError) when cached flags exist, the context matches, and the network fails.Additional Notes
Thread safety: the persistence latch is counted down by the datastore I/O thread (or by
setFlagsAndContext()on a successful fetch), which is completely separate from theflags-networkexecutor thread that callshasFlags(). No deadlock is possible.