Skip to content

[flags] fix: use read lock for removeListener#3132

Merged
gh-worker-dd-mergequeue-cf854d[bot] merged 5 commits into
developfrom
typo/writelock
Feb 20, 2026
Merged

[flags] fix: use read lock for removeListener#3132
gh-worker-dd-mergequeue-cf854d[bot] merged 5 commits into
developfrom
typo/writelock

Conversation

@typotter

@typotter typotter commented Jan 19, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

Uses a read lock in the removeListener method.
This locking fixes a race condition between removeListener and updateState

Motivation

Removing a listener needs to block state changes so that removed listeners do not receive notifications after removal.

Additional Notes

Anything else we should know when reviewing?

Review checklist (to be filled by reviewers)

  • Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
  • Make sure you discussed the feature or bugfix with the maintaining team in an Issue
  • Make sure each commit and the PR mention the Issue number (cf the CONTRIBUTING doc)

@typotter
typotter marked this pull request as ready for review January 19, 2026 17:35
@typotter
typotter requested a review from a team as a code owner January 19, 2026 17:35
@typotter typotter changed the title fix: use write lock for addListener [flags] fix: use write lock for addListener Jan 19, 2026
@datadog-official

This comment has been minimized.

@codecov-commenter

codecov-commenter commented Jan 19, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 71.42%. Comparing base (a90ef93) to head (ad4c4c8).
⚠️ Report is 265 commits behind head on develop.

Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #3132      +/-   ##
===========================================
+ Coverage    71.27%   71.42%   +0.15%     
===========================================
  Files          929      929              
  Lines        34450    34459       +9     
  Branches      5817     5817              
===========================================
+ Hits         24554    24611      +57     
+ Misses        8257     8228      -29     
+ Partials      1639     1620      -19     
Files with missing lines Coverage Δ
...atadog/android/flags/internal/FlagsStateManager.kt 92.86% <100.00%> (+0.55%) ⬆️

... and 36 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@Test
fun `M block getCurrentState calls W addListener() { slow listener notification }`() {
// Given
val stateOld = FlagsClientState.NotReady

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: probably should be oldState to be consistent with naming convention used in newState.

Or maybe for the clarity we can just inline FlagsClientState.NotReady where needed, because stateOld is val anyway and cannot be changed.

addListenerThread.start()
addListenerStarted.await()
getCurrentStateThread.start()
getCurrentStateAttempted.await()

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

getCurrentStateAttempted is not actually needed I think, the necessary wait for the getCurrentStateThread completion is already implemented by the getCurrentStateThread.join() call below.

val addListenerSlowCallbackStarted = CountDownLatch(1)
val getCurrentStateAttempted = CountDownLatch(1)

val operationTimestamps = mutableListOf<Pair<String, Long>>()

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

synchronized calls can be removed if type here is CopyOnWriteArrayList

}

@Test
fun `M block getCurrentState calls W addListener() { slow listener notification }`() {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we want to block getCurrentState when addListener is called, if addListener is not changing the state?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤔 We may have had the right lock here in the first place...

If addListener holds the read lock, then, ostensibly, nothing can write to the state (current state or the listeners). While addListener is technically mutating the protected state by adding a listener, the underlying mechanism itself is theadsafe/synchronized via CopyOnWriteArrayList so it's fine if parallel calls to addListener attempt to mutate the set of listeners. The order of listener subscription is not important to maintain here.

We should just be able to rely on the underlying thread-safety of CopyOnWriteArrayList to avoid clobbering of the listener list, right?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we shouldn't wrap adding listener with lock then?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to block writing to the current state until the first notification and addListener are complete. We don't get that atomicity with DDCoreSubscription, only that the calls underlying calls to add the listener to the list are thread safe and synchronized so no listener will be lost when parallel calls are made.

In effect, I believe we can drop this change altogether and rely on the read lock in conjunction with the underlying lock in CopyOnWriteArrayList

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think yes, we should. My thinking is the following:

  • getCurrentState covers only FlagsStateManager.currentState. Listeners is not a part of FlagsStateManager.currentState, so they are not related to the getCurrentState call.
  • FlagsStateManager.addListener call doesn't modify FlagsStateManager.currentState, it only reads. So that us why we added a read lock. If something is in the process of modifying FlagsStateManager.currentState, then there will be a wait to acquire read lock.
  • FlagsStateManager.addListener mutates underlying listeners collection, but anyway this is atomic and thread-safe, currentState value passed down is guarded by the read lock and if there is iteration already happening in updateState -> subscription.notifyListeners, then this new listener won't be a part of ongoing iteration (due to the usage of CopyOnWriteArrayList in DDSubscription).

So I'm curious why should we block getCurrentState call during FlagsStateManager.addListener invocation.

The race condition could otherwise result in a listener missing a state change

Is it about FlagsStateManager.currentState property here? From the code I see, listener shouldn't miss any state change, since read lock cannot be acquired if write lock is active, and the opposite.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, having read it more carefully, I agree with @nikita. The only thing that still makes me curious is the fact that we adding listener with a lock, but removing without. Yes, underlying CopyOnWriteArrayList will protect us from the ConcurrentModificationException, however if updateState and removeListener will be executed at the same time, removed listener could get an undesired update (from subscription.notifyListeners ) and could lead to undefined behavior. Not sure if this is a critical issue, but I'd prefer to maintain consistency here.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @satween. Agreed that there's an apparent lack of synchronicity on removing a listener.

We are not just "Adding a listener", however.
We need to

  • get the current state
  • notify the listener
  • add the listener to the list
    All without the currentState changing, so we block changes with a read lock.

When we remove the listener, have only that operation to complete so there isn't a risk of missing data, only, as you noted, receiving extra data. We would need to add an extra synchronization shared between the removeListener method and the updateState method. I think you're correct here though - a long-running listener callback could cause a removed listener to be notified after the call to removeListener completes

The scale of adding/removing listeners will be very low (0, or 1 or maybe a few) so there is a very small risk here of a superfluous event.

I'll come back to this with an updated lock after a couple of other tasks

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated to use a read lock on both add and remove listener to ensure that state changes are not sent to removed listeners

@typotter

Copy link
Copy Markdown
Contributor Author

Updated the locks. PTAL

@typotter typotter changed the title [flags] fix: use write lock for addListener [flags] fix: use read lock for removeListener Feb 17, 2026
@0xnm

0xnm commented Feb 18, 2026

Copy link
Copy Markdown
Member

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Keep it up!

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

0xnm
0xnm previously approved these changes Feb 18, 2026

// Then
// After removeListener returns, no more notifications should be received
synchronized(notificationsAfterRemove) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't quite understand why do we need synchronized in the assertions block. Assertions are supposed to be executed only when everything is completed and updateStateThread.join() should guarantee safe read without this block anyway (and it is not needed if CopyOnWriteArrayList is used as well).

override fun onStateChanged(newState: FlagsClientState) {
// Only track notifications that happen after removeListener completed
if (removeListenerCompleted.count == 0L) {
synchronized(notificationsAfterRemove) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A bit more fair will be to use CopyOnWriteArrayList for notificationsAfterRemove instead of using synchronized here.

// Listener that is slow during updateState notification
val slowListener = object : FlagsStateListener {
override fun onStateChanged(newState: FlagsClientState) {
if (newState == stateNew) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: having newState and stateNew is a bit confusing. Maybe we should rename stateNew to readyState?

0xnm
0xnm previously approved these changes Feb 18, 2026

// updateState should have been delayed by at least the sleep time
assertThat(updatedStateTime - initialStateTime).isGreaterThan(10_000_000) // ~10ms in nanoseconds
assertThat(initialStateTime).isLessThan(updatedStateTime)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: the line above already imposes that initialStateTime is less than updatedStateTime. If it is not the case, this line won't be even executed.

@typotter

Copy link
Copy Markdown
Contributor Author

/merge

@gh-worker-devflow-routing-ef8351

gh-worker-devflow-routing-ef8351 Bot commented Feb 20, 2026

Copy link
Copy Markdown

View all feedbacks in Devflow UI.

2026-02-20 15:19:51 UTC ℹ️ Start processing command /merge


2026-02-20 15:19:56 UTC ℹ️ MergeQueue: waiting for PR to be ready

This pull request is not mergeable according to GitHub. Common reasons include pending required checks, missing approvals, or merge conflicts — but it could also be blocked by other repository rules or settings.
It will be added to the queue as soon as checks pass and/or get approvals. View in MergeQueue UI.
Note: if you pushed new commits since the last approval, you may need additional approval.
You can remove it from the waiting list with /remove command.


2026-02-20 16:12:09 UTC ℹ️ MergeQueue: merge request added to the queue

The expected merge time in develop is approximately 1h (p90).


2026-02-20 17:12:59 UTC ℹ️ MergeQueue: This merge request was merged

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants