[flags] fix: use read lock for removeListener#3132
Conversation
This comment has been minimized.
This comment has been minimized.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## develop #3132 +/- ##
===========================================
+ Coverage 71.27% 71.42% +0.15%
===========================================
Files 929 929
Lines 34450 34459 +9
Branches 5817 5817
===========================================
+ Hits 24554 24611 +57
+ Misses 8257 8228 -29
+ Partials 1639 1620 -19
🚀 New features to boost your workflow:
|
| @Test | ||
| fun `M block getCurrentState calls W addListener() { slow listener notification }`() { | ||
| // Given | ||
| val stateOld = FlagsClientState.NotReady |
There was a problem hiding this comment.
nit: probably should be oldState to be consistent with naming convention used in newState.
Or maybe for the clarity we can just inline FlagsClientState.NotReady where needed, because stateOld is val anyway and cannot be changed.
| addListenerThread.start() | ||
| addListenerStarted.await() | ||
| getCurrentStateThread.start() | ||
| getCurrentStateAttempted.await() |
There was a problem hiding this comment.
getCurrentStateAttempted is not actually needed I think, the necessary wait for the getCurrentStateThread completion is already implemented by the getCurrentStateThread.join() call below.
| val addListenerSlowCallbackStarted = CountDownLatch(1) | ||
| val getCurrentStateAttempted = CountDownLatch(1) | ||
|
|
||
| val operationTimestamps = mutableListOf<Pair<String, Long>>() |
There was a problem hiding this comment.
synchronized calls can be removed if type here is CopyOnWriteArrayList
| } | ||
|
|
||
| @Test | ||
| fun `M block getCurrentState calls W addListener() { slow listener notification }`() { |
There was a problem hiding this comment.
Why do we want to block getCurrentState when addListener is called, if addListener is not changing the state?
There was a problem hiding this comment.
🤔 We may have had the right lock here in the first place...
If addListener holds the read lock, then, ostensibly, nothing can write to the state (current state or the listeners). While addListener is technically mutating the protected state by adding a listener, the underlying mechanism itself is theadsafe/synchronized via CopyOnWriteArrayList so it's fine if parallel calls to addListener attempt to mutate the set of listeners. The order of listener subscription is not important to maintain here.
We should just be able to rely on the underlying thread-safety of CopyOnWriteArrayList to avoid clobbering of the listener list, right?
There was a problem hiding this comment.
Maybe we shouldn't wrap adding listener with lock then?
There was a problem hiding this comment.
We need to block writing to the current state until the first notification and addListener are complete. We don't get that atomicity with DDCoreSubscription, only that the calls underlying calls to add the listener to the list are thread safe and synchronized so no listener will be lost when parallel calls are made.
In effect, I believe we can drop this change altogether and rely on the read lock in conjunction with the underlying lock in CopyOnWriteArrayList
There was a problem hiding this comment.
I think yes, we should. My thinking is the following:
getCurrentStatecovers onlyFlagsStateManager.currentState. Listeners is not a part ofFlagsStateManager.currentState, so they are not related to thegetCurrentStatecall.FlagsStateManager.addListenercall doesn't modifyFlagsStateManager.currentState, it only reads. So that us why we added a read lock. If something is in the process of modifyingFlagsStateManager.currentState, then there will be a wait to acquirereadlock.FlagsStateManager.addListenermutates underlying listeners collection, but anyway this is atomic and thread-safe,currentStatevalue passed down is guarded by the read lock and if there is iteration already happening inupdateState -> subscription.notifyListeners, then this new listener won't be a part of ongoing iteration (due to the usage ofCopyOnWriteArrayListinDDSubscription).
So I'm curious why should we block getCurrentState call during FlagsStateManager.addListener invocation.
The race condition could otherwise result in a listener missing a state change
Is it about FlagsStateManager.currentState property here? From the code I see, listener shouldn't miss any state change, since read lock cannot be acquired if write lock is active, and the opposite.
There was a problem hiding this comment.
Well, having read it more carefully, I agree with @nikita. The only thing that still makes me curious is the fact that we adding listener with a lock, but removing without. Yes, underlying CopyOnWriteArrayList will protect us from the ConcurrentModificationException, however if updateState and removeListener will be executed at the same time, removed listener could get an undesired update (from subscription.notifyListeners ) and could lead to undefined behavior. Not sure if this is a critical issue, but I'd prefer to maintain consistency here.
There was a problem hiding this comment.
Thanks @satween. Agreed that there's an apparent lack of synchronicity on removing a listener.
We are not just "Adding a listener", however.
We need to
- get the current state
- notify the listener
- add the listener to the list
All without thecurrentStatechanging, so we block changes with areadlock.
When we remove the listener, have only that operation to complete so there isn't a risk of missing data, only, as you noted, receiving extra data. We would need to add an extra synchronization shared between the removeListener method and the updateState method. I think you're correct here though - a long-running listener callback could cause a removed listener to be notified after the call to removeListener completes
The scale of adding/removing listeners will be very low (0, or 1 or maybe a few) so there is a very small risk here of a superfluous event.
I'll come back to this with an updated lock after a couple of other tasks
There was a problem hiding this comment.
Updated to use a read lock on both add and remove listener to ensure that state changes are not sent to removed listeners
|
Updated the locks. PTAL |
|
@codex review |
|
Codex Review: Didn't find any major issues. Keep it up! ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
|
||
| // Then | ||
| // After removeListener returns, no more notifications should be received | ||
| synchronized(notificationsAfterRemove) { |
There was a problem hiding this comment.
I don't quite understand why do we need synchronized in the assertions block. Assertions are supposed to be executed only when everything is completed and updateStateThread.join() should guarantee safe read without this block anyway (and it is not needed if CopyOnWriteArrayList is used as well).
| override fun onStateChanged(newState: FlagsClientState) { | ||
| // Only track notifications that happen after removeListener completed | ||
| if (removeListenerCompleted.count == 0L) { | ||
| synchronized(notificationsAfterRemove) { |
There was a problem hiding this comment.
A bit more fair will be to use CopyOnWriteArrayList for notificationsAfterRemove instead of using synchronized here.
| // Listener that is slow during updateState notification | ||
| val slowListener = object : FlagsStateListener { | ||
| override fun onStateChanged(newState: FlagsClientState) { | ||
| if (newState == stateNew) { |
There was a problem hiding this comment.
nit: having newState and stateNew is a bit confusing. Maybe we should rename stateNew to readyState?
|
|
||
| // updateState should have been delayed by at least the sleep time | ||
| assertThat(updatedStateTime - initialStateTime).isGreaterThan(10_000_000) // ~10ms in nanoseconds | ||
| assertThat(initialStateTime).isLessThan(updatedStateTime) |
There was a problem hiding this comment.
nit: the line above already imposes that initialStateTime is less than updatedStateTime. If it is not the case, this line won't be even executed.
|
/merge |
|
View all feedbacks in Devflow UI.
This pull request is not mergeable according to GitHub. Common reasons include pending required checks, missing approvals, or merge conflicts — but it could also be blocked by other repository rules or settings.
The expected merge time in
|
What does this PR do?
Uses a
readlock in theremoveListenermethod.This locking fixes a race condition between
removeListenerandupdateStateMotivation
Removing a listener needs to block state changes so that removed listeners do not receive notifications after removal.
Additional Notes
Anything else we should know when reviewing?
Review checklist (to be filled by reviewers)