Skip to content

Commit 7deed96

Browse files
uros-buros-db
andcommitted
[MINOR][SS] Use filterNot in HDFSBackedStateStoreProvider snapshot filtering
### What changes were proposed in this pull request? Replaces `filter(_.isSnapshot == false)` with the equivalent, idiomatic `filterNot(_.isSnapshot)` in `HDFSBackedStateStoreProvider.doSnapshot`. ### Why are the changes needed? `StoreFile.isSnapshot` is a plain `Boolean` field, so comparing it to `false` is redundant. `filterNot` is the idiomatic form used throughout Spark and reads more clearly. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Behavior-preserving one-line change; existing state-store tests cover the path. No new tests needed. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 4.8) Closes #57519 from uros-b/cleanup-hdfs-state-filternot. Lead-authored-by: Uros Bojanic <[email protected]> Co-authored-by: Uros Bojanic <[email protected]> Signed-off-by: Uros Bojanic <[email protected]> (cherry picked from commit fd31694) Signed-off-by: Uros Bojanic <[email protected]>
1 parent af68ad7 commit 7deed96

1 file changed

Lines changed: 1 addition & 1 deletion

File tree

‎sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1092,7 +1092,7 @@ private[sql] class HDFSBackedStateStoreProvider extends StateStoreProvider with
10921092
if (files.nonEmpty) {
10931093
val lastVersion = files.last.version
10941094
val deltaFilesForLastVersion =
1095-
filesForVersion(files, lastVersion).filter(_.isSnapshot == false)
1095+
filesForVersion(files, lastVersion).filterNot(_.isSnapshot)
10961096
synchronized { Option(loadedMaps.get(lastVersion)) } match {
10971097
case Some(map) =>
10981098
if (deltaFilesForLastVersion.size > storeConf.minDeltasForSnapshot) {

0 commit comments

Comments
 (0)