Description of the bug:

We're building code around a fairly large repo and we noticed Bazel server consistently OOMing when --remote_download_toplevel is used.

A heap dump was made ( unfortunately i don't think I can share it :( ), which points to four threads, all named "cli-update-thread" local variables taking roughly 80% of the memory. The variables stack ( from thread gc root inwards ):

UIEventHandler.stateTracker.executionProgressReceiver.eventBus.subscribers.table[0] ( of type InMemoryMemoizingEvaluator )
evaluator.graph.nodeMap -> contains many, many, many entries.

( Each of the threads seems to have their own InMemoryMemoizingEvaluator )

It should be noted that this happens very consistently using BWOB and does not happen basically at all using non-BWOB builds.

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Make a large build with BWOB running over a rather large repository. It won't crash every time, but it will crash eventually.

Which operating system are you running Bazel on?

Linux

What is the output of bazel info release?

release 6.1.0

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse master; git rev-parse HEAD ?

No response

Have you found anything relevant by searching the web?

Nothing

Any other information, logs, or outputs that you want to share?

Bazel before OOM threw a lot of logs like this:

230419 11:46:00.012:I 19 [com.google.devtools.build.lib.skyframe.HighWaterMarkLimiter.handle] Dropping unnecessary temporary state in response to memory pressure. actual=99 threshold=85 [CONTEXT ratelimit_period="10 SECONDS [skipped: 1]" ]
230419 11:46:02.591:I 19 [com.google.devtools.build.lib.metrics.PostGCMemoryUseRecorder.handleNotification] Memory use after full GC: 5546637224