EDIT: the problem has been substantially improved by #12297; check out the write-up there for details.

What did you do?

I observed a Prometheus where prometheus_tsdb_head_series varied from 4 million to 7 million over each compaction cycle.

The number of series in the WAL checkpoint is not observable via a metric, so I downloaded it and used the following code, adapted from TestReadCheckpointMultipleSegments to see the number of series:

	wt := newWriteToMock()
	watcher := NewWatcher(wMetrics, nil, nil, "", wt, dir, false, false)
	watcher.MaxSegment = -1
	watcher.setMetrics()

	lastCheckpoint, _, err := LastCheckpoint(watcher.walDir)
	err = watcher.readCheckpoint(lastCheckpoint, (*Watcher).readSegment)
	fmt.Println(len(wt.seriesSegmentIndexes))

What did you expect to see?

About 7 million series in the WAL checkpoint (same as the max observed number of series).

Reasoning: a WAL checkpoint is generated every head compaction* (= 2 hours with default settings) and the checkpoint covers about the same time, so the checkpoint should have about the same number of series as I observe in the head.

* unless the amount of data being collected is small, in which case it's every 2 head compactions.

What did you see instead? Under which circumstances?

The WAL checkpoint had 18 million series in it.

Prometheus version

I was looking at 2.41 but further tests show the same thing happens in 2.43.