What this PR does / why we need it:

This PR implements a new mechanism for the wal Watcher in Promtail, to know there are new records to be read. It uses a combination of:

• WIP implement doubling backoff for WAL watcher timer prometheus/prometheus#11950
• [tsdb] re-implement WAL watcher to read via a "notification" channel prometheus/prometheus#11949

The main idea is that the primary mechanism is a notification channel between the wal.Writer and wal.Watcher. The Watcher subscribes to write events the writer publishes, getting notified if the wal has been written. The same subscriptions design is used for cleanup events.

As a backup, the watcher has a timer that implements an exponential backoff strategy, which is constrained by a minimum and maximum that the user can configure.

Below the cpu difference is shown of running both main and this branch against the same scrape target.

The yellow line is the latest main build from where this branch started, and the green line is this branch. Both promtails tailing docker logs, and using the following metrics to get cpu usage from cadvisor:

avg by (name) (rate(container_cpu_usage_seconds_total{job=~".+", instance=~".+", name=~"promtail-wal-test_promtail.+"}[$__rate_interval]))

Which issue(s) this PR fixes:
Part of #8197

Special notes for your reviewer:

Checklist

• Reviewed the CONTRIBUTING.md guide (required)
• Documentation added
• Tests updated
• CHANGELOG.md updated
• Changes that require user attention or interaction to upgrade are documented in docs/sources/upgrading/_index.md