Skip to content

Improved sink debugging flow.#821

Merged
bwplotka merged 2 commits intomasterfrom
prw2
Jan 31, 2025
Merged

Improved sink debugging flow.#821
bwplotka merged 2 commits intomasterfrom
prw2

Conversation

@bwplotka
Copy link
Copy Markdown
Member

@bwplotka bwplotka commented Jan 31, 2025 •
edited
Loading

This improves PRW benchmark flow by:

  • Adding PRW 2.0 traffic which we should care about with the next Prometheus releases. We could drop PRW 1.0 communication given no development on that path, but since ppl still use it, let's keep it. I removed it for now as it depends on metadata-wal-records feature which is problematic currently: Deprecate and ultimately remove metadata-wal-records feature enabled. prometheus#15911
  • Updating sink with the latest issue handling features.
  • Creating different configmap for each Prometheus in the same file. This is to ensure we have pass a different PRW header so in the sink we will know which issue belongs to what Prometheus (e.g. prometheus from the PR is suddenly not sending help).
  • Meta Prom scrapes now sink

@ArthurSens
Copy link
Copy Markdown
Member

ArthurSens commented Jan 31, 2025 •
edited
Loading

Would you like to try running a custom benchmark from this branch to make sure it works?

bwplotka added a commit to prometheus/prometheus that referenced this pull request Jan 31, 2025
@bwplotka
[EXPERIMENT] Prombench #821 test
daaba22 
Testing prometheus/test-infra#821

Signed-off-by: Bartlomiej Plotka <[email protected]>
@bwplotka bwplotka mentioned this pull request Jan 31, 2025
Closed
@bwplotka
Copy link
Copy Markdown
Member Author

bwplotka commented Jan 31, 2025

I didn't plan this, no - wanted to spend my time somewhere else (low risk, low impact, can be fixed in seconds) 🙃

But given you pointed out, let's do this: prometheus/prometheus#15914

@bwplotka
Copy link
Copy Markdown
Member Author

bwplotka commented Jan 31, 2025

Well glad you did asked for it

2025-01-31 13:09:29.996 GMT
time=2025-01-31T13:09:29.996Z level=ERROR source=main.go:1465 msg="Failed to apply configuration" err="invalid remote write configuration, if you are using remote write version 2.0 the --enable-feature=metadata-wal-records feature flag must be enabled"

(:

@bwplotka bwplotka changed the title Added PRW 2.0 traffic, improved sink debugging flow. Improved sink debugging flow. Jan 31, 2025
@bwplotka
Copy link
Copy Markdown
Member Author

bwplotka commented Jan 31, 2025 •
edited
Loading

Removed PRW2 comms for now as it depends on metadata-wal-records feature which is problematic currently: prometheus/prometheus#15911

@bwplotka
Copy link
Copy Markdown
Member Author

bwplotka commented Jan 31, 2025

Tested prometheus/prometheus#15914 works now.

ArthurSens
ArthurSens approved these changes Jan 31, 2025
View reviewed changes
Copy link
Copy Markdown
Member

@ArthurSens ArthurSens left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Idea for the future: adding the new sink metrics/logs to a separate dashboard :)

@bwplotka bwplotka merged commit 4ff03e0 into master Jan 31, 2025
@bwplotka bwplotka deleted the prw2 branch January 31, 2025 14:32
kakkoyun
kakkoyun reviewed Jan 31, 2025
View reviewed changes
Copy link
Copy Markdown
Member

@kakkoyun kakkoyun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@bboreham
Copy link
Copy Markdown
Member

bboreham commented Feb 5, 2025

Why did the filenames change?
e.g. 3b_prometheus-test-pr_deployment.yaml -> 3_prometheus-test-pr_deployment.yaml

kushalShukla-web pushed a commit to kushalShukla-web/test-infra that referenced this pull request Feb 8, 2025
@bwplotka
Improved sink debugging flow. (prometheus#821)
4371345 
* Add PRW 2.0 traffic, improved sink debugging flow.

Signed-off-by: bwplotka <[email protected]>

* Removed PRW 2.0 as it depends on expensive metadata-wal-records feature.

Signed-off-by: bwplotka <[email protected]>

---------

Signed-off-by: bwplotka <[email protected]>
@bwplotka bwplotka mentioned this pull request Feb 17, 2025
Merged
@bboreham bboreham mentioned this pull request Feb 17, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@kakkoyun kakkoyun kakkoyun left review comments

@ArthurSens ArthurSens ArthurSens approved these changes

@bboreham bboreham Awaiting requested review from bboreham

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@bwplotka @ArthurSens @bboreham @kakkoyun