Skip to content

Add new monitoring service for prod services #10522

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
mekarpeles merged 10 commits into from
Mar 6, 2025
Merged

Add new monitoring service for prod services #10522

mekarpeles merged 10 commits into from
Mar 6, 2025

Conversation

@cdrini
Copy link
Collaborator

@cdrini cdrini commented Mar 1, 2025 •
edited by mekarpeles
Loading

Closes #10267

Related: #10550

Also merge: https://github.com/internetarchive/olsystem/pull/261

Breakdown

Solr

Screenshot 2025-09-23 at 11 37 33 AM

Worker Load

Screenshot 2025-09-23 at 11 36 45 AM

Bots

Screenshot 2025-09-23 at 11 37 01 AM

Adding Monitoring to a new Host

Technical

Testing

Copied over the compose.production.yaml and scripts/monitoring/Dockerfile to ol-covers0, and added to the bottom of that Dockerfile:

# TMP
RUN python -m pip install APScheduler==3.11.0
USER openlibrary
RUN rm scripts/monitoring/requirements.txt && HTTPS_PROXY="..." curl 'https://patch-diff.githubusercontent.com/raw/internetarchive/openlibrary/pull/10522.diff' | git apply
USER root

Then was able to run the new olsystem script to build the image, and up! It is currently running on covers and sending events to our dashboards:

$ docker compose logs -f monitoring
openlibrary-monitoring-1  | Starting monitoring on ol-covers0.us.archive.org
openlibrary-monitoring-1  | 3 job(s) registered:
openlibrary-monitoring-1  | log_workers_cur_fn (trigger: interval[0:01:00], pending)
openlibrary-monitoring-1  | log_recent_bot_traffic (trigger: interval[0:01:00], pending)
openlibrary-monitoring-1  | log_recent_http_statuses (trigger: interval[0:01:00], pending)
openlibrary-monitoring-1  | Monitoring started (ol-covers0.us.archive.org)
...

Screenshot

Stakeholders

@cdrini cdrini force-pushed the monitoring branch 3 times, most recently from b4bbc44 to 1f60eaa Compare March 3, 2025 23:30
@cdrini cdrini force-pushed the monitoring branch 2 times, most recently from 613b76c to f10bd49 Compare March 4, 2025 02:28
@cdrini cdrini marked this pull request as ready for review March 4, 2025 02:46
@mekarpeles mekarpeles merged commit 6da514e into internetarchive:master Mar 6, 2025
4 checks passed
@cdrini cdrini deleted the monitoring branch March 6, 2025 19:37
@cdrini cdrini mentioned this pull request Mar 7, 2025
Merged
@mekarpeles mekarpeles added Lead: @mekarpeles Issues overseen by Mek (Staff: Program Lead) [managed] Module: Monitoring Framework Type: Epic A feature or refactor that is big enough to require subissues. [managed] Theme: Monitoring labels Sep 23, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

Lead: @mekarpeles Issues overseen by Mek (Staff: Program Lead) [managed] Module: Monitoring Framework Theme: Monitoring Type: Epic A feature or refactor that is big enough to require subissues. [managed]

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Create MonitorDaemon docker container for ol-www0 to monitor HTTP status codes

2 participants

@cdrini @mekarpeles