Skip to content

better mysql healthcheck#2459

Merged
Hayden-IO merged 2 commits intosigstore:mainfrom
ramonpetgrave64:mysql-healthchecks
Apr 22, 2025
Merged

better mysql healthcheck#2459
Hayden-IO merged 2 commits intosigstore:mainfrom
ramonpetgrave64:mysql-healthchecks

Conversation

@ramonpetgrave64
Copy link
Copy Markdown
Contributor

@ramonpetgrave64 ramonpetgrave64 commented Apr 22, 2025

Summary

Improves the healthcheck for mysql in the docker-compose.yml file.

See docker-library/mysql#930.

With the previous healthcheck, the mysql service may be running, but the actual database may not be ready yet.

 ➜  rekor git:(main) docker compose up --wait
[+] Running 5/6
 ✔ Network rekor_default                  Created          0.1s 
 ✔ Container rekor-redis-server-1         Healthy         94.4s 
 ✔ Container rekor-mysql-1                Healthy         94.4s 
 ✔ Container rekor-trillian-log-signer-1  Healthy         94.6s 
 ✔ Container rekor-trillian-log-server-1  Healthy         94.3s 
 ⠦ Container rekor-rekor-server-1         Waiting         96.1s 
container rekor-rekor-server-1 is unhealthy

With the newer healthcheck mysql takes longer to be healthy, but docker compose up --wait succeeds.

➜  rekor git:(main) ✗ docker compose up --wait
 ✔ Network rekor_default                  Created          0.2s 
[+] Running 5/6kor-redis-server-1         Started          1.4s 
[+] Running 6/6r_default                  Created          0.2s 
 ✔ Network rekor_default                  Created          0.2s 
 ✔ Container rekor-redis-server-1         Healthy        128.3s 
 ✔ Container rekor-mysql-1                Healthy        128.4s 
 ✔ Container rekor-trillian-log-server-1  Healthy        128.2s 
 ✔ Container rekor-trillian-log-signer-1  Healthy        128.2s 
 ✔ Container rekor-rekor-server-1         Healthy        133.5s

TODO

  • add this same healthceck to fulcio's.

Release Note

Improve docker-compose.yml healthcheck for mysql.

Documentation

@ramonpetgrave64
better mysql healthcheck
dbc7f59 
Signed-off-by: Ramon Petgrave <[email protected]>
@ramonpetgrave64 ramonpetgrave64 requested a review from a team as a code owner April 22, 2025 16:20
@ramonpetgrave64 ramonpetgrave64 mentioned this pull request Apr 22, 2025
Merged
5 tasks
Hayden-IO
Hayden-IO reviewed Apr 22, 2025
View reviewed changes
@Hayden-IO
Copy link
Copy Markdown
Contributor

Hayden-IO commented Apr 22, 2025

These startup times are getting really high, this shouldn't take 2 minutes to startup. Do we need to be lowering intervals to be more frequent?

@codecov
Copy link
Copy Markdown

codecov bot commented Apr 22, 2025 •
edited
Loading

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 25.25%. Comparing base (488eb97) to head (0a801cb).
Report is 384 commits behind head on main.

Additional details and impacted files 
@@             Coverage Diff             @@
##             main    #2459       +/-   ##
===========================================
- Coverage   66.46%   25.25%   -41.21%     
===========================================
  Files          92      192      +100     
  Lines        9258    24874    +15616     
===========================================
+ Hits         6153     6283      +130     
- Misses       2359    17815    +15456     
- Partials      746      776       +30
Flag Coverage Δ
e2etests 46.64% <ø> (-0.92%) ⬇️
unittests 16.49% <ø> (-31.20%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@ramonpetgrave64
increase retries, lower interval
0a801cb 
Signed-off-by: Ramon Petgrave <[email protected]>
@Hayden-IO
Copy link
Copy Markdown
Contributor

Hayden-IO commented Apr 22, 2025

Running on macOS, docker compose up --wait takes about ~15s so I'm not sure why you're seeing those numbers.

However I did see an issue - when rekor-server first starts up it reports unhealthy because the log server is not yet up so it panics. The server recovers and restarts, and eventually becomes healthy.

@ramonpetgrave64
Copy link
Copy Markdown
Contributor Author

ramonpetgrave64 commented Apr 22, 2025

Yes, it's slower on my linux machine, maybe because of my partition type.

Hayden-IO
Hayden-IO approved these changes Apr 22, 2025
View reviewed changes
Copy link
Copy Markdown
Contributor

@Hayden-IO Hayden-IO left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll merge this, but I think this doesn't work unless there's artificial delays on starting up the log server.

@Hayden-IO Hayden-IO merged commit 5361696 into sigstore:main Apr 22, 2025
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Hayden-IO Hayden-IO Hayden-IO approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@ramonpetgrave64 @Hayden-IO