Skip to content

chore: improve mysql readiness checks#2397

Merged
bobcallaway merged 3 commits intosigstore:mainfrom
ramonpetgrave64:docker-compose-readiness-checks
Mar 11, 2025
Merged

chore: improve mysql readiness checks#2397
bobcallaway merged 3 commits intosigstore:mainfrom
ramonpetgrave64:docker-compose-readiness-checks

Conversation

@ramonpetgrave64
Copy link
Copy Markdown
Contributor

@ramonpetgrave64 ramonpetgrave64 commented Mar 7, 2025 •
edited
Loading

sigstore/rekor-tiles#38

Summary

Improves the docker-compose.yml file by enforcing healtchecks between mysql and the trillian services. It also increases the timeouts for the healthchecks.

See https://docs.docker.com/reference/dockerfile/#healthcheck

This improves the developer experience, so we don't see so many startup errors for the trillian services, like

 *  Executing task: docker logs --tail 1000 -f 6791b7d7b4e977cd371144b7a752018d4077d7b3c65a9a7b12a5d3da556bb3ef 

I0307 19:57:55.500513       1 main.go:97] **** Log Server Starting ****
W0307 19:57:55.502723       1 tree_storage.go:89] Failed to set strict mode on mysql db: dial tcp 192.168.8.4:3306: connect: connection refused
F0307 19:57:55.502831       1 main.go:118] Failed to get storage provider: dial tcp 192.168.8.4:3306: connect: connection refused
...
I0307 19:58:01.096252       1 main.go:97] **** Log Server Starting ****
W0307 19:58:01.098489       1 tree_storage.go:89] Failed to set strict mode on mysql db: dial tcp 192.168.8.4:3306: connect: connection refused
F0307 19:58:01.098555       1 main.go:118] Failed to get storage provider: dial tcp 192.168.8.4:3306: connect: connection refused
...

Testing process

On a personal machine, it can take 90s for mysql to be ready.

rekor git:(main) ✗ docker compose up -d
[+] Running 5/6
 ✔ Network rekor_default                  Created                                                0.2s 
 ✔ Container rekor-redis-server-1         Started                                                0.9s 
 ⠴ Container rekor-mysql-1                Waiting                                               74.6s 
 ✔ Container rekor-trillian-log-signer-1  Created                                                0.0s 
 ✔ Container rekor-trillian-log-server-1  Created                                                0.1s 
 ✔ Container rekor-rekor-server-1         Created                                                0.0s

trillian-log-signer starts cleanly

 *  Executing task: docker logs --tail 1000 -f e857bcb48a25bbe038a4100a7a68c50e94bf34e82756edfddf485193232cc151 

I0307 19:50:31.772888       1 main.go:108] **** Log Signer Starting ****
W0307 19:50:31.776053       1 main.go:147] **** Acting as master for all logs ****
I0307 19:50:31.776630       1 operation_manager.go:328] Log operation manager starting
I0307 19:50:31.776821       1 main.go:188] RPC server starting on 0.0.0.0:8090
I0307 19:50:31.777010       1 main.go:149] HTTP server starting on 0.0.0.0:8091
I0307 19:50:31.777957       1 operation_manager.go:285] Acting as master for 0 / 0 active logs: master for:
I0307 19:50:33.287933       1 operation_manager.go:243] create master election goroutine for 4643029436978467864
I0307 19:50:33.938962       1 runner.go:130] 4643029436978467864: Now, I am the master
I0307 19:50:33.995308       1 operation_manager.go:285] Acting as master for 1 / 1 active logs: master for: <log-4643029436978467864>
...
I0307 19:51:05.615560       1 operation_manager.go:453] 4643029436978467864: processed 1 items in 0.02 seconds (60.26 qps)
I0307 19:51:40.781837       1 runner.go:148] 4643029436978467864: queue up resignation of mastership
I0307 19:51:40.821169       1 runner.go:172] 4643029436978467864: deliberately resigning mastership
I0307 19:51:40.821239       1 runner.go:130] 4643029436978467864: Now, I am the master
I0307 19:53:06.594238       1 runner.go:148] 4643029436978467864: queue up resignation of mastership
I0307 19:53:06.655440       1 runner.go:172] 4643029436978467864: deliberately resigning mastership
I0307 19:53:06.655525       1 runner.go:130] 4643029436978467864: Now, I am the master

Release Note

Better timeouts and readiness checks in docker-compose.yml.

Documentation

@ramonpetgrave64
improve mysql readiness checks
fa27085 
Signed-off-by: Ramon Petgrave <[email protected]>
@ramonpetgrave64 ramonpetgrave64 force-pushed the docker-compose-readiness-checks branch from 4f21656 to fa27085 Compare March 7, 2025 20:01
@ramonpetgrave64 ramonpetgrave64 mentioned this pull request Mar 7, 2025
Closed
@codecov
Copy link
Copy Markdown

codecov bot commented Mar 7, 2025 •
edited
Loading

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 25.25%. Comparing base (488eb97) to head (b97adbf).
Report is 345 commits behind head on main.

Additional details and impacted files 
@@             Coverage Diff             @@
##             main    #2397       +/-   ##
===========================================
- Coverage   66.46%   25.25%   -41.21%     
===========================================
  Files          92      192      +100     
  Lines        9258    24857    +15599     
===========================================
+ Hits         6153     6277      +124     
- Misses       2359    17803    +15444     
- Partials      746      777       +31
Flag Coverage Δ
e2etests 46.70% <ø> (-0.86%) ⬇️
unittests 16.48% <ø> (-31.21%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@ramonpetgrave64 ramonpetgrave64 marked this pull request as ready for review March 7, 2025 20:52
@ramonpetgrave64 ramonpetgrave64 requested a review from a team as a code owner March 7, 2025 20:52
Hayden-IO
Hayden-IO reviewed Mar 10, 2025
View reviewed changes
docker-compose.yml
retries: 3
start_period: 10s
retries: 5
start_period: 90s
Copy link
Copy Markdown
Contributor

@Hayden-IO Hayden-IO Mar 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you developing on macOS or Linux? While this is just for developers so I'm not really concerned with bumping this, I want to make sure there's not a bug with the container.

Personally, I noticed that when I switched from developing on Linux to macOS, I saw more startup errors. Saw some mentions online of needing a different MySQL container but didn't dig into this more.

Copy link
Copy Markdown
Contributor Author

@ramonpetgrave64 ramonpetgrave64 Mar 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, the e2e-test.sh works on linux, but I see some platform-related errors when running it on mac, I suspect I need to remove the platform: specification and somehow find another multiplatform image for the gcp-pubsub-emulator

Copy link
Copy Markdown
Contributor Author

@ramonpetgrave64 ramonpetgrave64 Mar 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bobcallaway thanks, #2406 helps out, though I'm still having some perhaps unrelated port-forwarding issues on mac with rancher-desktop.

Copy link
Copy Markdown
Member

@bobcallaway bobcallaway Mar 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you share your specific commands that you're running which aren't working?

Copy link
Copy Markdown
Contributor Author

@ramonpetgrave64 ramonpetgrave64 Mar 11, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was running both e2e-test.sh and ./tests/e2e-test.sh. It turns out the issue was with my Mac. Somehow another program ssh was using the same port 3000 as rekor-server. I suspect it was because of vscode's remote development feature. But rancher didn't error or warn me about that conflict. It's working fine now.

@bobcallaway bobcallaway merged commit 26bfbf5 into sigstore:main Mar 11, 2025
16 checks passed
@ramonpetgrave64 ramonpetgrave64 mentioned this pull request Apr 1, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Hayden-IO Hayden-IO Hayden-IO left review comments

@bobcallaway bobcallaway bobcallaway approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@ramonpetgrave64 @bobcallaway @Hayden-IO