Hi 👋!
We are experiencing an unexpected behavior of mod_md where a certificate is not being renewed while it has now less than 20 days of validity out of its 90 days lifespan. No error is reported. We already encountered this issue a few times (twice?) during the last years, and saw that removing md/staging/<domain>/job.json unlocked the situation for us.
For now, I have not removed the file, so the system is still "blocked" with the old certificate that is not being renewed. I open this issue so that I can provide additional feedback, perform tests or verification, and anything than can help to find the root cause of the issue and implement a fix.
Setup overview
Operating System: Debian 13 (update from Debian 12 where we also had the issue)
Apache: 2.4.66-1~deb13u2 (installed from Debian repository)
mod_md: v2.6.6 (version bundled in the apache2-bin package)
number of vhosts with MDomain: 1 (so probably not a race condition between domains as suggested in an older issue)
How we discovered that removing md/staging/<domain>/job.json helped
Previous times our monitoring system told us that one of our certificates was expiring soon, we inspected apache logs (found no errors), then the md/ directory and saw that we had a file md/staging/<domain>/job.json that exist, while on other nodes where mod_md is used in the same way, the md/staging directory was empty. We emptied it and restarted apache, and mod_md immediately requested a new certificate, got one, and called our user script that restart the server, making our monitoring happy again.
Content of the /etc/apache2/md directory:
Here is what we currently have:
# find /etc/apache2/md -type f | xargs ls -lrt
-rw------- 1 root root 116 Jan 8 2025 /etc/apache2/md/md_store.json
-rw------- 1 root root 3272 Jan 8 2025 /etc/apache2/md/archive/yoda.biodevas.fr.1/fallback-privkey.pem
-rw------- 1 root root 1862 Jan 8 2025 /etc/apache2/md/archive/yoda.biodevas.fr.1/fallback-pubcert.pem
-rw------- 1 root root 560 Jan 8 2025 /etc/apache2/md/archive/yoda.biodevas.fr.1/md.json
-rw------- 1 root root 3944 Jan 8 2025 /etc/apache2/md/archive/yoda.biodevas.fr.2/pubcert.pem
-rw------- 1 root root 3272 Jan 8 2025 /etc/apache2/md/archive/yoda.biodevas.fr.2/privkey.pem
-rw------- 1 root root 5019 Jan 8 2025 /etc/apache2/md/archive/yoda.biodevas.fr.2/job.json
-rw-r--r-- 1 root root 2654 Jan 8 2025 /etc/apache2/md/accounts/ACME-.letsencrypt.org-0000/account.pem
-rw-r--r-- 1 root root 211 Jan 8 2025 /etc/apache2/md/accounts/ACME-.letsencrypt.org-0000/account.json
-rw------- 1 root root 708 Apr 1 2025 /etc/apache2/md/archive/yoda.biodevas.fr.2/md.json
-rw------- 1 root root 3988 Apr 1 2025 /etc/apache2/md/archive/yoda.biodevas.fr.3/pubcert.pem
-rw------- 1 root root 3272 Apr 1 2025 /etc/apache2/md/archive/yoda.biodevas.fr.3/privkey.pem
-rw------- 1 root root 4858 Apr 1 2025 /etc/apache2/md/archive/yoda.biodevas.fr.3/job.json
-rw------- 1 root root 619 Apr 1 2025 /etc/apache2/md/archive/yoda.biodevas.fr.3/md.json
-rw------- 1 root root 3935 Jun 1 2025 /etc/apache2/md/archive/yoda.biodevas.fr.4/pubcert.pem
-rw------- 1 root root 3272 Jun 1 2025 /etc/apache2/md/archive/yoda.biodevas.fr.4/privkey.pem
-rw------- 1 root root 4858 Jun 1 2025 /etc/apache2/md/archive/yoda.biodevas.fr.4/job.json
-rw------- 1 root root 620 Aug 31 2025 /etc/apache2/md/archive/yoda.biodevas.fr.4/md.json
-rw------- 1 root root 3935 Aug 31 2025 /etc/apache2/md/archive/yoda.biodevas.fr.5/pubcert.pem
-rw------- 1 root root 3272 Aug 31 2025 /etc/apache2/md/archive/yoda.biodevas.fr.5/privkey.pem
-rw------- 1 root root 620 Aug 31 2025 /etc/apache2/md/archive/yoda.biodevas.fr.5/md.json
-rw------- 1 root root 4858 Aug 31 2025 /etc/apache2/md/archive/yoda.biodevas.fr.5/job.json
-rw------- 1 root root 3940 Aug 31 2025 /etc/apache2/md/archive/yoda.biodevas.fr.6/pubcert.pem
-rw------- 1 root root 3272 Aug 31 2025 /etc/apache2/md/archive/yoda.biodevas.fr.6/privkey.pem
-rw------- 1 root root 620 Aug 31 2025 /etc/apache2/md/archive/yoda.biodevas.fr.6/md.json
-rw------- 1 root root 4310 Aug 31 2025 /etc/apache2/md/archive/yoda.biodevas.fr.6/job.json
-rw------- 1 root root 3935 Aug 31 2025 /etc/apache2/md/archive/yoda.biodevas.fr.7/pubcert.pem
-rw------- 1 root root 3272 Aug 31 2025 /etc/apache2/md/archive/yoda.biodevas.fr.7/privkey.pem
-rw------- 1 root root 620 Aug 31 2025 /etc/apache2/md/archive/yoda.biodevas.fr.7/md.json
-rw------- 1 root root 4310 Aug 31 2025 /etc/apache2/md/archive/yoda.biodevas.fr.7/job.json
-rw------- 1 root root 3940 Aug 31 2025 /etc/apache2/md/archive/yoda.biodevas.fr.8/pubcert.pem
-rw------- 1 root root 3272 Aug 31 2025 /etc/apache2/md/archive/yoda.biodevas.fr.8/privkey.pem
-rw------- 1 root root 4310 Aug 31 2025 /etc/apache2/md/archive/yoda.biodevas.fr.8/job.json
-rw------- 1 root root 619 Aug 31 2025 /etc/apache2/md/archive/yoda.biodevas.fr.8/md.json
-rw------- 1 root root 3935 Nov 30 21:49 /etc/apache2/md/archive/yoda.biodevas.fr.9/pubcert.pem
-rw------- 1 root root 3272 Nov 30 21:49 /etc/apache2/md/archive/yoda.biodevas.fr.9/privkey.pem
-rw------- 1 root root 4858 Nov 30 21:49 /etc/apache2/md/archive/yoda.biodevas.fr.9/job.json
-rw------- 1 root root 673 Jan 10 19:33 /etc/apache2/md/archive/yoda.biodevas.fr.9/md.json
-rw------- 1 root root 3948 Jan 28 23:25 /etc/apache2/md/domains/yoda.biodevas.fr/pubcert.pem
-rw------- 1 root root 3268 Jan 28 23:25 /etc/apache2/md/domains/yoda.biodevas.fr/privkey.pem
-rw------- 1 root root 673 Jan 28 23:25 /etc/apache2/md/domains/yoda.biodevas.fr/md.json
-rw------- 1 root root 4933 Jan 28 23:25 /etc/apache2/md/tmp/yoda.biodevas.fr/job.json
-rw-r--r-- 1 www-data www-data 4866 Jan 28 23:25 /etc/apache2/md/staging/yoda.biodevas.fr/job.json
-rw------- 1 root root 4960 Jan 28 23:25 /etc/apache2/md/domains/yoda.biodevas.fr/job.json
It looks like the md/staging/<domain>/job.json was created during the latest successful renewal but was not cleared after (only today I see it is an "old" file and not something related to a "current" renewal).
As all these files are old, I do not include their content in this issue, but if you think some would be interesting to check, I will be happy to share their content on request.
Apache Logs
We do not see any sort of errors in the log. As suggested in #334 we enabled debug logging for mod_md.
Here is what we got after setting LogLevel warn md:debug and starting apache (full content of /var/log/apache2/error.log):
[Thu Apr 09 18:40:14.347395 2026] [md:debug] [pid 2143625:tid 2143625] mod_md.c(618): AH10041: Server yoda.biodevas.fr:0 matches md yoda.biodevas.fr (config yoda.biodevas.fr[default, default], match-mode=0) for domain yoda.biodevas.fr, has now 1 MDs
[Thu Apr 09 18:40:14.347434 2026] [md:debug] [pid 2143625:tid 2143625] md_reg.c(862): sync MDs, start
[Thu Apr 09 18:40:14.347453 2026] [md:debug] [pid 2143625:tid 2143625] md_reg.c(937): sync MDs, 1 existing, 0 moved, 0 new.
[Thu Apr 09 18:40:14.347461 2026] [md:debug] [pid 2143625:tid 2143625] md_reg.c(1278): yoda.biodevas.fr: nothing staged
[Thu Apr 09 18:40:14.369040 2026] [md:info] [pid 2143626:tid 2143626] AH10071: mod_md (v2.6.6), initializing...
[Thu Apr 09 18:40:14.369098 2026] [md:debug] [pid 2143626:tid 2143626] mod_md.c(461): AH10037: server seems reachable via http: and reachable via https:
[Thu Apr 09 18:40:14.369102 2026] [md:debug] [pid 2143626:tid 2143626] mod_md.c(729): AH10039: Completed MD[yoda.biodevas.fr, CA=(null), Proto=ACME, Agreement=accepted, renew-mode=1 renew_window=33%, warn_window=10%
[Thu Apr 09 18:40:14.369105 2026] [md:debug] [pid 2143626:tid 2143626] mod_md.c(618): AH10041: Server yoda.biodevas.fr:0 matches md yoda.biodevas.fr (config yoda.biodevas.fr[default, default], match-mode=0) for domain yoda.biodevas.fr, has now 1 MDs
[Thu Apr 09 18:40:14.369123 2026] [md:debug] [pid 2143626:tid 2143626] md_reg.c(862): sync MDs, start
[Thu Apr 09 18:40:14.369144 2026] [md:debug] [pid 2143626:tid 2143626] md_reg.c(937): sync MDs, 1 existing, 0 moved, 0 new.
[Thu Apr 09 18:40:14.369149 2026] [md:debug] [pid 2143626:tid 2143626] md_reg.c(1278): yoda.biodevas.fr: nothing staged
[Thu Apr 09 18:40:14.370452 2026] [md:debug] [pid 2143626:tid 2143626] mod_md.c(565): AH10169: yoda.biodevas.fr: https server_rec for yoda.biodevas.fr does not have protocol acme-tls/1 enabled
[Thu Apr 09 18:40:14.370556 2026] [md:debug] [pid 2143626:tid 2143626] md_reg.c(272): md{yoda.biodevas.fr}: certificate(0) is ok
[Thu Apr 09 18:40:14.370561 2026] [md:debug] [pid 2143626:tid 2143626] md_reg.c(974): loading md yoda.biodevas.fr
[Thu Apr 09 18:40:14.370581 2026] [md:debug] [pid 2143626:tid 2143626] md_reg.c(976): loaded md yoda.biodevas.fr
[Thu Apr 09 18:40:14.370584 2026] [md:debug] [pid 2143626:tid 2143626] md_reg.c(1022): sync MDs, finish done
[Thu Apr 09 18:40:14.370587 2026] [md:debug] [pid 2143626:tid 2143626] md_reg.c(1147): yoda.biodevas.fr: init done
[Thu Apr 09 18:40:14.370605 2026] [md:debug] [pid 2143626:tid 2143626] mod_md.c(1017): AH10074: 1 out of 1 mds need watching
[Thu Apr 09 18:40:14.370690 2026] [md:debug] [pid 2143626:tid 2143626] mod_md_drive.c(407): AH10067: register md renew watchdog(_md_renew_)
[Thu Apr 09 18:40:14.371416 2026] [mpm_event:notice] [pid 2143626:tid 2143626] AH00489: Apache/2.4.66 (Debian) OpenSSL/3.5.5 configured -- resuming normal operations
[Thu Apr 09 18:40:14.371425 2026] [core:notice] [pid 2143626:tid 2143626] AH00094: Command line: '/usr/sbin/apache2'
[Thu Apr 09 18:40:14.371429 2026] [md:debug] [pid 2143628:tid 2143629] mod_md_drive.c(267): AH10054: md watchdog start, auto drive 1 mds
[Thu Apr 09 18:40:14.471667 2026] [md:debug] [pid 2143628:tid 2143629] mod_md_drive.c(272): AH10055: md watchdog run, auto drive 1 mds
[Thu Apr 09 18:40:14.472484 2026] [md:debug] [pid 2143628:tid 2143629] mod_md_drive.c(294): AH10107: next run in 8 hours 29 minutes 3 seconds
[Fri Apr 10 03:10:23.339721 2026] [md:debug] [pid 2143628:tid 2143629] mod_md_drive.c(272): AH10055: md watchdog run, auto drive 1 mds
[Fri Apr 10 03:10:23.340241 2026] [md:debug] [pid 2143628:tid 2143629] mod_md_drive.c(294): AH10107: next run in 10 hours 29 minutes 59 seconds
[Fri Apr 10 13:41:42.943678 2026] [md:debug] [pid 2143628:tid 2143629] mod_md_drive.c(272): AH10055: md watchdog run, auto drive 1 mds
[Fri Apr 10 13:41:42.944211 2026] [md:debug] [pid 2143628:tid 2143629] mod_md_drive.c(294): AH10107: next run in 9 hours 25 minutes 18 seconds
Can we provide anything else that could help?
We can try things during ~2 weeks, then we will have to use our workaround to ensure the service does not break.
Thank you!
Cc @neomilium (coworker who will be able to follow-up if I am unavailable)
Hi 👋!
We are experiencing an unexpected behavior of
mod_mdwhere a certificate is not being renewed while it has now less than 20 days of validity out of its 90 days lifespan. No error is reported. We already encountered this issue a few times (twice?) during the last years, and saw that removingmd/staging/<domain>/job.jsonunlocked the situation for us.For now, I have not removed the file, so the system is still "blocked" with the old certificate that is not being renewed. I open this issue so that I can provide additional feedback, perform tests or verification, and anything than can help to find the root cause of the issue and implement a fix.
Setup overview
Operating System: Debian 13 (update from Debian 12 where we also had the issue)
Apache: 2.4.66-1~deb13u2 (installed from Debian repository)
mod_md: v2.6.6 (version bundled in the
apache2-binpackage)number of vhosts with MDomain: 1 (so probably not a race condition between domains as suggested in an older issue)
How we discovered that removing
md/staging/<domain>/job.jsonhelpedPrevious times our monitoring system told us that one of our certificates was expiring soon, we inspected apache logs (found no errors), then the
md/directory and saw that we had a filemd/staging/<domain>/job.jsonthat exist, while on other nodes wheremod_mdis used in the same way, themd/stagingdirectory was empty. We emptied it and restarted apache, andmod_mdimmediately requested a new certificate, got one, and called our user script that restart the server, making our monitoring happy again.Content of the
/etc/apache2/mddirectory:Here is what we currently have:
It looks like the
md/staging/<domain>/job.jsonwas created during the latest successful renewal but was not cleared after (only today I see it is an "old" file and not something related to a "current" renewal).As all these files are old, I do not include their content in this issue, but if you think some would be interesting to check, I will be happy to share their content on request.
Apache Logs
We do not see any sort of errors in the log. As suggested in #334 we enabled debug logging for
mod_md.Here is what we got after setting
LogLevel warn md:debugand starting apache (full content of/var/log/apache2/error.log):Can we provide anything else that could help?
We can try things during ~2 weeks, then we will have to use our workaround to ensure the service does not break.
Thank you!
Cc @neomilium (coworker who will be able to follow-up if I am unavailable)