caddyhttp: Refactor cert Managers (fix #5415) by mholt · Pull Request #5533 · caddyserver/caddy

mholt · 2023-05-11T18:46:43Z

Implicit Managers (Tailscale currently) are now configured in their own automation policy that enables on-demand TLS. Explicit managers now enable on-demand TLS, which defers cert issuance (which won't be needed or used if the cert manager returns the cert in the first place).

@BioEvo Do you think you could test this out to verify? I tried your config and it worked for me. I also tried a bunch of other configs and AFAIK they worked alright. But never hurts to have you verify! 😉 This was a complicated change for me (to keep as simple as possible).

francislavoie

All makes sense to me!

BioEvo · 2023-11-23T00:22:15Z

I'm so sorry that I somehow missed this issue, apparently it is too late to reply to this issue since this patch had been verified. And of course it works well when I update caddy to v2.7.5 on my server.

BioEvo · 2023-11-23T01:36:28Z

With further tests, It works great on single SNI, but on wildcard sites, It goes to:
Error: loading initial config: loading new config: loading http app module: provision http: getting tls app: loading tls app module: provision tls: provisioning automation policy 0: on-demand TLS cannot be enabled without an 'ask' endpoint to prevent abuse; please refer to documentation for details

mholt · 2023-11-23T01:37:42Z

Can you please share your config?

BioEvo · 2023-11-23T02:08:19Z

Can you please share your config?

Sure, here is the demo:

I put a self-signed certificate in 192.168.1.100:/etc/caddy/cert, and set my hosts to :

192.168.1.100 abc.mydemo.com

single SNI works well with caddy:

http://127.0.0.127:8080 {
	file_server {
		root /etc/caddy/cert
	}
}

https://abc.mydemo.com {
	tls {
		get_certificate http http://127.0.0.127:8080/demo.crt
	}
}

caddy[1508388]: caddy.HomeDir=/var/lib/caddy
caddy[1508388]: caddy.AppDataDir=/var/lib/caddy/.local/share/caddy
caddy[1508388]: caddy.AppConfigDir=/var/lib/caddy/.config/caddy
caddy[1508388]: caddy.ConfigAutosavePath=/var/lib/caddy/.config/caddy/autosave.json
caddy[1508388]: caddy.Version=v2.7.5 h1:HoysvZkLcN2xJExEepaFHK92Qgs7xAiCFydN5x5Hs6Q=
caddy[1508388]: runtime.GOOS=linux
caddy[1508388]: runtime.GOARCH=amd64
caddy[1508388]: runtime.Compiler=gc
caddy[1508388]: runtime.NumCPU=8
caddy[1508388]: runtime.GOMAXPROCS=8
caddy[1508388]: runtime.Version=go1.20.6
caddy[1508388]: os.Getwd=/
caddy[1508388]: LANG=C.UTF-8
caddy[1508388]: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
caddy[1508388]: NOTIFY_SOCKET=/run/systemd/notify
caddy[1508388]: HOME=/var/lib/caddy
caddy[1508388]: LOGNAME=caddy
caddy[1508388]: USER=caddy
caddy[1508388]: INVOCATION_ID=4ce9088720484ae1b323545a7125bcb0
caddy[1508388]: JOURNAL_STREAM=8:53886522
caddy[1508388]: SYSTEMD_EXEC_PID=1508388
caddy[1508388]: {"level":"info","ts":1700704914.7279859,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
caddy[1508388]: {"level":"info","ts":1700704914.730437,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
caddy[1508388]: {"level":"info","ts":1700704914.7306917,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
caddy[1508388]: {"level":"info","ts":1700704914.7307115,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
caddy[1508388]: {"level":"info","ts":1700704914.7307274,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc000810380"}
caddy[1508388]: {"level":"info","ts":1700704914.731253,"logger":"http.log","msg":"server running","name":"srv1","protocols":["h1","h2","h3"]}
caddy[1508388]: {"level":"info","ts":1700704914.7313204,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
caddy[1508388]: {"level":"info","ts":1700704914.7313592,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
caddy[1508388]: {"level":"info","ts":1700704914.7314951,"msg":"failed to sufficiently increase send buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details."}
caddy[1508388]: {"level":"info","ts":1700704914.7315896,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
caddy[1508388]: {"level":"info","ts":1700704914.7315967,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["abc.mydemo.com"]}
caddy[1508388]: {"level":"info","ts":1700704914.7318127,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/var/lib/caddy/.local/share/caddy"}
caddy[1508388]: {"level":"info","ts":1700704914.731961,"msg":"autosaved config (load with --resume flag)","file":"/var/lib/caddy/.config/caddy/autosave.json"}
caddy[1508388]: {"level":"info","ts":1700704914.7320542,"msg":"serving initial configuration"}
systemd[1]: Started Caddy.

wildcard domain name goes to on-demand TLS:

http://127.0.0.127:8080 {
	file_server {
		root /etc/caddy/cert
	}
}

https://*.mydemo.com {
	tls {
		get_certificate http http://127.0.0.127:8080/demo.crt
	}
}

caddy[1511300]: caddy.HomeDir=/var/lib/caddy
caddy[1511300]: caddy.AppDataDir=/var/lib/caddy/.local/share/caddy
caddy[1511300]: caddy.AppConfigDir=/var/lib/caddy/.config/caddy
caddy[1511300]: caddy.ConfigAutosavePath=/var/lib/caddy/.config/caddy/autosave.json
caddy[1511300]: caddy.Version=v2.7.5 h1:HoysvZkLcN2xJExEepaFHK92Qgs7xAiCFydN5x5Hs6Q=
caddy[1511300]: runtime.GOOS=linux
caddy[1511300]: runtime.GOARCH=amd64
caddy[1511300]: runtime.Compiler=gc
caddy[1511300]: runtime.NumCPU=8
caddy[1511300]: runtime.GOMAXPROCS=8
caddy[1511300]: runtime.Version=go1.20.6
caddy[1511300]: os.Getwd=/
caddy[1511300]: LANG=C.UTF-8
caddy[1511300]: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
caddy[1511300]: NOTIFY_SOCKET=/run/systemd/notify
caddy[1511300]: HOME=/var/lib/caddy
caddy[1511300]: LOGNAME=caddy
caddy[1511300]: USER=caddy
caddy[1511300]: INVOCATION_ID=b26aac921be54fd2ac9b1ab2a622d6d7
caddy[1511300]: JOURNAL_STREAM=8:53897707
caddy[1511300]: SYSTEMD_EXEC_PID=1511300
caddy[1511300]: {"level":"info","ts":1700705032.699861,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
caddy[1511300]: {"level":"info","ts":1700705032.7041433,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
caddy[1511300]: {"level":"info","ts":1700705032.7045429,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc000409400"}
caddy[1511300]: {"level":"info","ts":1700705032.7045615,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc000409400"}
caddy[1511300]: Error: loading initial config: loading new config: loading http app module: provision http: getting tls app: loading tls app module: provision tls: provisioning automation policy 0: on-demand TLS cannot be enabled without an 'ask' endpoint to prevent abuse; please refer to documentation for details
systemd[1]: caddy.service: Main process exited, code=exited, status=1/FAILURE

BioEvo · 2023-11-23T03:34:16Z

more tests:

/etc/caddy/cert/wildcard.crt : certificate for *.mydemo.com
/etc/caddy/cert/single.crt : certificate for single.mydemo.com

get_certificate with wildcard goes to on-demand TLS:

http://127.0.0.127:8080 {
	file_server {
		root /etc/caddy/cert
	}
}

https://*.mydemo.com {
	tls {
		get_certificate http http://127.0.0.127:8080/wildcard.crt
	}
}

certificate file with wildcard works well:

http://127.0.0.127:8080 {
	file_server {
		root /etc/caddy/cert
	}
}

https://*.mydemo.com {
	tls /etc/caddy/cert/wildcard.crt /etc/caddy/cert/wildcard.crt
}

get_certificate with single SNI goes works well:

http://127.0.0.127:8080 {
	file_server {
		root /etc/caddy/cert
	}
}

https://single.mydemo.com {
	tls {
		get_certificate http http://127.0.0.127:8080/single.crt
	}
}

certificate file with single SNI goes works well:

http://127.0.0.127:8080 {
	file_server {
		root /etc/caddy/cert
	}
}

https://single.mydemo.com {
	tls /etc/caddy/cert/single.crt /etc/caddy/cert/single.crt
}

certificate file for single SNI and wildcard: works well

http://127.0.0.127:8080 {
	file_server {
		root /etc/caddy/cert
	}
}

https://*.mydemo.com {
	tls /etc/caddy/cert/wildcard.crt /etc/caddy/cert/wildcard.crt
}

https://single.mydemo.com {
	tls /etc/caddy/cert/single.crt /etc/caddy/cert/single.crt
}

get_certificate for single SNI and certificate file for wildcard: goes to on-demand TLS

http://127.0.0.127:8080 {
	file_server {
		root /etc/caddy/cert
	}
}

https://*.mydemo.com {
	tls /etc/caddy/cert/wildcard.crt /etc/caddy/cert/wildcard.crt
}

https://single.mydemo.com {
	tls {
		get_certificate http http://127.0.0.127:8080/single.crt
	}
}

get_certificate for wildcard and certificate file for single SNI: stays alive, and try to get certificate by acme for *.mydemo.com

http://127.0.0.127:8080 {
	file_server {
		root /etc/caddy/cert
	}
}

https://*.mydemo.com {
	tls {
		get_certificate http http://127.0.0.127:8080/wildcard.crt
	}
}

https://single.mydemo.com {
	tls /etc/caddy/cert/single.crt /etc/caddy/cert/single.crt
}

get_certificate for single SNI and certificate file for wildcard, but single site comes first: goes to on-demand TLS

http://127.0.0.127:8080 {
	file_server {
		root /etc/caddy/cert
	}
}

https://single.mydemo.com {
	tls {
		get_certificate http http://127.0.0.127:8080/single.crt
	}
}

https://*.mydemo.com {
	tls /etc/caddy/cert/wildcard.crt /etc/caddy/cert/wildcard.crt
}

mholt · 2023-11-27T22:15:14Z

Thanks; will look at this soon -- will be a bit busy for a while though as our baby is coming home, so... yeah. Bear with me

mholt · 2024-04-15T13:34:50Z

@BioEvo I finally had a chance to look at this.

I am seeing that Caddy will serve the wildcard certificate if it already exists in memory (the "cache") even if the config says that there's a specific cert to use for a specific subdomain. But it serves the wildcard because technically it satisfies the request for the specific subdomain too. Is that the behavior you're saying is undesirable?

caddyhttp: Refactor cert Managers (fix #5415)

d8f61ca

mholt added this to the v2.7.0 milestone May 11, 2023

mholt mentioned this pull request May 11, 2023

Enabling get_certificate should disable cert automation #5415

Closed

Merge branch 'master' into certmanagers

58ef430

francislavoie reviewed May 11, 2023

View reviewed changes

Comment thread modules/caddyhttp/autohttps.go

francislavoie reviewed May 11, 2023

View reviewed changes

Comment thread modules/caddytls/automation.go

francislavoie approved these changes May 11, 2023

View reviewed changes

mholt linked an issue May 13, 2023 that may be closed by this pull request

Enabling get_certificate should disable cert automation #5415

Closed

mholt merged commit 96919ac into master May 15, 2023

mholt deleted the certmanagers branch May 15, 2023 16:47

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

caddyhttp: Refactor cert Managers (fix #5415)#5533

caddyhttp: Refactor cert Managers (fix #5415)#5533
mholt merged 2 commits intomasterfrom
certmanagers

mholt commented May 11, 2023

Uh oh!

Uh oh!

Uh oh!

francislavoie left a comment

Uh oh!

BioEvo commented Nov 23, 2023

Uh oh!

BioEvo commented Nov 23, 2023

Uh oh!

mholt commented Nov 23, 2023

Uh oh!

BioEvo commented Nov 23, 2023

Uh oh!

BioEvo commented Nov 23, 2023 •

edited

Loading

Uh oh!

mholt commented Nov 27, 2023

Uh oh!

mholt commented Apr 15, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

mholt commented May 11, 2023

Uh oh!

Uh oh!

Uh oh!

francislavoie left a comment

Choose a reason for hiding this comment

Uh oh!

BioEvo commented Nov 23, 2023

Uh oh!

BioEvo commented Nov 23, 2023

Uh oh!

mholt commented Nov 23, 2023

Uh oh!

BioEvo commented Nov 23, 2023

Uh oh!

BioEvo commented Nov 23, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mholt commented Nov 27, 2023

Uh oh!

mholt commented Apr 15, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

BioEvo commented Nov 23, 2023 •

edited

Loading