Permission denied when first run after fresh installation.

[jerry-wolf]

OS: Ubuntu 22.04
Install: https://caddyserver.com/docs/install#debian-ubuntu-raspbian

# systemctl --version
systemd 249 (249.11-0ubuntu3.12)
+PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY -P11KIT -QRENCODE +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified

# apt-show-versions caddy
caddy:amd64/any-version 2.8.0 uptodate

Caddyfile

{
        email my-username@my-domain.com
}

my-domain.com {
        reverse_proxy 172.17.0.2:3000
}

5月 30 05:38:26 my-hostname systemd[1]: Starting Caddy...
5月 30 05:38:26 my-hostname caddy[411531]: caddy.HomeDir=/var/lib/caddy
5月 30 05:38:26 my-hostname caddy[411531]: caddy.AppDataDir=/var/lib/caddy/.local/share/caddy
5月 30 05:38:26 my-hostname caddy[411531]: caddy.AppConfigDir=/var/lib/caddy/.config/caddy
5月 30 05:38:26 my-hostname caddy[411531]: caddy.ConfigAutosavePath=/var/lib/caddy/.config/caddy/autosave.json
5月 30 05:38:26 my-hostname caddy[411531]: caddy.Version=v2.8.0 h1:7ZCvB9R7qBsEydqBkYCOHaMNrDEF/fj0ZouySV2D474=
5月 30 05:38:26 my-hostname caddy[411531]: runtime.GOOS=linux
5月 30 05:38:26 my-hostname caddy[411531]: runtime.GOARCH=amd64
5月 30 05:38:26 my-hostname caddy[411531]: runtime.Compiler=gc
5月 30 05:38:26 my-hostname caddy[411531]: runtime.NumCPU=2
5月 30 05:38:26 my-hostname caddy[411531]: runtime.GOMAXPROCS=2
5月 30 05:38:26 my-hostname caddy[411531]: runtime.Version=go1.22.3
5月 30 05:38:26 my-hostname caddy[411531]: os.Getwd=/
5月 30 05:38:26 my-hostname caddy[411531]: LANG=zh_CN.UTF-8
5月 30 05:38:26 my-hostname caddy[411531]: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
5月 30 05:38:26 my-hostname caddy[411531]: NOTIFY_SOCKET=/run/systemd/notify
5月 30 05:38:26 my-hostname caddy[411531]: HOME=/var/lib/caddy
5月 30 05:38:26 my-hostname caddy[411531]: LOGNAME=caddy
5月 30 05:38:26 my-hostname caddy[411531]: USER=caddy
5月 30 05:38:26 my-hostname caddy[411531]: INVOCATION_ID=59a85e7ead1a4b42a3f073b4984622bd
5月 30 05:38:26 my-hostname caddy[411531]: JOURNAL_STREAM=8:2420821
5月 30 05:38:26 my-hostname caddy[411531]: SYSTEMD_EXEC_PID=411531
5月 30 05:38:26 my-hostname caddy[411531]: {"level":"info","ts":1717018706.865005,"msg":"using config from file","file":"/etc/caddy/Caddyfile"}
5月 30 05:38:26 my-hostname caddy[411531]: {"level":"info","ts":1717018706.8670456,"msg":"adapted config to JSON","adapter":"caddyfile"}
5月 30 05:38:26 my-hostname caddy[411531]: {"level":"warn","ts":1717018706.8670666,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":12}
5月 30 05:38:26 my-hostname caddy[411531]: {"level":"info","ts":1717018706.8696983,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
5月 30 05:38:26 my-hostname caddy[411531]: {"level":"info","ts":1717018706.8699424,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
5月 30 05:38:26 my-hostname caddy[411531]: {"level":"info","ts":1717018706.8699946,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
5月 30 05:38:26 my-hostname caddy[411531]: {"level":"info","ts":1717018706.8704922,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00017b400"}
5月 30 05:38:26 my-hostname caddy[411531]: {"level":"info","ts":1717018706.8705041,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
5月 30 05:38:26 my-hostname caddy[411531]: {"level":"warn","ts":1717018706.8705664,"logger":"tls","msg":"unable to get instance ID; storage clean stamps will be incomplete","error":"open /var/lib/caddy/.local/share/caddy/instance.uuid: permission denied"}
5月 30 05:38:26 my-hostname caddy[411531]: {"level":"error","ts":1717018706.871128,"logger":"tls","msg":"could not clean default/global storage","error":"unable to acquire storage_clean lock: creating lock file: open /var/lib/caddy/.local/share/caddy/locks/storage_clean.lock: permission denied"}
5月 30 05:38:26 my-hostname caddy[411531]: {"level":"info","ts":1717018706.8711424,"logger":"tls","msg":"finished cleaning storage units"}
5月 30 05:38:26 my-hostname caddy[411531]: {"level":"info","ts":1717018706.871646,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
5月 30 05:38:26 my-hostname caddy[411531]: {"level":"info","ts":1717018706.8719206,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
5月 30 05:38:26 my-hostname caddy[411531]: {"level":"info","ts":1717018706.872082,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["my-domain.com"]}
5月 30 05:38:26 my-hostname caddy[411531]: {"level":"info","ts":1717018706.8726678,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc00017b400"}
5月 30 05:38:26 my-hostname caddy[411531]: Error: loading initial config: loading new config: http app module: start: finalizing automatic HTTPS: managing certificates for [my-domain.com]: automate: manage [my-domain.com]: my-domain.com: caching certificate: open /var/lib/caddy/.local/share/caddy/certificates/acme-v02.api.letsencrypt.org-directory/my-domain.com/my-domain.com.key: permission denied
5月 30 05:38:26 my-hostname systemd[1]: caddy.service: Main process exited, code=exited, status=1/FAILURE
5月 30 05:38:26 my-hostname systemd[1]: caddy.service: Failed with result 'exit-code'.
5月 30 05:38:26 my-hostname systemd[1]: Failed to start Caddy.

Something Found: /var/lib/caddy/.local/ lacks "x" permissions for owner.

[mholt]

Interesting... allegedly, the fix is in #6340 -- previously reported in #6341. But I wasn't able to reproduce it.

Additionally, I'm confused as to why the storage isn't being "checked" before obtaining a certificate in the first place: https://github.com/caddyserver/certmagic/blob/bd400cc9fb55cc863a80028a0ab2f67d0b07504c/config.go#L1154-L1158

This is something I thought we solved years ago. If the storage isn't writeable then it should give an error much earlier.

[ParichayDidwania]

Facing exact same issue here. Downgrading is the only option which works.

[DeArchX]

same here

[mholt]

This is strange. Why can I not reproduce this by setting $HOME to a folder that doesn't exist at first? What is the difference, and why are only some people experiencing this?

I'm mobile now so just jotting some thoughts down.

[qf3l3k]

Same issue here: Error: loading initial config: loading new config: http app module: start: finalizing automatic HTTPS: managing certificates for
After updating packages and server reboot unable to get cert for any service.
Removed caddy and reinstalled it. Same issue.

EDIT
After downgrading from 2.8.0 to 2.7.6 all seems to be fine.
Just small issue with cert renewals (Too many requests) as previous got wiped with caddy uninstall. Fortunately it is test server.

Issue happened on Ubuntu 22.04.4 LTS, bare metal server with x86 CPU

[mohammed90]

The workaround for now is to run chmod -R 700 /var/lib/caddy/.local/share/caddy. You may need to sudo that command.

[kingRayhan]

I am getting similar issue @qf3l3k

[mohammed90]

Fellas, we're aware of the issue itself. Go easy on the "Me too!". If you'd like to help, you can provide contextual information (is it an upgrade? completely fresh install? OS details? etc.).

[kingRayhan]

Fellas, we're aware of the issue itself. Go easy on the "Me too!". If you'd like to help, you can provide contextual information (is it an upgrade? completely fresh install? OS details? etc.).

I just installed caddy on a newly booted ubuntu-22.04 from aws-ec2
caddy version: v2.8.0 h1:7ZCvB9R7qBsEydqBkYCOHaMNrDEF/fj0ZouySV2D474=

[mholt]

Should be fixed in #6340

[DeArchX]

Fellas, we're aware of the issue itself. Go easy on the "Me too!". If you'd like to help, you can provide contextual information (is it an upgrade? completely fresh install? OS details? etc.).

sorry about that