core: Refactor, improve listener logic #5089
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Deprecate:
Prefer caddy.NetworkAddress.Listen() instead.
Change:
caddy.ListenQUIC (hopefully to remove later)
caddy.ListenerFunc signature (add context and ListenConfig)
Don't emit Alt-Svc header advertising h3 over HTTP/3
Use quic.ListenEarly instead of quic.ListenEarlyAddr; this gives us more flexibility (e.g. possibility of HTTP/3 over UDS) but also introduces a new issue:
Listening on Unix socket? quic-go/quic-go#3560 (comment)
Unlink unix socket before and after use
I'm not sure that
ListenAll()
will be useful. You'd think it would be, except the HTTP app, at least, sets up each listener one-by-one. That could probably be refactored and simplified withListenAll()
, but I haven't gotten there yet. EDIT: I haven't refactored the http app to use ListenAll, but I did find it to be quite convenient in the layer4 module so I think I'll at least keep it (experimentally) for now.(I'm keeping it for now because I wrote it and then realized that wouldn't be suitable for this refactor. I kept it because I already wrote it. I might delete it before merging this PR though.)
This took several days. I've done fairly extensive testing on this, but also went through many phases of "oops, I broke it" eventually followed by "duh, that was stupid" or "argh, that's annoying" and found fixes or workarounds for all the problems I noticed.
I have NO idea if this works on Windows or Mac yet. I can only run tests on Linux here, so we'll see what the CI says. I did my best to at least ensure it builds on Windows.
This change was mostly motivated by HTTP/3 and Unix sockets and the interop there. HTTP/3 has to be disabled if a site binds to unix sockets and other HTTP versions are also enabled, because a unix socket can't be both STREAM and DGRAM. But if you enable only HTTP/3 and bind to a unix socket, it should work. However, I couldn't test this because no client I know of (Firefox,
curl --http3
, etc) supports HTTP/3 over unix sockets. And frankly, I don't blame them, because I can't think of why that'd ever be useful or necessary. Nonetheless, the code in this PR should enable that use case, and the program runs with a config like that, but I haven't verified with a client.Overall the refactor is mostly a welcome change, although it does deprecate some
Listen*
functions in thecaddy
package. I doubt these are used by any plugins other than mylayer4
app, so I'm not too worried about deprecating and eventually removing them.Some things are frustrating about this though. Socket reuse is now dependent on OS and network type; so like, TCP sockets are reused differently on Windows and Linux, and different still for Unix sockets regardless of OS.
QUIC is a slight pain point still because although the listener is basically just a net.PacketConn, API constraints and an implementation choice in quic-go make the Close behavior of listeners a little tricky/surprising. Actually, quic-go's implementation kind of makes sense in general, and would seem like a good idea, but in our case it does add a bit of bulk to the code.
Ideally, we'd get rid of ListenQUIC and just have our
serveHTTP()
method wrap the returned listener in aquic.EarlyListener
. And maybe there's a way to do that already, I just haven't discovered it yet. We might need to wrap the EarlyListener with our own concrete type, but it'd still likely be cleaner than what we have now.This change also doesn't emit the Alt-Svc header on HTTP/3. It doesn't make sense to advertise h3 when the client is already using it. I just hope clients won't stop using it if the header disappears. I have noticed that Firefox likes to stop using HTTP/3 intermittently or randomly sometimes...
Anyway, there's a lot to unpack here. Overall I think this implementation of listeners is more reliable, flexible, and correct than what we had before, even if a little more complicated. I think some of the rough edges can be smoothed out with some cooperation and clever tinkering.
Please test this out. It'll probably get merged either way but the more field use it gets, the better! I'll deploy it to the Caddy website, for example, before tagging a release.