implement listener wrapper for use with http app

[WeidiDeng]

No description provided.

[mholt]

This is cool, thank you @WeidiDeng !

Does it work? 😃 What do you use it for / what does your config look like?

I'll take a closer look soon :)

[WeidiDeng]

This is cool, thank you @WeidiDeng !

Does it work? 😃 What do you use it for / what does your config look like?

I'll take a closer look soon :)

First adapt this caddyfile

example.com {
        encode gzip
        tls path.crt path.key
        reverse_proxy https://example.com {
        header_up Host {upstream_hostport}
    }
}

Then add this to appropriate path

 "listener_wrappers":
                    [
                        {
                            "wrapper": "layer4",
                            "routes":
                            [
                                {
                                    "match":
                                    [
                                        {
                                            "http":
                                            []
                                        }
                                    ]
                                },
                                {
                                    "match":
                                    [
                                        {
                                            "tls":
                                            {}
                                        }
                                    ],
                                    "handle":
                                    [
                                        {
                                            "handler": "tls"
                                        }
                                    ]
                                }
                            ]
                        }
                    ],
                    "automatic_https":
                    {
                        "disable": true
                    }

This allows caddy to handle https and plain http at once with the same port (no http2 though, only http1.1). listener piping is implicit if handlers are not terminal.

I only tested in dev server though, because a specialized piping is used in my caddy. I think there are some tweaks that can be made with caddy http version, like h2c is not compatible with h2 (one is over plaintext, another is over tls). Also if tls alpn settings only allow h2 or http1.1, golang http server will only server h2 or h1 (because http server checks tls.Conn
negotiated protocol explicitly).

[mholt]

Thank you for this. Let's give it a try! :) I may ask for your help if issues arise later.

[coolaj86]

Then add this to appropriate path

@WeidiDeng Which is the appropriate path?

Could you give a full config example that shows the context of where this belongs?

[coolaj86]

Problem: How to let caddy handle h2?

Grepping around the code and issues I finally realized that "listener_wrappers" is not just a comment like "srv0", but is actually a directive for "http" (NOT "layer4").

So I got things working with http/1.1, just like they were with the layer4 reverse proxy, but I was hoping that h2 would pass through to the caddy handle and get reverse proxied down to http/1.1 as needed automatically, as happens normally.

"minimal config"

In particular, if I remove the alpn policy, the reverse proxy to the python program on localhost:3000 fails with no log in either caddy or that program - the connection just closes:

{
                              "connection_policies": [
                                {
                                  "alpn": ["http/1.1"]  <==== have to put this down there
                                }
                              ]
}

{
  "apps": {
    "http": {
      "servers": {
        "srv0": {
          "listen": [":443"],
          "automatic_https": {
            "disable": true
          },
          "routes": [
            {
              "match": [
                {
                  "host": ["lxc100.duckdns.org"]
                }
              ],
              "handle": [
                {
                  "handler": "subroute",
                  "routes": [
                    {
                      "handle": [
                        {
                          "encodings": {
                            "gzip": {}
                          },
                          "handler": "encode",
                          "prefer": ["gzip"]
                        },
                        {
                          "handler": "reverse_proxy",
                          "headers": {
                            "request": {
                              "set": {
                                "Host": [
                                  "{http.reverse_proxy.upstream.hostport}"
                                ]
                              }
                            }
                          },
                          "transport": {
                            "protocol": "http"
                          },
                          "upstreams": [
                            {
                              "dial": "localhost:3000"
                            }
                          ]
                        }
                      ]
                    }
                  ]
                }
              ],
              "terminal": true
            }
          ],
          "listener_wrappers": [
            {
              "wrapper": "layer4",
              "routes": [
                {
                  "match": [
                    {
                      "tls": {}
                    }
                  ],
                  "handle": [
                    {
                      "handler": "subroute",
                      "routes": [
                        {
                          "match": [
                            {
                              "tls": {
                                "sni": ["lxc100.duckdns.org"]
                              }
                            }
                          ],
                          "handle": [
                            {
                              "handler": "tls",
                              "connection_policies": [
                                {
                                  "alpn": ["http/1.1"]
                                }
                              ]
                            },
                            {
                              "handler": "subroute",
                              "routes": [
                                {
                                  "match": [{ "ssh": {} }],
                                  "handle": [
                                    {
                                      "handler": "proxy",
                                      "upstreams": [
                                        { "dial": ["localhost:22"] }
                                      ]
                                    }
                                  ]
                                },
                                {
                                  "match": [
                                    {
                                      "http": [
                                        {
                                          "host": ["lxc100.duckdns.org"]
                                        }
                                      ]
                                    }
                                  ]
                                }
                              ]
                            }
                          ]
                        }
                      ]
                    }
                  ]
                }
              ]
            }
          ]
        }
      }
    },
    "tls": {
      "certificates": {
        "automate": [
          "localhost",
          "192.168.0.4",
          "lxc100.duckdns.org",
          "lxc101.duckdns.org"
        ]
      },
      "automation": {
        "policies": [
          {
            "subjects": ["localhost", "192.168.0.4"],
            "issuers": [
              {
                "module": "internal"
              }
            ]
          },
          {
            "subjects": ["lxc100.duckdns.org", "lxc101.duckdns.org"],
            "issuers": [
              {
                "challenges": {
                  "dns": {
                    "provider": {
                      "api_token": "{env.DUCKDNS_API_TOKEN}",
                      "name": "duckdns"
                    }
                  }
                },
                "module": "acme"
              },
              {
                "challenges": {
                  "dns": {
                    "provider": {
                      "api_token": "{env.DUCKDNS_API_TOKEN}",
                      "name": "duckdns"
                    }
                  }
                },
                "module": "zerossl"
              }
            ]
          }
        ]
      }
    }
  }
}

My Struggle Reading the Config

The thing I struggle with with so much of the caddy config is that it's very difficult to tell what's part of the API and what's a comment or a convention.

For example, "http" is a key, while "srv0" appears to be used merely as a comment. So while "servers" is semantically a set (unsorted array), it's using the syntax of a struct (key-value map):

{
  "http": {
    "servers": {
      "srv0": {}
    }
  }
}

So when I see something like "listener_wrappers" and I can't find it in a code search of the repository, I think it must be an arbitrary comment, not a struct member / identifier.

I don't know how to learn about these things other than trial and error. Is there a resource that shows more context? It seems most of the doc pages I've visited just show literal {} or similar very short snippets with no idea of where it goes or how to use it.

[WeidiDeng]

Documentation is done by @mholt. And I'll admit some part of it is very confusing, 😅.

Currently any listener wrapper that doesn't return a *tls.Conn will break h2, there is a pr pending.

[mholt]

Yes, sorry, it's confusing. Server names (keys in the "servers" object) are the only things that are "comments" as far as I can recall. They are primarily useful for logging, as opposed to a positional index. (Also, server order doesn't matter, so an object is more fitting.)

Technically listener_wrapper plugins are not restricted to just the HTTP app. They operate on the listener level, so any module that uses a listener can use listener wrappers. It just so happens that only the http module supports them so far. AFAIK there's no reason we can't add listener_wrapper support to this l4 module.