Support for Gateway API

[mark-church]

The Gateway API is a new spec from K8s SIG-Network that models load balancing, but through a new set of resources instead of Ingress. It defines its load balancer IP address and hostname routing rules in different resources like this example. I'm just filing this as a tracker to record requests, use-cases, and interest for external-dns support of the Gateway API resources.

cc @bowei @robscott

[seanmalloy]

This would be a very nice feature.

/help

[k8s-ci-robot]

@seanmalloy:
This request has been marked as needing help from a contributor.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

In response to this:

This would be a very nice feature.

/help

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[sgreene570]

cc @danehans

[mark-church]

Are there any external-dns contributors out there that would be interested in talking to the folks working on Gateway API to learn more about it? I'm happy to help facilitate and bring folks up to speed. It would be great to start some discussion on this topic to see how and whether automating DNS records based on Gateway API resources could work.

[abursavich]

I'm not sure if I've contributed to external-dns before, but this is a feature I want/need and I'd be happy to implement it.

It might be good to set some expectations on API version support. For instance, we might want to be liberal about dropping support for old alpha versions as new versions come along. It looks like v1alpha1 is on its way out and v1alpha2 is coming along.

[abursavich]

I just read through the API. Each route can have multiple hostnames, multiple routes can overlap hostnames, each route can be attached to multiple gateways, and each gateway can have multiple addresses... It'll be interesting figuring out what to do when the same hostname maps to multiple gateways.

[abursavich]

I looked at this over the weekend and have a few updates:

• The way external-dns works, the sources output all the different endpoints and don't get a say in how to break ties. Something could be done to try to force the Gateway API tie-breaking philosophy into external-dns, but I'm not sure if it's worth it yet.
• Adding sources for HTTPRoute and TLSRoute are fairly trivial. Adding sources for TCPRoute and UDPRoute are doable via annotations and/or templates, but the Gateway API doesn't natively provide hostnames for them. The first pass might not include them.
• By far the biggest hurdle to implementing this is upgrading the Kubernetes dependencies of external-dns to support the versions that the Gateway API uses. As evidenced by this comment, it makes the unstructured dynamic clients used by a handful of sources completely crap the bed in tests. I can probably fix all the tests, but they're using fake clients and I'm not sure if the real clients will be broken.

[bowei]

@robscott ^^^ we were kicking around the idea of adding hostname to TCP, UDP route.

[abursavich]

FWIW to any Gateway API people who are less familiar with external-dns, there are two existing mechanisms that could be used to add hostnames to resources without them. There's an annotation (external-dns.alpha.kubernetes.io/hostname) and a command line flag (fqdn-template). More details in the FAQ.

[abursavich]

By far the biggest hurdle to implementing this is upgrading the Kubernetes dependencies of external-dns to support the versions that the Gateway API uses. As evidenced by this comment, it makes the unstructured dynamic clients used by a handful of sources completely crap the bed in tests. I can probably fix all the tests, but they're using fake clients and I'm not sure if the real clients will be broken.

I have this all sorted out now and have gotten rid of all unstructured dynamic clients. I've kept all the commits logically separated and as small as possible, but there's no denying that it's a lot of changes. I don't really think it'll work well to just throw a bunch of PRs over the fence without the larger context. If any external-dns owners would like to work with me to get it through, that'd be really appreciated.

[abursavich]

My initial implementation of this was pretty simple. It just got the hostnames for a given route and the ip addresses from the gateways listed in the route's status. Done! But looking the spec over again, I think it might need to go deeper into gateway listeners (hostname, protocol, routes, namespaces, kinds, etc.) to determine which hostnames are associated with which gateways/addresses. A route may have multiple hostnames where each is associated with different gateways/addresses and the DNS records should reflect that... So this will need to implement the full core gateway route selection logic, unless the RouteParentStatus gets updated to specify which hostnames the gateway is allowing for the route.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[abursavich]

/remove-lifecycle stale

[mark-church]

@abursavich I have been meaning to provide feedback for a while and I am just getting around to it. Are you still working on this or planning for another iteration of your implementation? We should have you attend the weekly Gateway OSS meeting at some point to discuss this design in detail.

There are two design challenges when it comes to DNS and Gateway-Route interaction:

1. It's possible to have configs in which DNS names are assigned to more than one Gateway IP address which is invalid
2. DNS names can be assigned in the Gateway.spec.listener.hostname and also in the HTTPRoute.spec.hostnames field. We have to decide what the interaction between these two fields means.

Here's an example diagram that explores the design aspects of DNS with Gateway. There are three Gateways and each maps to a unique IP.

Here's a rough sketch of how these resources could be interpreted:

• DNS names can be configured from the HTTPRoute or the Gateway, but there are rules or how these fields interact which determine when they are programmed.
• HTTPRoutes a1 and a2 both specify a.example.com but because they are for the same Gateway there is no conflict.
• HTTPRoutes b1 and b2 both specify b.example.com but for different Gateways. This is a problem because the DNS server has multiple entries and it may not be the user's intention to resolve b.example.com to both IP1 + IP2. This is a conflict and should result in b.example.com not being programmed at all.
• Gateway.spec.listener.hostname can be used to program a DNS entry or to act as a filter for which HTTPRoute.spec.hostnames fields are programmed. When the Gateway hostname has a wildcard it acts as a filter and allows any HTTPRoutes that match the wildcard to program DNS. For example:
  • For gw1, HTTPRoutes a1, a2, and b1 are allowed to program DNS because they all match *.example.com.
  • For gw2, if the listener.hostname is empty then it acts as a * and allows any HTTPRoute DNS names to be programmed.
  • For gw3, the listener.hostname is configured explicitly on the Gateway. d.example.com from HTTPRoute d1 is not programmed because it does not match gw3.

If the Gateway is managed by an administrator then this Gateway-Route interaction is powerful. It creates a programmatic interface to allow admins to control which domains app owners/HTTPRoute owners can configure DNS names. It enables the following use-case where an administrator can give DNS subdomain ownership to specific Namespaces by restricting the domain using the Gateway DNS filter and also restricting the Namespaces which can use the Gateway:

cc @bowei @robscott @hbagdi

[abursavich]

My hope was to get this iteration merged before possibly following up with future iterations. I just keep rebasing it to deal with new merge conflicts.

The way it works right now is by looking at each Route in isolation from other Routes. It only pairs Gateways that have admitted the Route (according to the Route's status) and Hostnames for which there is a matching Listener. I had to implement Listener matching on my own, but it includes Protocol matching (HTTPRoutes match HTTP or HTTPS protocols, while other protocols require exact matches), Hostname matching (supporting single-level subdomain wildcard or full wildcard on the Listener Hostname), and AllowedRoutes namespace matching.

Each Route may generate a list of hostnames each with its own set of Gateway IP addresses (these may be from one or more Gateways). Handling conflicts where the same hostname claims different IP addresses and settings (e.g. TTL) is left to the existing external-dns logic, as there may be conflicts across sources (e.g. Gateway API, Ingress, Istio, etc.).

One known deficiency in the current implementation is that it doesn't support "reducing" wildcard hostnames from the Route to the Gateway/Listener. For example, if the Route includes *.foo.com and the Listener includes bar.foo.com, then they won't match. Likewise, if the Route includes * and the Listener includes foo.com... However, it does work in the opposite direction (e.g. bar.foo.com in the Route and *.foo.com in the Listener).

[abursavich]

I went ahead and fixed the above mentioned issue with wildcards.

[abursavich]

I presented this at the Gateway OSS meeting today, as suggested by @mark-church, so I wrote up a short doc about the design and implementation.

[gAmUssA]

@abursavich @mark-church team, any updates on this? Thank you

[abursavich]

The design has been reviewed and approved in a meeting of the SIG-Network Gateway API working group. The code has been reviewed and approved by a Gateway API owner. Now we're waiting on review from a external-dns owner.

I reassigned the PR to a new owner this morning because I noticed that the status of the existing assignee is "💤 Taking a break from open source." I'm not sure how long that's been the case.

[abursavich]

🍾 🥂 🥳 🎉

[BeatrizBaldaia]

Hi!
I am using external-dns and k8s gateway api. I am using helm to deploy both. After adding the gateway-http source to the external-dns helm values, which will also configure the cluster role and etc, I was expecting it to create the A and TXT records for the hostnames that I pass in the HTTPRoute resource specs, but it that is not happening... In the logs there is only this message:

{"level":"debug","msg":"No endpoints could be generated from HTTPRoute core/test-route-1","time":"2023-09-26T15:32:38Z"}

Do I need to pass any annotation to the HTTPRoute resource? I don't want to use the annotation external-dns.alpha.kubernetes.io/hostname since one HTTPRoute can specify multiple hostnames.

More details:

• The Gateway has one listener for hostname named like "*.stage.region.example"
• The HTTPRoute has only one hostname named like "app.stage.region.example"
• The HTTPRoute backendRef is a Service which is in the same namespace as the route
• Gateway and HTTPRoute are in the same namespace

Update:

Problem solved. I missed adding the parentRefs in the HTTPRoute. I thought it wouldn't be necessary it's an optional field.

What is still strange is that, even though in the HTTPRoute status the parentRef is present:

status:
  parents:
  - conditions:
    - lastTransitionTime: "2023-09-26T17:30:03Z"
      message: Accepted HTTPRoute
      observedGeneration: 3
      reason: Accepted
      status: "True"
      type: Accepted
    controllerName: io.cilium/gateway-controller
    parentRef:
      group: gateway.networking.k8s.io
      kind: Gateway
      name: test
      sectionName: test-listener-1

In the Gateway status no route is attached to the Gateway listener, following the Gateway status:

status:
  addresses:
  - type: Hostname
    value: <ADDRESS>
  conditions:
  - lastTransitionTime: "2023-09-26T16:29:39Z"
    message: Gateway successfully scheduled
    observedGeneration: 2
    reason: Scheduled
    status: "True"
    type: Scheduled
  - lastTransitionTime: "2023-09-26T16:29:39Z"
    message: Gateway successfully reconciled
    observedGeneration: 2
    reason: Ready
    status: "True"
    type: Ready
  listeners:
  - attachedRoutes: 0
    conditions:
    - lastTransitionTime: "2023-09-26T16:29:39Z"
      message: Listener Ready
      observedGeneration: 2
      reason: Ready
      status: "True"
      type: Ready
    name: test-listener-1
    supportedKinds:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute