Prevent replicas running on the same node

[adyanth]

When running size > 1, there is not much use in running them on the same node.

Look into adding a soft (preferredDuringSchedulingIgnoredDuringExecution) podAntiAffinity with topologyKey: kubernetes.io/hostname so that replicas won't end up on the same node.

[adyanth]

See #39 and #52

[matthewhembree]

This will be nice!

In the meantime, I used two Kyverno poliices to get podAntiAffinity and topologySpreadConstraints.

podAntiAffinity:

# cluster-tunnel-policy-antiaffinity.yaml
apiVersion: kyverno.io/v1
kind: Policy
metadata:
  name: cfargotunnel-insert-pod-antiaffinity
  namespace: cloudflare-operator-system
  annotations:
    policies.kyverno.io/title: Add Pod Anti-Affinity
    policies.kyverno.io/category: Sample
    policies.kyverno.io/subject: Deployment, Pod
    policies.kyverno.io/minversion: 1.6.0
    policies.kyverno.io/description: >-
      Applications may involve multiple replicas of the same Pod for availability as well as scale
      purposes, yet Kubernetes does not by default provide a solution for availability. This policy
      sets a Pod anti-affinity configuration on Deployments which contain an `cfargotunnel.com/app`
      label if it is not already present.
spec:
  rules:
    - name: insert-pod-antiaffinity
      match:
        any:
        - resources:
            kinds:
              - Deployment
      preconditions:
        # This precondition selects Pods with the label `cfargotunnel.com/app` defined
        all:
        - key: "{{request.object.spec.template.metadata.labels.\"cfargotunnel.com/app\" || ''}}"
          operator: NotEquals
          value: ""
      # Mutates the Deployment resource to add fields.
      mutate:
        patchStrategicMerge:
          spec:
            template:
              spec:
                # Add the `affinity`if not already specified.
                +(affinity):
                  +(podAntiAffinity):
                    +(preferredDuringSchedulingIgnoredDuringExecution):
                      - weight: 1
                        podAffinityTerm:
                          topologyKey: "kubernetes.io/hostname"
                          labelSelector:
                            matchExpressions:
                            - key: cfargotunnel.com/app
                              operator: In
                              values:
                              - "{{request.object.spec.template.metadata.labels.\"cfargotunnel.com/app\"}}"

topologySpreadConstraints:

# cluster-tunnel-policy-topologyspread.yaml
apiVersion: kyverno.io/v1
kind: Policy
metadata:
  name: cfargotunnel-spread-pods
  namespace: cloudflare-operator-system
  annotations:
    policies.kyverno.io/title: Spread Pods Across Nodes
    policies.kyverno.io/category: Sample
    policies.kyverno.io/subject: Deployment, Pod
    policies.kyverno.io/minversion: 1.6.0
    policies.kyverno.io/description: >-
      Deployments to a Kubernetes cluster with multiple availability zones often need to
      distribute those replicas to align with those zones to ensure site-level failures
      do not impact availability. This policy matches Deployments with the label
      `cfargotunnel.com/app=cloudflared` and mutates them to spread Pods across zones.
spec:
  rules:
    - name: spread-pods-across-nodes
      # Matches any Deployment with the label `cfargotunnel.com/app=cloudflared`
      match:
        any:
        - resources:
            kinds:
            - Deployment
            selector:
              matchLabels:
                cfargotunnel.com/app: cloudflared
      # Mutates the incoming Deployment.
      mutate:
        patchStrategicMerge:
          spec:
            template:
              spec:
                # Adds the topologySpreadConstraints field if non-existent in the request.
                +(topologySpreadConstraints):
                - maxSkew: 1
                  topologyKey: topology.kubernetes.io/zone
                  whenUnsatisfiable: DoNotSchedule
                  labelSelector:
                    matchLabels:
                      cfargotunnel.com/app: cloudflared