Add fallback to node drain using --disable-eviction flag

[utkuozdemir]

Signed-off-by: Utku Ozdemir uoz@protonmail.com

What type of PR is this?
/kind feature

What this PR does / why we need it:
Explained in #8093:

Kubespray drains every node one by one during a cluster upgrade.
Node drain uses the eviction logic by default.
When there is a PodDisruptionBudget on a workload, that pdb resource can prevent pods that it protects from being evicted from the node.
When this happens, Kubespray upgrade fails and sometimes can leave the cluster half-upgraded (some nodes on older, some nodes on newer version).

The command kubectl drain has the flag --disable-eviction for such cases. When the flag is used, pods are deleted using regular kubectl delete pod logic instead of eviction.

There are setups where the cluster owners and the application owners are not the same people/team. The most common example is multi-tenant clusters, where there can be many PDBs of different tenants on a cluster, and any of them can prevent the cluster upgrades until it is deleted.

This proposal makes it possible to "forcefully" upgrade the clusters in such cases, while giving some grace period for the PDB-protected pods.

Which issue(s) this PR fixes:
Fixes #8093

Special notes for your reviewer:
If this gets approved, I would like to backport it to release-2.16 and release-2.15 branches.

Does this PR introduce a user-facing change?:

Add an optional fallback to node drain during cluster upgrades using `--disable-eviction` flag

[k8s-ci-robot]

Hi @utkuozdemir. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[oomichi]

/cc @oomichi

[oomichi]

Thanks for updating.

/ok-to-test
/lgtm

[floryut]

@utkuozdemir Thank you for that 👍

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: floryut, utkuozdemir

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [floryut]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[utkuozdemir]

@floryut Happy to contribute :). Is it ok if I backport this to 2.16 and 2.15?

[floryut]

@floryut Happy to contribute :). Is it ok if I backport this to 2.16 and 2.15?

2.16 might be possible, 2.15 not really, way too much work to make it usable as of now :)