[k8s] Optimize ingress creation to avoid nginx hot-reloads

[romilbhardwaj]

Each ingress rule creation triggers a hot reload of the nginx controller. Since previously SkyPilot generates the listed ingresses sequentially, this could lead to multiple reloads of the Nginx-Ingress-Controller within a brief period.

Consequently, the Nginx-Controller pod might spawn an excessive number of sub-processes. This surge triggers Kubernetes to kill and restart the Nginx due to the podPidsLimit parameter, which is typically set to a default value like 1024, and thus disrupts normal request handling.

This PR fixes this issue by batching ingress creation into one object containing multiple rules.

Tested (run the relevant ones):

• Code formatting: bash format.sh
• Tested: sky launch -c test --ports 8888-8899

[Michaelvll]

Thanks for fixing this @romilbhardwaj! LGTM with two questions.

[Michaelvll]

Could we add the PR description in the comments here or some other places in the code for why we batch the rules? : )

Suggested change

	content = network_utils.fill_ingress_template(
	namespace=provider_config.get('namespace', 'default'),
	service_details=service_details,
	ingress_name=f'{cluster_name_on_cloud}-skypilot-ingress',
	selector_key='skypilot-cluster',
	selector_value=cluster_name_on_cloud,
	)
	content = network_utils.fill_ingress_template(
	namespace=provider_config.get('namespace', 'default'),
	service_details=service_details,
	ingress_name=f'{cluster_name_on_cloud}-skypilot-ingress',
	selector_key='skypilot-cluster',
	selector_value=cluster_name_on_cloud,
	)

[Michaelvll]

Just curious, if we are changing the ingress for all the ports in batch, will this cause the other ports being unavailable for a while when a new port is added?

Also, just to confirm, when adding a new port, the original endpoint for the existing ports do not change?

[romilbhardwaj]

Just curious, if we are changing the ingress for all the ports in batch, will this cause the other ports being unavailable for a while when a new port is added?

Yes, that's true. Nginx will do a hot reload whenever a new ingress rule is created. As a result, ports may be unavailable for a short duration before getting restotred.

Also, just to confirm, when adding a new port, the original endpoint for the existing ports do not change?

Yes, the original endpoint will remain the same, since the k8s service remains the same.