[Test/Azure] Fix the torch version in examples for smoke test and change the credential for Azure

[Michaelvll]

Closes #1327

PyTorch bumps up the version to 1.13.0 yesterday, which causes the cuda version mismatch problem, when directly installing the latest PyTorch==1.13.0 pip install torch.

The backward compatibility should be guaranteed by our _RAY_YAML_KEYS_TO_RESTORE_FOR_BACK_COMPATIBILITY in backend_utils.py, as the ami is set in the node_config section.

skypilot/sky/backends/backend_utils.py

Line 126 in 855d4b4

_RAY_YAML_KEYS_TO_RESTORE_FOR_BACK_COMPATIBILITY = {

Update: Have to revert the aws/gcp/azure image, since failed test_cancel_azure (The latest 22.09.14 azure image has an older nvidia driver 470.141.03 cuda version 11.3 than our previous azure image 21.11.04 with nvidia-driver 495.29.05, cuda 11.5).

Future TODO:

• Ask the user to use older version of the pytorch to be able to run with CUDA lower than 11.6 [Docs] Add instruction for installing common ML framework #1333

Tested:

• ./tests/run_smoke_tests.sh

[concretevitamin]

Thanks @Michaelvll. Some questions.

[concretevitamin]

Q: Just to understand, if the old image also comes with CUDA 11.6, this image upgrade is not necessary to fix the pytorch issue?

(I'm weary of us frequently updating images. If there's no strong reasons or user demands, I think keeping the images stable is better.)

[Michaelvll]

Good point!

The latest version of the AMI supports the PyTorch with the CUDA 11.7, which is the one installed by default with pip install torch, but the previous version does not.

I agree that we should not frequently update the images, and since the other cloud does not support PyTorch with CUDA 11.7. I am now leaning toward reverting it.

[suquark]

have you checked if the AMI has an older version of CUDA under /usr/local (or other places)? then we only need to softlink /usr/local/cuda to the older version. this is how you are supposed to handle backwards compatibility

[suquark]

e.g., in setup we can have sudo rm /usr/local/cuda && sudo ln -s /usr/local/cudaxxxx /usr/local/cuda

[Michaelvll]

I think the backward compatibility of cuda is handled well even without re-do the soft link, as the pip install torch==1.12.1+cu113 works well on the VM with cuda 11.6 installed.

[concretevitamin]

cc @suquark to take a look.

IIUC, @suquark added the change here. According to #1332 (comment) if this new change doesn't fix the non-deterministic error maybe we should keep the old version?

[Michaelvll]

Since we asked the user to generate the credentials with azure-cli in our documents and oursky check only checks the azure-cli crendentials, we may want to keep it consistent here for the credential to avoid unexpected errors caused by another credential manager, e.g. the exclude_managed_identity_credential we added before was to avoid the azure credential manager confusing the node_provider.

[suquark]

I am ok with this change, since we really only need azure cli credential. allowing other credentials may result in using wrong credentials. we can revert it later if really other credentials are required later

[concretevitamin]

LGTM. Questions

• now this PR upgrades GCP images only; is this required?
• should we update the PR title?

[Michaelvll]

Passed the smoke tests. Merging.