[Core] Upgrade ray to 2.3.0 #1618

Michaelvll · 2023-01-23T22:10:39Z

This is a routine upgrade of the ray version we are using from 2.0.1 to 2.3.0.

TODO:

Figure out the backward compatibility

Tested (run the relevant ones):

…nto upgrade-ray-2.2

Michaelvll · 2023-01-26T20:59:45Z

sky/skylet/ray_patches/__init__.py

-    from ray.dashboard.modules.job import job_manager
-    _run_patch(job_manager.__file__, _to_absolute('job_manager.py.patch'))
-


Reverted this, as our patch will make the ray job submit stuck randomly in the latest ray, and the tests are all passed without the patch. It is possible that the await ray_func.remote() not raising exception problem is fixed.

TODO:

Find out the ray's PR that solves the problem (optional) (possibly fixed by [Serve][Core] Fix serve_long_running memory leak by fixing GCS pubsub asyncio task leak ray-project/ray#29187)

Try large scale job submission

Nice catch! Do we have the snippet used when developing the patch?

sky/skylet/ray_patches/__init__.py

concretevitamin

Thanks @Michaelvll for upgrading this. This will pave the way for Graviton. Did a pass.

Shall we run smoke tests --aws as well, since node provider is being updated?
The back compat on 3 clouds may need to be tested.

concretevitamin · 2023-01-28T05:37:32Z

sky/backends/cloud_vm_ray_backend.py

@@ -193,7 +193,7 @@ def add_prologue(self,
        # Should use 'auto' or 'ray://<internal_head_ip>:10001' rather than
        # 'ray://localhost:10001', or 'ray://127.0.0.1:10001', for public cloud.
        # Otherwise, it will a bug of ray job failed to get the placement group
-        # in ray <= 2.0.1.
+        # in ray <= 2.2.0.


This seems unneeded?

Removed this in #1734.

concretevitamin · 2023-01-28T05:38:01Z

sky/setup_files/setup.py

@@ -64,7 +64,7 @@ def parse_readme(readme: str) -> str:

 install_requires = [
    'wheel',
-    # NOTE: ray 2.0.1 requires click<=8.0.4,>=7.0; We disable the
+    # NOTE: ray 2.2.0 requires click<=8.0.4,>=7.0; We disable the


Is this true for 2.2?

Ahh, good catch. They recently support click>8.0.4 in ray==2.2.0 ray-project/ray#29574

TODO:

Test with the latest click 8.1.3 pytest tests/test_smoke.py

sky/skylet/providers/aws/cloudwatch/cloudwatch_helper.py

sky/skylet/providers/aws/node_provider.py

concretevitamin · 2023-01-28T05:52:09Z

sky/skylet/ray_patches/log_monitor.py.patch

-0a1,3
-> # Adapted from https://github.com/ray-project/ray/blob/ray-2.0.1/python/ray/_private/log_monitor.py
+0a1,4
+> # Adapted from https://github.com/ray-project/ray/blob/ray-2.2.0/python/ray/_private/log_monitor.py
 > # Fixed the problem for progress bar, as the latest version does not preserve \r for progress bar.
 > # The change is adapted from https://github.com/ray-project/ray/blob/ray-1.10.0/python/ray/_private/log_monitor.py#L299-L300


Do we want both L2 and L4?

Ahh, sorry for the confusion. Fix the comment to:

# It is reverted to https://github.com/ray-project/ray/blob/ray-1.10.0/python/ray/_private/log_monitor.py#L299-L300

concretevitamin · 2023-01-28T05:54:27Z

sky/skylet/ray_patches/__init__.py

-    from ray.dashboard.modules.job import job_manager
-    _run_patch(job_manager.__file__, _to_absolute('job_manager.py.patch'))
-


Nice catch! Do we have the snippet used when developing the patch?

sky/skylet/ray_patches/__init__.py

…nto upgrade-ray-2.2

concretevitamin

Thanks @Michaelvll. Some questions.

concretevitamin · 2023-02-14T16:56:45Z

sky/skylet/ray_patches/log_monitor.py.patch

@@ -1,7 +1,7 @@
 0a1,4
 > # Adapted from https://github.com/ray-project/ray/blob/ray-2.2.0/python/ray/_private/log_monitor.py
 > # Fixed the problem for progress bar, as the latest version does not preserve \r for progress bar.
-> # The change is adapted from https://github.com/ray-project/ray/blob/ray-1.10.0/python/ray/_private/log_monitor.py#L299-L300
+> # It is reverted to https://github.com/ray-project/ray/blob/ray-1.10.0/python/ray/_private/log_monitor.py#L299-L300


A bit confused: what is reverted to what...? Not understanding this line given L2.

Fixed in #1734. PTAL. : )

concretevitamin · 2023-02-14T16:58:17Z

sky/templates/azure-ray.yml.j2

@@ -124,8 +124,8 @@ setup_commands:
    (type -a pip | grep -q pip3) || echo 'alias pip=pip3' >> ~/.bashrc;
    which conda > /dev/null 2>&1 || (wget -nc https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh && bash Miniconda3-latest-Linux-x86_64.sh -b && eval "$(/home/azureuser/miniconda3/bin/conda shell.bash hook)" && conda init && conda config --set auto_activate_base true);
    source ~/.bashrc;
-    (pip3 list | grep ray | grep {{ray_version}} 2>&1 > /dev/null || pip3 install -U ray[default]=={{ray_version}}) && mkdir -p ~/sky_workdir && mkdir -p ~/.sky/sky_app && touch ~/.sudo_as_admin_successful;
-    (pip3 list | grep skypilot && [ "$(cat {{sky_remote_path}}/current_sky_wheel_hash)" == "{{sky_wheel_hash}}" ]) || (pip3 uninstall skypilot -y; pip3 install "$(echo {{sky_remote_path}}/{{sky_wheel_hash}}/skypilot-{{sky_version}}*.whl)[azure]" && echo "{{sky_wheel_hash}}" > {{sky_remote_path}}/current_sky_wheel_hash || exit 1);
+    (pip3 list | grep "ray " | grep {{ray_version}} 2>&1 > /dev/null || pip3 install --exists-action w -U ray[default]=={{ray_version}}) && mkdir -p ~/sky_workdir && mkdir -p ~/.sky/sky_app && touch ~/.sudo_as_admin_successful;


nit: spell out --exists-action wipe for clarity

Why do we need it now? It seems like the grep would've ensured when we reach pip3 install there's no existing package?

This is needed, because otherwise, the original patched files will not be removed from the package, causing the upgraded ray package corrupted due to the staled files. This makes sure that the existing VM upgraded with sky launch (upgrading ray version) will not have staled files in the ray package

nit: spell out --exists-action wipe for clarity

It seems only w is valid, but wipe is not.

concretevitamin · 2023-02-14T17:04:32Z

sky/skylet/ray_patches/__init__.py

-    from ray.dashboard.modules.job import job_manager
-    _run_patch(job_manager.__file__, _to_absolute('job_manager.py.patch'))
-


concretevitamin · 2023-02-14T17:05:07Z

sky/backends/cloud_vm_ray_backend.py

@@ -193,7 +193,7 @@ def add_prologue(self,
        # Should use 'auto' or 'ray://<internal_head_ip>:10001' rather than
        # 'ray://localhost:10001', or 'ray://127.0.0.1:10001', for public cloud.
        # Otherwise, it will a bug of ray job failed to get the placement group
-        # in ray <= 2.0.1.
+        # in ray <= 2.2.0.


concretevitamin · 2023-02-14T17:05:55Z

sky/setup_files/setup.py

@@ -64,9 +64,9 @@ def parse_readme(readme: str) -> str:

 install_requires = [
    'wheel',
-    # NOTE: ray 2.0.1 requires click<=8.0.4,>=7.0; We disable the
+    # NOTE: ray 2.2.0 requires click>=7.0; We disable the


Local ray versions may be older than 2.2. Does that mean the click<=8.0.4 constraint is still needed?

I decided to upgrade the local ray version to 2.2.0 due to a bunch of the conflicts of the dependencies in #1734 . Wdyt?

…nto upgrade-ray-2.3

* update the patches * upgrade node providers * fix azure config.py * print sky queue * add back azure disk size * fix job manager * fix hash * longer timeout * fix test smoke * Remove the patch for job_manager * longer timeout for azure_region test * address comments * format * fix templates * pip install --exists-action * Upgrade to 2.3 instead * upgrade to ray 2.3 * update patches for 2.4 * adopt changes for azure providers: a777a028b8dbd7bbae9a7393c98f6cd65f98a5f5 * fix license * fix patch for log monitor * sleep longer for the multi-echo * longer waiting time * longer wait time * fix click dependencies * update setup.py * Fix #1618 (comment) * fix #1618 (comment) * revert test_smoke * fix comment * revert to w instead of wipe * rewording * minor fix

Michaelvll added 2 commits January 23, 2023 13:36

update the patches

ca94eba

upgrade node providers

1651e7f

WoosukKwon mentioned this pull request Jan 24, 2023

Support AWS Graviton instances #1586

Open

Michaelvll added 11 commits January 24, 2023 09:52

Merge branch 'master' of github.com:concretevitamin/sky-experiments i…

55dd5b4

…nto upgrade-ray-2.2

Merge branch 'master' of github.com:concretevitamin/sky-experiments i…

dc3c14f

…nto upgrade-ray-2.2

fix azure config.py

d4ea222

print sky queue

ffb6f7b

add back azure disk size

d33593e

fix job manager

380d8b6

Merge branch 'master' of github.com:concretevitamin/sky-experiments i…

16fe424

…nto upgrade-ray-2.2

fix hash

c21d46d

longer timeout

99cc5dc

fix test smoke

5ad228d

Remove the patch for job_manager

e3f0c60

Michaelvll commented Jan 26, 2023

View reviewed changes

longer timeout for azure_region test

3e42635

Michaelvll marked this pull request as ready for review January 27, 2023 07:24

concretevitamin self-requested a review January 28, 2023 05:39

concretevitamin reviewed Jan 28, 2023

View reviewed changes

Michaelvll added 6 commits January 31, 2023 19:57

Merge branch 'master' of github.com:concretevitamin/sky-experiments i…

e0d8e7c

…nto upgrade-ray-2.2

address comments

0cb298b

Merge branch 'master' of github.com:concretevitamin/sky-experiments i…

366173b

…nto upgrade-ray-2.2

format

caee0e1

fix templates

4e280a3

pip install --exists-action

582b0ba

concretevitamin reviewed Feb 14, 2023

View reviewed changes

Upgrade to 2.3 instead

4351433

Michaelvll changed the title ~~[Core] Upgrade ray to 2.2.0~~ [Core] Upgrade ray to 2.3.0 Feb 27, 2023

Michaelvll added 2 commits February 27, 2023 15:28

upgrade to ray 2.3

79627b8

Merge branch 'master' of github.com:concretevitamin/sky-experiments i…

9cc992e

…nto upgrade-ray-2.3

Michaelvll closed this Feb 27, 2023

Michaelvll deleted the upgrade-ray-2.2 branch February 27, 2023 23:31

Michaelvll restored the upgrade-ray-2.2 branch February 27, 2023 23:31

Michaelvll deleted the upgrade-ray-2.2 branch February 27, 2023 23:32

Michaelvll added a commit that referenced this pull request May 23, 2023

Fix #1618 (comment)

52571e9

Michaelvll added a commit that referenced this pull request May 23, 2023

fix #1618 (comment)

b312244

Michaelvll mentioned this pull request May 23, 2023

[Core] Upgrade ray to 2.4.0 #1734

Merged

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Core] Upgrade ray to 2.3.0 #1618

[Core] Upgrade ray to 2.3.0 #1618

Michaelvll commented Jan 23, 2023 •

edited

Loading

Michaelvll Jan 26, 2023 •

edited

Loading

concretevitamin Jan 28, 2023

concretevitamin Feb 14, 2023

concretevitamin left a comment •

edited

Loading

concretevitamin Jan 28, 2023

concretevitamin Feb 14, 2023

Michaelvll May 23, 2023

concretevitamin Jan 28, 2023

Michaelvll Feb 1, 2023 •

edited

Loading

concretevitamin Jan 28, 2023

Michaelvll Feb 1, 2023

concretevitamin Jan 28, 2023

concretevitamin left a comment

concretevitamin Feb 14, 2023

Michaelvll May 23, 2023

concretevitamin Feb 14, 2023

Michaelvll May 23, 2023 •

edited

Loading

concretevitamin Feb 14, 2023

concretevitamin Feb 14, 2023

concretevitamin Feb 14, 2023

Michaelvll May 23, 2023

		from ray.dashboard.modules.job import job_manager
		_run_patch(job_manager.__file__, _to_absolute('job_manager.py.patch'))

[Core] Upgrade ray to 2.3.0 #1618

[Core] Upgrade ray to 2.3.0 #1618

Conversation

Michaelvll commented Jan 23, 2023 • edited Loading

Michaelvll Jan 26, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

concretevitamin left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Michaelvll Feb 1, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

concretevitamin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Michaelvll May 23, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Michaelvll commented Jan 23, 2023 •

edited

Loading

Michaelvll Jan 26, 2023 •

edited

Loading

concretevitamin left a comment •

edited

Loading

Michaelvll Feb 1, 2023 •

edited

Loading

Michaelvll May 23, 2023 •

edited

Loading