Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Core] skylet periodically update job status #3469

Merged
merged 1 commit into from
Apr 24, 2024
Merged

Conversation

Michaelvll
Copy link
Collaborator

@Michaelvll Michaelvll commented Apr 23, 2024

The periodic job status update was accidentally dropped due to the job queue scheduling. This will cause job failed in ray job failed to be set to FAILED but stuck in RUNNING.

The following could reproduce the issue:

  1. git reset --hard 1bdc (the commit before the fix for the ray.wait timeout, [Core] Fix jobs longer than 12 days #3460)
  2. Add timeout=600 for ray.wait to simulate the issue in cloud_vm_ray_backend.py
  3. sky serve up -n test-index-recover-2 examples/serve/http_server/task.yaml --cloud gcp

After 10 mins, the serve controller job fails, but the service is shown READY forever, and sky queue sky-serve-controller-<hash> shows the controller job is running, while ray job list shows the job is failed already on the controller VM.

Tested (run the relevant ones):

  • Code formatting: bash format.sh
  • Any manual or new tests for this PR (please specify below)
  • All smoke tests: pytest tests/test_smoke.py
  • Relevant individual smoke tests: pytest tests/test_smoke.py::test_fill_in_the_name
  • Backward compatibility tests: bash tests/backward_comaptibility_tests.sh

@Michaelvll Michaelvll requested a review from cblmemo April 23, 2024 20:54
Copy link
Collaborator

@cblmemo cblmemo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the quick fix! LGTM.

@Michaelvll Michaelvll merged commit f131124 into master Apr 24, 2024
20 checks passed
@Michaelvll Michaelvll deleted the update-job-status branch April 24, 2024 19:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants