Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Does sat finetune support to resume the lastest checkpoint to continue training #372

Open
trouble-maker007 opened this issue Sep 27, 2024 · 4 comments
Assignees

Comments

@trouble-maker007
Copy link

I found that if the training processing break, it can not resume the lastest checkpoint to continue training

@zRzRzRzRzRzRzR zRzRzRzRzRzRzR self-assigned this Sep 27, 2024
@zRzRzRzRzRzRzR
Copy link
Member

yes, do you using sat or diffusers, for diffusers, just add resume_from_checkpoint

@trouble-maker007
Copy link
Author

trouble-maker007 commented Sep 27, 2024

@zRzRzRzRzRzRzR I use sat, I can't find the configuration parameters in the SAT code, how to add the resume_from_checkpoint?

@zRzRzRzRzRzRzR
Copy link
Member

SAT indeed does not provide a direct solution, but there is a way
You directly change the address of the trained model to the training address, and modify the name of that folder to 1, and also change the number in latest to 1
In this way, the SAT system mistakenly thinks that this is a model that has just started training and can continue training.
(This is indeed because the framework is not well adapted)

@trouble-maker007
Copy link
Author

@zRzRzRzRzRzRzR thanks for quickly response, looking forward to the diffusers version

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants