[fix] Iterables are consumed and are not linked lists.. #365

blefaudeux · 2021-02-04T21:33:17Z

Before submitting

Was this discussed/approved via a Github issue? (no need for typos, doc improvements)
Did you read the contributor guideline?
Did you make sure to update the docs?
Did you write any new necessary tests?

What does this PR do?

Fixes #364 + adds a repro case in the unit test so that this does not happen again. I learnt one thing today, Iterables in python are consumed, they're not just a linked list..

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.
cc @stas00

Did you have fun?

Make sure you had fun coding 🙃

blefaudeux · 2021-02-04T21:36:41Z

fairscale/optim/oss.py

        if self._local_params is None:
-            self._local_params = chain(


the catch in #363 was that I had the wrong mental model for this in Python, I thought that self._local_params was just the top of a linked list, which was never reassigned -> always pointing to the right place. That's actually completely wrong, all the elements in the chain are consumed so this only worked for the first step through

blefaudeux · 2021-02-04T21:36:51Z

tests/optim/test_oss.py

@@ -632,6 +632,9 @@ def check(norm):
        print(f"Checking norm {norm}")
        check(norm)

+        # Check twice, catch an hypothetic iterator dumb mistake


make sure that I don't do this mistake again

stas00 · 2021-02-04T21:44:36Z

I validated that this PR fixes the regression.

min-xu-ai

nice. thanks for the test. I wasn't too familiar with chain() when I first saw it yesterday. I guess learning new thing everyday. :-)

blefaudeux · 2021-02-05T17:15:12Z

nice. thanks for the test. I wasn't too familiar with chain() when I first saw it yesterday. I guess learning new thing everyday. :-)

yeah, chain as such works fine, actually the issue was that I was building an iterator, which is single use, I could have guessed. The first pass goes fine, so the unit tests passed, then the iterator was consumed and the subsequent calls would fail.. The chain removal is because it's not trivial to clone iterators in python, so I figured that just returning a list (of pointers..) was the most efficient and pythonic way

* [chore] Fix lint errors that broke master (#348) authored-by: Anjali Sridhar <anj@devfair0443.h2.fair> * [fix] ShardedDDP - cpu testfix - remove Gloo/CPU (#350) * no idea about the root issue, but it proved to be fairly narrowed (gloo+cpu+python3.8+no cuda installed) so I guess that's out of scope for fairscale * [feat][OSS] elastic and pytorch compatible checkpoints (#310) * adding a test to prove the inter operability with upstream pytorch * updating the changelog * eager state pruning * pytorch 1.5 compat * [fix] ShardedDDP - properly handle post device change (#353) * adding the .to(device) support + unit testing * doc update * [feat] Add AdaScaleWrapper (#347) * [feat] Add AdaScaleWrapper - This enables a different API for wrapping an optimizer with AdaScale. - This also enables AdaScale to be wrapped by OSS. - However, OSS wrapping AdaScale results in different optimization, which future research will be needed to study its effects. testing: add unit tests. * addressed comment: typo * [refactor] Refactor and enable multiprocess nn.Pipe benchmarks. (#319) * mp cleanup * round of multiprocess refactoring * test golden run * print cuda stats * fix lint errors * enable multiprocess pipe benchmarks * set world size to be available gpus * more changes * use synthetic loaders for intermediate pipeline stages * merged master * fix for the devices property * dataloader fix * modify rank check * print wps stats * enable verification * fix logging * fix flag name * fix flag name * check for rank * fix indent * pass args * pass args * modify golden data * remove unused print messsage * fix lint errors * add comments * fix benchmarks Co-authored-by: Anjali Sridhar <anj@devfair0443.h2.fair> * [refactor] pipe: simplify balance and module checks (#346) * [chore] v0.1.5 (#355) * [chore] disheartening switch off of a OSS cpu test (#356) * precise skip, only if agent has only cpu * [feat][minor] OSS Benchmark - regression test + background testing new optims (#352) * restoring the regression test, adding a test of the for_each optims * fix the regression test on circleci * removing unused flags * [refactor] multiprocess_pipe: cleanup __init__ (#357) * [refactor] multiprocess_pipe: remove retain_graph __init__ param (#358) It is not currently being used so we can simplify the interface by removing it. * [refactor] multiprocess_pipe: focus on LazyModule usage (#360) * [feat] ShardedDDP : Adding a proper DDP parity / AMP unit test, overdue (#361) * Adding a proper ddp parity / AMP unit test, overdue * catch non-AMP pytorch * [perf][OSS] Clip grad norm : minor obvious speedup (#363) cache this iterator, easy speed up * [refactor] multiprocess_pipe: remove pipelined_backward (#362) * [perf] ShardedDDP - small memory use reduction - minor speedup (#366) * minor * minor * [fix] repro+fix (#365) fix a broken earlier commit, only worked for the first step * [refactor] OSS only use flat buffers (#371) * flat params all along, way simpler * updating the docstring * [refactor] AsyncPipe: do not sub-class MultiProcessPipe (#370) * [refactor] remove multiprocess dependency on async (#373) * [fix] Workaround need for pip --no-build-isolation (#375) * Add fairscale.nn.misc.checkpoint_activations (#376) * Add fairscale.utils.containers Co-authored-by: Min Xu <24926999+min-xu-ai@users.noreply.github.com> * Add fairscale.nn.misc.checkpoint_activations Co-authored-by: Sam Shleifer <sshleifer@gmail.com> Co-authored-by: Min Xu <24926999+min-xu-ai@users.noreply.github.com> Co-authored-by: Sam Shleifer <sshleifer@gmail.com> * [chore] v0.1.6 (#377) * v0.1.6 Co-authored-by: anj-s <32556631+anj-s@users.noreply.github.com> Co-authored-by: Benjamin Lefaudeux <blefaudeux@users.noreply.github.com> Co-authored-by: Anjali Sridhar <anj@devfair0443.h2.fair> Co-authored-by: msbaines <35972327+msbaines@users.noreply.github.com> Co-authored-by: Leonard Lausen <leonard@lausen.nl> Co-authored-by: Myle Ott <myleott@fb.com> Co-authored-by: Sam Shleifer <sshleifer@gmail.com>

repro+fix

e17ed71

blefaudeux requested review from min-xu-ai, anj-s and msbaines February 4, 2021 21:33

facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Feb 4, 2021

blefaudeux commented Feb 4, 2021

View reviewed changes

blefaudeux changed the title ~~[fix] Iterables are consumed and not linked lists..~~ [fix] Iterables are consumed and are not linked lists.. Feb 4, 2021

min-xu-ai approved these changes Feb 5, 2021

View reviewed changes

blefaudeux merged commit 8778fa6 into master Feb 5, 2021

blefaudeux deleted the issue_364 branch February 11, 2021 01:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[fix] Iterables are consumed and are not linked lists.. #365

[fix] Iterables are consumed and are not linked lists.. #365

blefaudeux commented Feb 4, 2021

blefaudeux Feb 4, 2021

blefaudeux Feb 4, 2021

stas00 commented Feb 4, 2021

min-xu-ai left a comment

blefaudeux commented Feb 5, 2021 •

edited

Loading

[fix] Iterables are consumed and are not linked lists.. #365

[fix] Iterables are consumed and are not linked lists.. #365

Conversation

blefaudeux commented Feb 4, 2021

Before submitting

What does this PR do?

PR review

Did you have fun?

blefaudeux Feb 4, 2021

Choose a reason for hiding this comment

blefaudeux Feb 4, 2021

Choose a reason for hiding this comment

stas00 commented Feb 4, 2021

min-xu-ai left a comment

Choose a reason for hiding this comment

blefaudeux commented Feb 5, 2021 • edited Loading

blefaudeux commented Feb 5, 2021 •

edited

Loading