Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use decorator one_rank_only on Checkpoint and BaseLogger #1055

Closed
sdesrozis opened this issue May 19, 2020 · 1 comment
Closed

Use decorator one_rank_only on Checkpoint and BaseLogger #1055

sdesrozis opened this issue May 19, 2020 · 1 comment
Assignees
@sdesrozis
Labels
enhancement

Comments

@sdesrozis
Copy link
Contributor

🚀 Feature

Handlers may not be parallel and be triggered on the master proc (rank=0). The idea is to use the decorator one_rank_only (in branch idist) on __call__ methods or specific working methods for Checkpoint and BaseLogger.
@sdesrozis sdesrozis self-assigned this May 19, 2020
@erip
Copy link
Contributor

erip commented May 20, 2020

One thing to keep in mind is that in the XLA context, this constraint is maintained internally in xm.save. I haven't looked at the implementation of one_rank_only yet, so it might be a moot point. 😄

@sdesrozis sdesrozis mentioned this issue May 26, 2020
Merged
3 tasks
vfdev-5 added a commit that referenced this issue May 29, 2020
@sdesrozis @vfdev-5
Fix #1055 (#1068)
89e1358 
* issue_1055

* autopep8 fix

* decorate and refactor getattr

* remove decoration - need further discussions

* Added missing decorator for plx

* Added note about dist-friendly interface

* Updated Checkpoint to dist config + TPU

* autopep8 fix

* [WIP] Checkpoint in dist config

* autopep8 fix

* [WIP] Checkpoint in dist config

* autopep8 fix

* [WIP] Checkpoint on XLA

* autopep8 fix

* Fix checkpoint tests on XLA

* Put back Loggers as dist-unfriendly
+ tests for contrib savers

* Updated tests for XLA
- removed neptune xla tests

* autopep8 fix

* minor fix for coverage

* [WIP] New XLA tests for trains logger

* Fixed distrib tests for trains

Co-authored-by: Desroziers <sylvain.desroziers@ifpen.fr>
Co-authored-by: AutoPEP8 <>
Co-authored-by: vfdev-5 <vfdev.5@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@sdesrozis sdesrozis

Labels
enhancement
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@erip @sdesrozis