feat(activation_checkpointing): add `non_reentrant_checkpoint` to support inputs require no grad - #4118

Merged

tjruwase merged 27 commits into

deepspeedai:masterfrom

hughpu:feat/non-reentrant-checkpoint

Aug 29, 2023

Commits on Aug 7, 2023

feat: add `non_reentrant_checkpoint`
hughpu
committed
a20c79c
feat: add missing output postprocess and change the hook to record leaf forward tensor refs
hughpu
committed
8aeba5f
fix: make the multi_grad_hook registered after graph construction
hughpu
committed
ee04fa8

Commits on Aug 8, 2023

fix: backward compatibility for multi_tensor_hook
hughpu
committed
51f833d
fix: nonlocal reference error of deepspeed_saved_tensors
hughpu
committed
b29c1ef
fix: reduce repeating hook registration
hughpu
committed
37e7c23

Commits on Aug 9, 2023

Merge branch 'microsoft:master' into feat/non-reentrant-checkpoint
hughpu
authored
d7c5440
test: add test for `activation_checkpointing.checkpointing.non_reentrant_checkpoint`
hughpu
committed
e22c487
Pass correct node size for ZeRO++ (#4085)

authored and
hughpu
committed
4d2a274
add deepspeed chat arxiv report (#4110)

authored and
hughpu
committed
d4d070b
style: change flake8 detected style missmatch
hughpu
committed
aaf309e
test: hack to clone the `test_activation_checkpointing` module for reuse and add regression tests
hughpu
committed
a910922
doc: explain the introduction of `non_reentrant_checkpoint`
hughpu
committed
fc919b1
doc: explain the test of `non_reentrant_checkpoint`
hughpu
committed
b6a0a44
Merge branch 'microsoft:master' into feat/non-reentrant-checkpoint
hughpu
authored
8ec86a4
Merge branch 'master' into feat/non-reentrant-checkpoint
hughpu
authored
78c0d65

Commits on Aug 10, 2023

Merge branch 'master' into feat/non-reentrant-checkpoint
hughpu
authored
e4eff23

Commits on Aug 11, 2023

Merge branch 'master' into feat/non-reentrant-checkpoint
hughpu
authored
a6c7871

Commits on Aug 15, 2023

Merge branch 'master' into feat/non-reentrant-checkpoint
hughpu
authored
fbbb760

Commits on Aug 16, 2023

Commits on Aug 17, 2023

Merge branch 'master' into feat/non-reentrant-checkpoint
hughpu
authored
c17cc3d

Commits on Aug 18, 2023

Merge branch 'master' into feat/non-reentrant-checkpoint
hughpu
authored
a680399

Commits on Aug 19, 2023

Merge branch 'master' into feat/non-reentrant-checkpoint
hughpu
authored
13e766d

Commits on Aug 21, 2023

Merge branch 'master' into feat/non-reentrant-checkpoint
hughpu
authored
a46e326

Commits on Aug 23, 2023

Merge branch 'master' into feat/non-reentrant-checkpoint
hughpu
authored
b5c03f4

Commits on Aug 24, 2023

Merge branch 'master' into feat/non-reentrant-checkpoint
tjruwase
authored
13a026d