Skip to content

Grow DynamicScale after growth_interval finite steps, not one more - #5614

Open
vineethsaivs wants to merge 1 commit into
google:mainfrom
vineethsaivs:fix/dynamic-scale-growth-interval
Open

vineethsaivs wants to merge 1 commit into
google:mainfrom
vineethsaivs:fix/dynamic-scale-growth-interval

Conversation

@vineethsaivs

Copy link
Copy Markdown

DynamicScale grows the loss scale on the (growth_interval + 1)th finite step in a row, not after growth_interval finite steps as its docstring says. With growth_interval=1 the scale grows every other step instead of every step.

Cause: grow checks fin_steps == growth_interval, but fin_steps is the count before the current step, so it only reaches growth_interval one step later.

Fix: count the current step, fin_steps + 1 == growth_interval. This matches torch's GradScaler, which grows on the growth_intervalth successful step.

Test: new tests/dynamic_scale_test.py. With growth_interval=3 and scale=4, six finite steps now give 4, 4, 8, 8, 8, 16 (before: 4, 4, 4, 8, 8, 8), and a non-finite step still backs off and restarts the count. Both tests fail on main and pass here.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant