This repository was archived by the owner on Apr 23, 2025. It is now read-only.
Repository navigation
This repository was archived by the owner on Apr 23, 2025. It is now read-only.
GPT-2 currently exhausts all available GPU memory on an 8 GB GPU #673
Description
In testing PR #671, we noticed that the GPT-2 model now exhausts all available memory on 8 GB GPUs (example: GTX 1080) for both eager mode and X10 runtimes. It did not do this previously, so at some point the RAM usage of this model has increased to the point where it can no longer train on these GPUs.
We should investigate why this happened and see if memory usage for this model can be brought back down.
Metadata
Metadata
Assignees
Labels
No labels
Activity
xihui-wu commentedon Oct 5, 2020
As I tested an 16GB GPU VM, I do see that among the last nearly 2/3 of the epochs (10 epochs in total), a peak memory usage of 9187MB happens once in each epoch, and they happen around last training batch.
xihui-wu commentedon Nov 18, 2020
I just verified again on a new 16GB GPU DLVM instance created today, issue sustains.