`sub_group_size` is not documented

[stas00]

besides 2 inconsistent examples:

./docs/_tutorials/zero.md:-    "sub_group_size": 1e9 
./docs/_tutorials/zero.md:+    "sub_group_size": 1e9, 
./docs/_pages/config-json.md:    "sub_group_size" : 1e12,

sub_group_size is not documented.

And it's unclear in what situations to raise or lower it and by how much.

Thank you!

@samyam

[stas00]

Elsewhere @samyam shared:

---

sub_group_size controls the granularity in which parameters are updated during optimizer steps. Parameters are grouped into buckets of sub_group_size and each buckets is updated one at a time. When used with NVMe offload in ZeRO-Infinity, sub_group_size therefore controls the granularity in which model states are moved in and out of CPU memory from NVMe during the optimizer step. This prevents running out of CPU memory for extremely large models.

Most users can leave sub_group_size to its default value of 1e9 when not using NVMe offload. The user may chose to change its value in the following cases:

1. Running into OOM during optimizer step : User can try reducing sub_group_size to reduce memory utilization of temporary buffers
2. Optimizer Step is taking a long time: User can try increasing sub_group_size to improve bandwidth utilization as a result of increased data size

---

I will add it for now into HF Deepspeed docs, as I don't think there is a place for tuning notes anywhere in core docs, but would love to see all those HF docs move here, as they belong to the core.