Skip to content

setup: UD-Q4_K_XL / UD-IQ4_XS keep their RAM budget on a layer split (#642) #1564

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
ShevchenkoVadim wants to merge 1 commit into from
+114 −17
Closed

setup: UD-Q4_K_XL / UD-IQ4_XS keep their RAM budget on a layer split (#642) #1564

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion docs/MULTI_GPU.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -113,7 +113,8 @@ leaves out the ones every card's cache holds, not only the first card's, and whe
the hottest by the expert profile over all the layers. An adaptive swap copies the evicted expert back into RAM from
the card that owns its layer. Before, `--resident-experts` with a split ran as `--mmap-experts`, and setup recommended
one card in the low-RAM mode; with an engine that has it (`RESIDENT_SPLIT_ENGINE` in setup.py), setup keeps the cards
together and the experts no card holds in RAM. Swift 1.5 IQ3_XXS at
together and the experts no card holds in RAM. The same copy, sized by a RAM budget (`--resident-budget-gib`, the
Unsloth UD-Q4_K_XL and UD-IQ4_XS configs), is kept on a split as well ([UNSLOTH_Q4.md](UNSLOTH_Q4.md)). Swift 1.5 IQ3_XXS at
160K (q4_0 KV, `--prefill 4096`, `--spec 4` with the stock draft layer), RTX 4060 Ti (layers 0-19) + RTX 5080 (20-47),
i9-14900KF, 32 GB of RAM, Windows 11, four greedy prompts at a time, decode tok/s:

Expand Down
13 changes: 10 additions & 3 deletions docs/UNSLOTH_Q4.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,9 +26,16 @@ What setup does differently for this model:
engine 0.1.32 or newer (checked before anything is downloaded), and an NVIDIA
GPU: it has not been run on AMD cards (its prompt kernels for the Q4_K / Q5_K experts are NVIDIA-only), so with
`--backend hip` setup says so and asks before the download (#429; `--model UD-Q4_K_XL --yes` tries it). One GPU
by default: the RAM budget below has no layer split (the engine refuses `--resident-budget-gib` with one). No
by default (the tested setup); on several, see the next point. No
experimental speed projection (not tested with it). Images (#967) are an option, with a warning that this file is untested with them (reported working, #971).
- Several GPUs (#498): when the RAM holds the GGUF files and 24 GB more (~135 GB of RAM) and two or more cards can
- Several GPUs, with an engine that keeps the resident RAM copy on a layer split (0.1.40 and newer, #642): the RAM
budget is kept on the split. Each card caches the experts of its own layers, and the budget holds the hottest of
the rest by the whole expert profile (the experts no card holds), so any RAM works as on one card. Setup asks (one
GPU stays the default; `--gpus 0,1`, "2" at setup or "y" when a start offers both cards takes the split), and a
start on several GPUs keeps `--resident-budget-gib`. Measured on an RTX 3090 (x16) + RTX 3080 Ti (x4), 91 GB of RAM,
budget 64 GiB, engine 0.1.41: decode 45 tok/s against 38 on the 3090 alone (the auto split gives the x4 card two
layers); 57.6 GiB of experts in RAM, no file reads per request.
- Several GPUs with an older engine (#498): when the RAM holds the GGUF files and 24 GB more (~135 GB of RAM) and two or more cards can
share it, setup asks (one GPU stays the default; `--gpus 0,1` takes the split). The split runs **without** the RAM budget: all 77 GB of experts are loaded into RAM from the GGUFs at
start, the files pass through the OS file cache while they load, and the config gets `"gpu": [0, 1]` and
`"layer_split": "auto"`. Measured on 2x RTX 3090 with 165 GiB (#498): decode 31 tok/s on one card with the budget,
Expand Down Expand Up @@ -243,7 +250,7 @@ PLE table, a Q6_K head) is UD-Q4_K_XL's. Three shards, 93.7 GB:

Setup treats it like UD-Q4_K_XL (the list above): the same RAM budget (your RAM less 24 GB; at most all 55 GiB of its
experts, so a PC with ~80 GB of RAM or more holds all of them), the pack with `--compat-bf16`, no `experts.bin`, one
GPU by default. It does not ask on AMD: its experts' formats have prompt kernels there too. Images are an option (asked,
GPU by default (several keep the budget, as above). It does not ask on AMD: its experts' formats have prompt kernels there too. Images are an option (asked,
off by default; NVIDIA): the image path has no restriction for this pack, which uses the original model's image
encoder; not yet run with images.

Expand Down
50 changes: 39 additions & 11 deletions setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1100,13 +1100,15 @@ def unsloth_split_need_gb(model="UD-Q4_K_XL") -> float:


def split_budget(cfg: dict, yes: bool = False, explicit: bool = False) -> bool:
"""#498: a UD-Q4_K_XL config (its RAM budget, --resident-budget-gib) started on several GPUs. The engine refuses
"""#498: a UD-Q4_K_XL config (its RAM budget, --resident-budget-gib) started on several GPUs. The engine refused
the budget with a layer split (it exited with code 2), so the split runs without it - all the experts loaded into
RAM at start - where the RAM holds the GGUFs and 24 GB more; else setup says so and asks (#737: a recommendation,
not a wall - 128 GB ran it fine): a "no" stops, before the config is saved; `explicit` (--gpus) with --yes goes on.
True when the config changed."""
#642: the engines from RESIDENT_SPLIT_ENGINE keep the resident RAM copy on a split - every card's cache left out
of it, the rest ranked by the whole expert profile - and the budget is that copy's size, so it is kept, as
split_mmap keeps --resident-experts. True when the config changed."""
a = cfg.get("args", [])
if "--resident-budget-gib" not in a:
if "--resident-budget-gib" not in a or resident_split():
return False
model = budget_model(cfg)
need, ram = unsloth_split_need_gb(model), ram_gb()
Expand Down Expand Up @@ -1174,9 +1176,11 @@ def offer_together(cfg_path: Path, cfg: dict, yes: bool) -> dict:
can = together_ok(found)
if not can:
return cfg
# #498: UD-Q4_K_XL's RAM budget has no layer split; without it the RAM must hold the GGUFs and 24 GB more
# #498: UD-Q4_K_XL's RAM budget had no layer split; without it the RAM must hold the GGUFs and 24 GB more. #642:
# the engines from RESIDENT_SPLIT_ENGINE keep the budget on both cards, so any RAM is offered them (one GPU stays
# the recommendation, the tested setup)
budget = "--resident-budget-gib" in cfg.get("args", [])
if budget and ram_gb() < unsloth_split_need_gb(budget_model(cfg)):
if budget and not resident_split() and ram_gb() < unsloth_split_need_gb(budget_model(cfg)):
return cfg
pair = can[:2]
cfg["gpus_asked"] = True
Expand All @@ -1190,7 +1194,11 @@ def offer_together(cfg_path: Path, cfg: dict, yes: bool) -> dict:
say(" This model runs in the low-RAM mode with its experts kept in RAM, on one GPU (recommended: steady RAM")
say(" use). On both, the experts the GPUs do not hold are read through the OS file cache instead: faster in")
say(" two reports (#364, #384), but RAM can fill up to 0 free during long prompts.")
if budget:
if budget and resident_split():
say(f" This model ({budget_model(cfg)}) runs on one GPU with a RAM budget of its experts (recommended: the tested")
say(" setup). On both it keeps the budget: each card caches the experts of its own layers, and the RAM holds")
say(" the hottest of the rest (#642).")
elif budget:
say(f" This model ({budget_model(cfg)}) runs on one GPU with a RAM budget of its experts (recommended: the tested")
say(" setup). On both it has no budget: all its experts are loaded into RAM at start, which this PC's RAM")
say(" holds - about twice as fast in #498 (2x RTX 3090: 31 -> 64-78 tokens/s).")
Expand Down Expand Up @@ -3386,8 +3394,27 @@ def unsloth_together(a, model, ram, gpu, chosen) -> bool:
split runs without it: all its experts loaded into RAM from the GGUFs at start, as with the 2-3-bit models - only
where the RAM holds the GGUF files and 24 GB more (unsloth_split_need_gb; 165 GiB, 2x RTX 3090: 31 -> 64-78
tok/s). An explicit --gpus is honoured there; otherwise asked, one GPU by default (--yes: one GPU, as before); an
explicit --resident-budget-gib keeps one GPU. True: all of them."""
explicit --resident-budget-gib keeps one GPU. #642: the engines from RESIDENT_SPLIT_ENGINE keep the budget on a
split (the experts no card holds, the hottest by the whole profile), so there the cards go together with it, at
any RAM: --gpus, or "2" when asked; one GPU stays the recommendation (--yes alone: one GPU, as before). True: all
of them."""
names = " + ".join(gpu_name(g) for g in chosen)
if resident_split():
note = "with its RAM budget (each card caches its own layers' experts, the RAM the hottest of the rest; #642)"
if a.gpus:
ok(f"{model} on {names}, as you chose (--gpus), {note}")
return True
if not a.yes:
say()
say(f" {model} runs with a RAM budget of its experts, on one GPU or on {names} together:")
say(f" 1) {gpu_name(gpu)} only (recommended: the tested setup)")
say(f" 2) {names} together: each card caches the experts of its own layers (#642)")
if ask(f"{model}: which GPUs?", ["1", "2"], "1", a.yes) == "2":
ok(f"{model} on {names}, {note}")
return True
warn(f"{model} runs on one GPU: using {gpu_name(gpu)} only (--gpus " + ",".join(str(g["index"]) for g in chosen)
+ f" shares it across {names}, with its RAM budget)")
return False
need = unsloth_split_need_gb(model)
if ram < need:
note = (f"{model} on several GPUs has no RAM budget and needs ~{need:.0f} GB of RAM (its GGUF files and "
Expand Down Expand Up @@ -4990,8 +5017,9 @@ def main() -> int:
budget, q4_split = None, False
if MODELS[model].get("budget"):
# Unsloth's UD-Q4_K_XL: a RAM budget of experts, the rest from the GGUF on the SSD - not the low-RAM mode (no
# experts.bin: it would be another 77 GB on the disk), and one GPU (the budget mode has no layer split) unless
# the RAM holds the GGUFs and 24 GB more: then several, without the budget, if asked for (#498)
# experts.bin: it would be another 77 GB on the disk), and one GPU (the budget mode had no layer split) unless
# the RAM holds the GGUFs and 24 GB more: then several, without the budget, if asked for (#498). #642: from
# RESIDENT_SPLIT_ENGINE several GPUs, when asked for, keep the budget (unsloth_together)
if MODELS[model].get("experimental"):
warn(f"{model} is EXPERIMENTAL (docs/UNSLOTH_Q4.md): most of its experts are read from the SSD while it "
"answers, so it is several times slower than the 2-3-bit models; quality checked against llama.cpp")
Expand All @@ -5014,7 +5042,7 @@ def main() -> int:
budget = budget_choice(model, ram, a.resident_budget_gib)
if multi and not unsloth_together(a, model, ram, gpu, chosen):
multi, sel, chosen = [], [gpu["index"]], [gpu]
q4_split = bool(multi) # #498: on several GPUs without the RAM budget
q4_split = bool(multi) and not resident_split() # #498: on several GPUs without the RAM budget (#642: kept)
if not q4_split:
ok(f"RAM budget: {budget:g} GiB of {model}'s experts in RAM, the rest read from the model files on the SSD")
if a.low_ram not in ("auto", "off"):
Expand Down Expand Up @@ -5411,7 +5439,7 @@ def main() -> int:
elif a.kv_streaming == "on":
warn("--kv-streaming on: a context under 64K is not streamed (the attention's window holds all of it): off")
if budget is not None and not q4_split: # UD-Q4_K_XL: the experts read from the GGUF in place, the most-used N
args += ["--resident-budget-gib", f"{budget:g}"] # GiB kept in RAM (#498: a layer split has no budget)
args += ["--resident-budget-gib", f"{budget:g}"] # GiB kept in RAM (#498: a layer split had no budget)
if vision != "none":
args += ["--vision", "--vram-reserve-mib", str(VISION[vision]["reserve_mib"])]
if vision == "gpu" and a.vram_reserve_mib is None and 0 < gpu.get("vram_gb", 0.0) <= 12.5:
Expand Down
65 changes: 63 additions & 2 deletions tools/test_setup_unsloth.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -335,7 +335,7 @@ def test_one_gpu_and_images_with_a_warning(self):
"""#967: images are allowed with UD-Q4_K_XL, said to be untested."""
code, out, cfg = self.main(["--context", "8192", "--vision", "yes", "--low-ram", "on"], n_gpus=2)
self.assertEqual(code, 0, out)
self.assertIn("(NVIDIA GeForce RTX 5070, 12 GB) only (--gpus 0,1 uses them together anyway)", out)
self.assertIn("(NVIDIA GeForce RTX 5070, 12 GB) only (--gpus 0,1 shares it across", out)
self.assertNotIn("images are not available", out)
self.assertIn("images: on", out)
self.assertIn("images with UD-Q4_K_XL are untested", out)
Expand All @@ -346,6 +346,7 @@ def test_one_gpu_and_images_with_a_warning(self):
self.assertNotIn("--mmap-experts", cfg["args"])
self.assertFalse(any("--experts-bin" in r for r in self.runs))

@mock.patch.object(setup, "MIN_ENGINE", (0, 1, 39)) # before RESIDENT_SPLIT_ENGINE: no budget on a split
def test_split_when_the_ram_holds_it(self):
# #498: 165 GiB and --gpus 0,1: the split without the RAM budget (the engine refuses the pair)
code, out, cfg = self.main(["--context", "8192", "--gpus", "0,1"], n_gpus=2, ram=165.0)
Expand All @@ -358,6 +359,7 @@ def test_split_when_the_ram_holds_it(self):
self.assertIn("OS file cache", out)
self.assertNotIn("RAM budget: ", out)

@mock.patch.object(setup, "MIN_ENGINE", (0, 1, 39)) # before RESIDENT_SPLIT_ENGINE: no budget on a split
def test_split_without_the_ram_is_a_warning_not_a_wall(self):
# #737: below the worst-case estimate (a 128 GB PC ran it) the split is still the user's call
need = setup.unsloth_split_need_gb()
Expand All @@ -374,6 +376,7 @@ def test_split_without_the_ram_is_a_warning_not_a_wall(self):
self.assertEqual(cfg["gpu"], [0, 1])
self.assertNotIn("--resident-budget-gib", cfg["args"])

@mock.patch.object(setup, "MIN_ENGINE", (0, 1, 39)) # before RESIDENT_SPLIT_ENGINE: no budget on a split
def test_an_explicit_budget_keeps_one_gpu(self):
code, out, cfg = self.main(["--context", "8192", "--gpus", "0,1", "--resident-budget-gib", "60"], n_gpus=2,
ram=165.0)
Expand All @@ -382,6 +385,38 @@ def test_an_explicit_budget_keeps_one_gpu(self):
self.assertNotIn("layer_split", cfg)
self.assertEqual(cfg["args"][cfg["args"].index("--resident-budget-gib") + 1], "60")

def test_split_keeps_the_budget(self):
"""#642: with a resident-split engine the cards asked for keep the RAM budget, at any RAM."""
with mock.patch.object(setup, "MIN_ENGINE", setup.RESIDENT_SPLIT_ENGINE):
code, out, cfg = self.main(["--context", "8192", "--gpus", "0,1"], n_gpus=2) # 64 GB: no 135 GB asked
self.assertEqual(code, 0, out)
self.assertEqual(cfg["gpu"], [0, 1])
self.assertEqual(cfg["layer_split"], "auto")
self.assertEqual(cfg["args"][cfg["args"].index("--resident-budget-gib") + 1], "40")
for flag in ("--mmap-experts", "--resident-experts"):
self.assertNotIn(flag, cfg["args"])
self.assertIn("as you chose (--gpus), with its RAM budget", out)
self.assertIn("RAM budget: 40 GiB", out)
self.assertNotIn("worst case", out)
code, out, cfg = self.main(["--context", "8192", "--gpus", "0,1", "--resident-budget-gib", "30"], n_gpus=2)
self.assertEqual(code, 0, out) # an explicit budget too
self.assertEqual(cfg["gpu"], [0, 1])
self.assertEqual(cfg["args"][cfg["args"].index("--resident-budget-gib") + 1], "30")
self.assertNotIn("has no layer split", out)

def test_asked_with_the_budget_one_gpu_by_default(self):
"""#642: asked as before (one GPU recommended); "2" takes both cards with the budget."""
with mock.patch.object(setup, "MIN_ENGINE", setup.RESIDENT_SPLIT_ENGINE):
code, out, cfg = self.main(["--context", "8192"], n_gpus=2, answers={}) # Enter: one GPU
self.assertEqual(code, 0, out)
self.assertTrue(any(f"{M}: which GPUs?" in q and "[1]" in q for q in self.asked), self.asked)
self.assertNotIn("layer_split", cfg)
self.assertIn("--resident-budget-gib", cfg["args"])
code, out, cfg = self.main(["--context", "8192"], n_gpus=2, answers={"which GPUs?": "2"})
self.assertEqual(code, 0, out)
self.assertEqual(cfg["gpu"], [0, 1])
self.assertIn("--resident-budget-gib", cfg["args"])

def test_yes_alone_keeps_one_gpu(self):
code, out, cfg = self.main(["--context", "8192"], n_gpus=2, ram=165.0)
self.assertEqual(code, 0, out)
Expand Down Expand Up @@ -523,9 +558,15 @@ def test_amd_is_not_asked(self):

class LayerSplit(unittest.TestCase):
"""#498: UD-Q4_K_XL across GPUs - asked at setup (one GPU by default), and a start with --gpus no longer keeps
the RAM budget the engine refuses with a split (it exited with code 2)."""
the RAM budget the engine refused with a split (it exited with code 2). #642: from RESIDENT_SPLIT_ENGINE the
budget is kept on the split."""

def setUp(self):
# the behaviour before RESIDENT_SPLIT_ENGINE (an engine from before 0.1.40); the tests of the newer engine
# patch MIN_ENGINE themselves
p = mock.patch.object(setup, "MIN_ENGINE", (0, 1, 39))
p.start()
self.addCleanup(p.stop)
sys.path.insert(0, str(ROOT / "tools"))
from test_setup_golden import card
self.found = [card(i, "NVIDIA GeForce RTX 3090", 24.0, "86") for i in range(2)]
Expand Down Expand Up @@ -601,6 +642,26 @@ def test_offered_at_a_start(self):
self.assertEqual(cfg["gpu"], 0)
self.assertNotIn("Use both", out)

def test_resident_split_engine_keeps_the_budget(self):
"""#642: a start with --gpus keeps the budget at any RAM; offered both cards (Enter: one, as before)."""
with mock.patch.object(setup, "MIN_ENGINE", setup.RESIDENT_SPLIT_ENGINE):
for ram in (165.0, 63.7):
code, out, cfg, started = self.start(ram)
self.assertIsNone(code, out)
self.assertTrue(started)
self.assertEqual(cfg["gpu"], [0, 1])
self.assertEqual(cfg["args"], ["--pack", "p", "--resident-budget-gib", "40", "--kv", "int8"])
self.assertNotIn("no RAM budget", out)
self.assertNotIn("cannot share its RAM budget", out)
code, out, cfg, _ = self.start(63.7, gpu=None, offered="") # offered at 64 GB too; Enter: one GPU
self.assertIsNone(code, out)
self.assertIn("On both it keeps the budget", out)
self.assertIn("staying on one GPU", out)
self.assertEqual(cfg["gpu"], 0)
code, out, cfg, _ = self.start(63.7, gpu=None, offered="y")
self.assertEqual(cfg["gpu"], [0, 1])
self.assertIn("--resident-budget-gib", cfg["args"])


if __name__ == "__main__":
unittest.main()