REAM — Remote Expert Aggregation Merge

One-click REAM model compression on cloud GPUs via Vast.ai. Compress large MoE models (256 experts → 128) and quantize to GGUF for consumer hardware.

What It Does

1. Provisions a cloud GPU instance on Vast.ai (cheapest available ≥60 GB VRAM)
2. Sets up the REAM environment (dependencies, patches, llama.cpp)
3. Merges experts using the REAM algorithm (Samsung SAIL Montreal, arXiv:2604.04356)
4. Converts the merged model to GGUF (bf16)
5. Quantizes to consumer-friendly formats (Q3_K_S, Q4_K_M, etc.)
6. Downloads the final GGUF files to your local machine
7. Destroys the cloud instance to stop billing

Architecture

┌─────────────┐     ┌──────────────────────────────────┐     ┌──────────────┐
│  Flask UI   │────▶│  Vast.ai API (SSH + REST)        │────▶│  Cloud GPU   │
│  localhost   │     │  instance create/destroy/exec     │     │  REAM merge  │
│  :5000       │     │                                   │     │  GGUF convert│
└─────────────┘     └──────────────────────────────────┘     │  quantize    │
                                                              └──────┬───────┘
                                                                     │ SCP
                                                              ┌──────▼───────┐
                                                              │  Local disk  │
                                                              │  .gguf files │
                                                              └──────────────┘

Supported Models

Model	Total Params	Experts	Merge Target	VRAM Needed	Est. Time
Qwen3.6-35B-A3B	35B	256	128 (50%)	80 GB	~6 hrs
Qwen3-30B-A3B	30B	128	96 (25%)	64 GB	~5 hrs
Qwen3-235B-A22B	235B	128	96 (25%)	400 GB	~12 hrs
GLM-4.5-Air	94B	128	96 (25%)	160 GB	~8 hrs
Qwen3.5-122B-A10B	122B	128	96 (25%)	200 GB	~9 hrs

Quantized Output Sizes (Qwen3.6-35B-A3B → 128 experts)

Quant	Size	BPW	Fits	Quality
Q3_K_S	7.7 GB	3.54	RTX 5060 Ti 16GB	Good
Q4_K_M	10.8 GB	4.90	RTX 4070 Ti 16GB+	Better
Q5_K_M	14.5 GB	5.69	RTX 3090/4090 24GB	Best

Quick Start

Prerequisites

• Python 3.10+
• Vast.ai account with API key (get one here)
• SSH key added to Vast.ai (manage keys)

Install

git clone <this-repo>
cd REAM
pip install -r requirements.txt

Run

python app.py

Open http://localhost:5000/configure in your browser.

Configuration

1. Select model from the catalog
2. Enter Vast.ai API key (saved locally, never sent to third parties)
3. Search GPUs — the app finds cheapest ≥60 GB VRAM instances
4. Click Run — everything is automated from here

Cloud Setup Details (Vast.ai)

Docker Image

pytorch/pytorch:2.4.0-cuda12.4-cudnn9-runtime

Required Packages (installed on instance)

torch==2.9.0+cu128          # CRITICAL: pin exact version, torchvision upgrade breaks this
transformers (git)          # CRITICAL: must be git install, pip version lacks Qwen3.5 MoE support
scipy numpy datasets        # REAM dependencies
build-essential cmake       # llama.cpp compilation

Critical Patches Applied on Instance

1. transformers moe_utils.py patch (applied after git install):

# transformers 5.13 renamed input_embeds → inputs_embeds and removed cache_position
# This causes create_causal_mask() TypeError in REAM's moe_utils.py
sed -i 's/input_embeds/inputs_embeds/g' /workspace/ream/ream/moe_utils.py
sed -i '/cache_position/d' /workspace/ream/ream/moe_utils.py

2. torch version pin (prevents torchvision from upgrading torch):

pip install torch==2.9.0+cu128 --index-url https://download.pytorch.org/whl/cu128
# NEVER run: pip install torchvision  (this upgrades torch to 2.11 and breaks causal mask)

3. Package removals (conflict with REAM):

pip uninstall -y torchaudio torchelastic  # break REAM's model loading

REAM Merge Parameters (validated)

python merge.py \
  --model Qwen/Qwen3.6-35B-A3B \
  --merge_size 128 \
  --group_size 16 \
  --saliency reap \
  --merging logits+weights \
  --grouping ream \
  --dataset c4+math+code \
  --mix_ratio 0.0,0.3,0.7 \
  --batch_size 32 \        # 32 for 96GB GPU, 8 for 80GB
  --save_path /workspace/output/Qwen3.6-35B-A3B-REAM-128

GGUF Conversion

# CRITICAL: Always use bf16, never f16 (Qwen3.6 GDN layers overflow fp16)
python convert_hf_to_gguf.py --outtype bf16 /workspace/output/Qwen3.6-35B-A3B-REAM-128 \
  --outfile /workspace/output/Qwen3.6-35B-A3B-REAM-128-bf16.gguf

Post-Conversion GGUF Patching (automatic)

The workflow now automatically patches GGUF metadata after conversion and quantization. If you need to re-patch manually (e.g., GGUF files from another source):

python output/patch_gguf_nextn.py output/Qwen3.6-35B-A3B-REAM-128-Q4_K_M.gguf

Quantization

# Build llama.cpp on instance
apt-get install -y build-essential cmake
git clone https://github.com/ggerganov/llama.cpp /workspace/llama.cpp
cd /workspace/llama.cpp
cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j$(nproc) --target llama-quantize

# Quantize
./build/bin/llama-quantize /workspace/output/model-bf16.gguf \
  /workspace/output/model-Q4_K_M.gguf Q4_K_M

SSH Execution Notes

• Vast.ai instances use root user
• SSH port varies per instance (e.g., ssh9.vast.ai:30272)
• No nproc on some containers — use explicit -j 8 for make
• cmake may not be pre-installed — always install build-essential first
• Large file transfers: use scp -P <port> (not rsync, may not be installed)

GPU Pricing Reference (Vast.ai, June 2026)

GPU	VRAM	$/hr	REAM Time	Total Cost
RTX PRO 6000 Blackwell	96 GB	€0.70	~6 hrs	~€4.20
H100 SXM	80 GB	~$2.00	~3.5 hrs	~$7.00
A100 80GB	80 GB	~$1.00	~4.5 hrs	~$4.50
RTX 4090	24 GB	~$0.30	N/A (too small)	—

Project Structure

REAM/
├── app.py                  # Flask web application (UI + API routes)
├── vast_manager.py         # Vast.ai API integration (SSH, instance lifecycle)
├── ream_worker.py          # REAM merge, GGUF conversion, quantization
├── requirements.txt        # Python dependencies
├── templates/              # Flask HTML templates
│   ├── base.html
│   ├── configure.html      # Main workflow UI
│   ├── hardware.html
│   └── documentation.html
├── static/
│   ├── css/
│   └── js/app.js           # Frontend logic
├── output/                 # Downloaded GGUF files + tools
│   ├── patch_gguf.py       # Binary GGUF block_count fix
│   ├── patch_gguf_nextn.py # Binary GGUF nextn_predict_layers fix
│   └── inspect_gguf.py     # GGUF metadata inspector
├── logs/                   # Application + remote instance logs
├── tasks/                  # Task state persistence
└── docs/                   # Findings and integration notes

Known Issues

GGUF Metadata Mismatch

convert_hf_to_gguf.py (llama.cpp b9820) writes wrong values for Qwen3.6-35B-A3B:

• qwen35moe.block_count = 41 → should be 40
• qwen35moe.nextn_predict_layers = 1 → should be 0

Root cause: The converter counts the output/lm_head layer as an extra block, and the original Qwen3.6 config includes a draft model spec layer that REAM merge doesn't carry.

Current workaround: Binary patch the GGUF file with patch_gguf_nextn.py.

Proper fix (TODO): Patch config.json in the merged HF model before GGUF conversion, or use a newer llama.cpp that handles this gracefully.

torchvision Upgrade Trap

Running pip install torchvision on the instance upgrades torch from 2.9.0 to 2.11.0, which breaks create_causal_mask() — the cache_position parameter was removed. Always pin torch version explicitly.

Development

# Run with auto-reload
python app.py

# The app starts on http://localhost:5000
# Configure page: http://localhost:5000/configure

License

This tool uses the REAM method by Samsung SAIL Montreal. REAM is released under Apache 2.0. The original repository: https://github.com/SamsungSAILMontreal/ream