Proposal: allow arg.cpp to import/export configs from/to INI file

A while ago, we introduced the notion of "preset" in #10932 which was implemented using the existing arg.cpp infrastructure.

I'm now thinking about extending this into a new level. As the title said, I'm proposing adding an extension on top of arg.cpp that will allow import/export arguments from/to INI file

Use cases for this feature are:

Allow user to write their own preset, as suggested in arg : add model catalog #13385
Allow user to share their config more easily
Providing a way to specify per-model config in the newly added llama-server router mode server: introduce API for serving / loading / unloading multiple models #17470
Allow new CLI experience to save certain user preferences - just like how the "Settings" on web UI works; ref: cli: new CLI experience #17824

Why choosing ini?

It is simple to understand, simple to edit
It provides multiple "sections" - each section can be used as a new preset
The parser/writer can be simple - For example, this parser/writer is just ~700 LOC. We can do even better because we only use a subset of the language (only string value, no nested config)

Example of a config file:

(the config name is the env var name - already defined inside arg.cpp)

LLAMA_CONFIG_VERSION = 1

[gemma-4b-vision]
LLAMA_ARG_HF_REPO = ggml-org/gemma-3-4b-it-qat-GGUF
; this is a small model, I can offload everything to VRAM
LLAMA_ARG_N_GPU_LAYERS = 99999
LLAMA_ARG_N_CTX = 0
LLAMA_ARG_JINJA = true

[gemma-12b-vision]
LLAMA_ARG_HF_REPO = ggml-org/gemma-3-12b-it-qat-GGUF
; my system doesn't have enough VRAM, only offload some
LLAMA_ARG_N_GPU_LAYERS = 8
LLAMA_ARG_N_CTX = 0
LLAMA_ARG_JINJA = true

@ServeurpersoCom

CC @ServeurpersoCom and @ggerganov for discussion

We need to be able to customize the quantization as a parameter. A good GGUF collection has multiple quantizations: one for small context with high quality, one for large context but higher quantization. And everything else follows from that: --n-cpu-moe, --ctx-size, etc

With the PEG parser in common, you can parse an INI like this:

auto ini_parser = build_peg_parser([](auto & p) {
    // newline ::= "\r\n" / "\n" / "\r"
    auto newline = p.rule("newline", p.literal("\r\n") | p.literal("\n") | p.literal("\r"));

    // whitespace := [ \t]+
    auto ws = p.rule("whitespace", p.chars("[ \t]", 0, -1));

    // comment ::= [;#] (!newline .)*
    auto comment = p.rule("comment", p.chars("[;#]", 1, 1) + p.zero_or_more(p.negate(newline) + p.any()));

    // eol-start ::= ws ([;#] / newline / EOF)
    auto eol_start = p.rule("eol-start", ws + (p.chars("[;#]", 1, 1) | newline | p.end()));

    // eol ::= ws comment? (newline / EOF)
    auto eol = p.rule("eol", ws + p.optional(comment) + (newline | p.end()));

    // spacing ::= ([ \t\r\n] / comment)*
    auto spacing = p.rule("spacing", p.zero_or_more(p.chars("[ \t\r\n]", 1, 1) | comment));

    // value ::= (!eol-start .)*
    auto value = p.rule("value", p.zero_or_more(p.negate(eol_start) + p.any()));

    // ident ::= [a-zA-Z_] [a-zA-Z0-9_.-]*
    auto ident = p.rule("ident", p.chars("[a-zA-Z_]", 1, 1) + p.chars("[a-zA-Z0-9_.-]", 0, -1));

    // kv-pair ::= ident ws "=" ws value eol spacing
    auto kv_pair = p.rule("kv-pair", p.tag("key", ident) + ws + "=" + ws + p.tag("value", value) + eol + spacing);

    // header ::= "[" ws ident ws "]" eol spacing
    auto header = p.rule("header", "[" + ws + p.tag("section-name", ident) + ws + "]" + eol + spacing);

    // section ::= header kv-pair*
    auto section = p.rule("section", header + p.zero_or_more(kv_pair));

    // default-section ::= kv-pair+
    auto default_section = p.rule("default-section", p.one_or_more(kv_pair));

    // ini ::= spacing default-section? section* EOF
    auto ini = p.rule("ini", spacing + p.optional(default_section) + p.zero_or_more(section) + p.end());

    return ini;
});

Output

[default]
  LLAMA_CONFIG_VERSION = 1

[gemma-4b-vision]
  LLAMA_ARG_HF_REPO = ggml-org/gemma-3-4b-it-qat-GGUF
  LLAMA_ARG_JINJA = true
  LLAMA_ARG_N_CTX = 0
  LLAMA_ARG_N_GPU_LAYERS = 99999

[gemma-12b-vision]
  LLAMA_ARG_HF_REPO = ggml-org/gemma-3-12b-it-qat-GGUF
  LLAMA_ARG_JINJA = true
  LLAMA_ARG_N_CTX = 0
  LLAMA_ARG_N_GPU_LAYERS = 8

Complete Example

auto ini_parser = build_peg_parser([](auto & p) {
    // newline ::= "\r\n" / "\n" / "\r"
    auto newline = p.rule("newline", p.literal("\r\n") | p.literal("\n") | p.literal("\r"));

    // whitespace := [ \t]+
    auto ws = p.rule("whitespace", p.chars("[ \t]", 0, -1));

    // comment ::= [;#] (!newline .)*
    auto comment = p.rule("comment", p.chars("[;#]", 1, 1) + p.zero_or_more(p.negate(newline) + p.any()));

    // eol-start ::= ws ([;#] / newline / EOF)
    auto eol_start = p.rule("eol-start", ws + (p.chars("[;#]", 1, 1) | newline | p.end()));

    // eol ::= ws comment? (newline / EOF)
    auto eol = p.rule("eol", ws + p.optional(comment) + (newline | p.end()));

    // spacing ::= ([ \t\r\n] / comment)*
    auto spacing = p.rule("spacing", p.zero_or_more(p.chars("[ \t\r\n]", 1, 1) | comment));

    // value ::= (!eol-start .)*
    auto value = p.rule("value", p.zero_or_more(p.negate(eol_start) + p.any()));

    // ident ::= [a-zA-Z_] [a-zA-Z0-9_.-]*
    auto ident = p.rule("ident", p.chars("[a-zA-Z_]", 1, 1) + p.chars("[a-zA-Z0-9_.-]", 0, -1));

    // kv-pair ::= ident ws "=" ws value eol spacing
    auto kv_pair = p.rule("kv-pair", p.tag("key", ident) + ws + "=" + ws + p.tag("value", value) + eol + spacing);

    // header ::= "[" ws ident ws "]" eol spacing
    auto header = p.rule("header", "[" + ws + p.tag("section-name", ident) + ws + "]" + eol + spacing);

    // section ::= header kv-pair*
    auto section = p.rule("section", header + p.zero_or_more(kv_pair));

    // default-section ::= kv-pair+
    auto default_section = p.rule("default-section", p.one_or_more(kv_pair));

    // ini ::= spacing default-section? section* EOF
    auto ini = p.rule("ini", spacing + p.optional(default_section) + p.zero_or_more(section) + p.end());

    return ini;
});

common_peg_parse_context ctx(
    "LLAMA_CONFIG_VERSION = 1\n"
    "\n"
    "[gemma-4b-vision]\n"
    "LLAMA_ARG_HF_REPO = ggml-org/gemma-3-4b-it-qat-GGUF\n"
    "; this is a small model, I can offload everything to VRAM\n"
    "LLAMA_ARG_N_GPU_LAYERS = 99999\n"
    "LLAMA_ARG_N_CTX = 0 ; full context\n"
    "LLAMA_ARG_JINJA = true\n"
    "\n"
    "[gemma-12b-vision]\n"
    "LLAMA_ARG_HF_REPO = ggml-org/gemma-3-12b-it-qat-GGUF\n"
    "; my system doesn't have enough VRAM, only offload some\n"
    "LLAMA_ARG_N_GPU_LAYERS = 8\n"
    "LLAMA_ARG_N_CTX = 0\n"
    "LLAMA_ARG_JINJA = true\n"
);

// Parse and generate an AST
auto result = ini_parser.parse(ctx);
if (!t.assert_true(result.success())) {
    t.log("error at pos " + std::to_string(result.end) + ": " + ctx.input.substr(result.end));
}

// Visit AST to build config
struct model_config {
    std::string name;
    std::map<std::string, std::string> values;
};

std::vector<model_config> cfg;
cfg.push_back({"default", {}});

std::string current_key;

ctx.ast.visit(result, [&](const auto & node) {
    if (node.tag == "section-name") {
        cfg.push_back({std::string(node.text), {}});
    } else if (node.tag == "key") {
        current_key = std::string(node.text);
    } else if (node.tag == "value" && !current_key.empty()) {
        cfg.back().values[current_key] = std::string(node.text);
        current_key.clear();
    }
});

for (const auto & section : cfg) {
    if (section.name.empty()) {
        t.log("[default]");
    } else {
        t.log("[" + section.name + "]");
    }
    for (const auto & [key, value] : section.values) {
        t.log("  " + key + " = " + value);
    }
    t.log("");
}

Just including it as an option. Looping through lines with a series of regex might be less lines, although a quick draft from an LLM shows it slightly above (with superfluous comments of course).

One observation: as a user I'm more familiar with the arguments than the env vars. I assume env vars are easier to integrate with arg.cpp though. Maybe an alternative is to iterate through common_params_context.options and map argument names to their env counterpart, if it exists? You can trim leading dashes - to make it even cleaner.

So the config then becomes something like:

version = 1

[gemma-4b-vision]
hf = ggml-org/gemma-3-4b-it-qat-GGUF
; this is a small model, I can offload everything to VRAM
n-gpu-layers = 99999
ctx-size = 0
jinja = true

[gemma-12b-vision]
hf = ggml-org/gemma-3-12b-it-qat-GGUF
; my system doesn't have enough VRAM, only offload some
n-gpu-layers = 8
ctx-size = 0
jinja = true

This is exactly what I needed, because I had already started with a proof of concept in JSON, whereas INI is much more practical; you can even add comments!

With the PEG parser in common, you can parse an INI like this:

That's exactly what was needed!

The proposal sounds good. I'll just give my primary use case here as an example of what would be useful to me:

Need to run 4 models
- One for chat/agents (gpt-oss)
- One for vision tasks (Qwen3 VL)
- One for FIM (Qwen2.5/3) with cache reuse (--cache-reuse 256), draft model (-md), custom batch size (-ub) and slots (-np)
- One for embeddings (Gemma) with (--embeddings)

If I can configure all this in a single config file and then simply start:

llama-server --config mycfg.ini

This would be great.

Definitely INI over JSON.

I write a POC for multi-model inference servers targeting small/medium teams. The goal is declarative, user-friendly configuration with zero operational friction. I uses the existing PEG parser and common_arg_get_env_name() API to ensure compatibility with llama.cpp's arg system. Final design should match your vision for production multi-tenant inference

Auto-generated config.ini: Created at /config.ini on first run, one [vendor/model] section per discovered model (HF-style layout)
CLI to INI templating: All router flags (except blacklist: --port, -m, etc.) are converted to LLAMA_ARG_* env var names and injected as initial config template for each model
Standard INI format: Booleans stored as =true, regular values as =value, users can override with =false for explicit opt-out
Config priority over CLI: Existing user modifications in INI are preserved (never overwritten by new CLI args)
Hot-sync new args: When operators add CLI flags to the router, they're automatically synced to all model sections in INI (if not already present)
Env var passthrough: Child processes receive config as LLAMA_ARG_*= environment variables (empty for bools), respecting llama.cpp's native conventions
Per-model customization: Operators edit INI to override any parameter per model (e.g., desired qwant, --n-cpu-moe and --ctx-size parameters)

With this system, you can override whatever you want per model, to always have each model configuration optimized for your GPU, and to reset you just delete the ini file! And it allows beginners to discover the arguments of llama.cpp as they go along ! + future administration via the GUI.
Adding custom path customization to a mycfg.ini file poses no problem.

WDYT ?

It works for me. It discovers my entire collection of GGUF files, including mmproj files (by default, via folders BF16 > F16 > F32). I have the same use case; I have an embedding model to vectorize a large video description dataset. All with the same llama-server in router mode.

@ServeurpersoCom

Nice works @ServeurpersoCom ! I'll let you finish the implementation and push the PR

@aldehir

One observation: as a user I'm more familiar with the arguments than the env vars. I assume env vars are easier to integrate with arg.cpp though. Maybe an alternative is to iterate through common_params_context.options and map argument names to their env counterpart, if it exists? You can trim leading dashes - to make it even cleaner.

@aldehir Yeah I think we can have both. I'm choosing env var because it's a bit easier to decide when doing implementing the writer (because one arg can have both the short -a and long form --abc)

But I think we can just support both longest arg + env

One observation: as a user I'm more familiar with the arguments than the env vars. I assume env vars are easier to integrate with arg.cpp though. Maybe an alternative is to iterate through common_params_context.options and map argument names to their env counterpart, if it exists? You can trim leading dashes - to make it even cleaner.

@aldehir Yeah I think we can have both. I'm choosing env var because it's a bit easier to decide when doing implementing the writer (because one arg can have both the short -a and long form --abc)

But I think we can just support both longest arg + env

Env vars have another benefit: they prevent shell injection attacks from user-edited INI values, making GUI administration safe by design !

Env vars have another benefit: they prevent shell injection attacks from user-edited INI values, making GUI administration safe by design !

Right, I wasn't advocating for passing by args, but rather mapping the arg name to the appropriate env var. Those associations are already made in common_arg.

I'm in team "long-arg + ENV support", but actually kudos for the INI approach, I'm kind of fed up with people using JSON for everything :)

@ngxson

This, together with @ngxson 's #13385 will be huge.

I ported over the config I used for llama-swap and this seems to be working well as a replacement. One question: is the router able to swap models into and out of VRAM as needed? I currently run into cudaMalloc failed: out of memory once I load enough models.

I ported over the config I used for llama-swap and this seems to be working well as a replacement. One question: is the router able to swap models into and out of VRAM as needed? I currently run into cudaMalloc failed: out of memory once I load enough models.

The router uses LRU (Least Recently Used) eviction based on model count, not memory detection. Use "--models-max N" to limit concurrent models: when you hit N models, the least recently used one unloads.
For a default llama-swap behavior, use "--models-max 1". Note: there is no "model group" or memory estimator for now

Navigation Menu

Repository navigation

Proposal: allow arg.cpp to import/export configs from/to INI file #17850

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Proposal: allow arg.cpp to import/export configs from/to INI file #17850

Description

Activity

ngxson commented on Dec 7, 2025

ServeurpersoCom commented on Dec 7, 2025

aldehir commented on Dec 7, 2025

ServeurpersoCom commented on Dec 8, 2025

ggerganov commented on Dec 8, 2025

ServeurpersoCom commented on Dec 8, 2025

ServeurpersoCom commented on Dec 8, 2025

ngxson commented on Dec 8, 2025

ngxson commented on Dec 8, 2025

ServeurpersoCom commented on Dec 8, 2025

aldehir commented on Dec 8, 2025

pwilkin commented on Dec 8, 2025

pwilkin commented on Dec 8, 2025

dwrz commented on Dec 11, 2025

ServeurpersoCom commented on Dec 11, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions