[RFC] Online quantization knowledge, skills and Skill Indexer selection #254

Current state: Proposed since 2026-10-29 with no linked PR; implementation and goal-acceptance owners are TBD, and the one review comment (2026-10-29) proposing Slice C acceptance cases has no author reply.
Decision needed: The initial online-quantization scope, the first model/engine/backend and skill set, the Skill Indexer's discovery role, and where evaluation criteria are frozen, from the maintainers.
Next action: The maintainers record those decisions and designate an acceptance owner; no implementer is assigned.

Status: Proposed for discussion. Prepared on 2026-10-29.
Governing implementation Issue and acceptance owner: TBD.

Decision requested

Add a small, source-grounded online quantization knowledge and skill set, and
define when the existing Skill Indexer should surface it. Focus on converting
weights during engine initialization, quantizing activations during forward
execution, and quantizing KV as the cache is populated and consumed. Separate
finding relevant guidance from selecting and executing an experiment.

The recommended first delivery combines reusable diagnosis and evaluation
procedures with one engine-owned online quantization procedure. Reuse collect and the
whole-resource selection delivered by RFC #128.
The host agent remains responsible for checking the task contract, choosing
experiments and acting within the user's authority.

This proposal defines knowledge and selection behavior. Automatic search
execution, new quantization kernels and performance claims require their own
accepted scope and evidence.

Meaning of online quantization

Weights converted once during initialization are not assumed to be re-quantized
for every request. Online conversion and dynamic scale estimation are separate
properties. A runtime KV path may use fixed, calibrated or dynamically computed
scales; each source must state which behavior it implements.

The vLLM online quantization documentation
describes load-time weight conversion from BF16/FP16 without a pre-quantized
checkpoint or calibration dataset, with scheme-dependent activation behavior.
This is an initial implementation reference, not a claim that every online
scheme is calibration-free or supported on every backend.

PTQ describes when quantization is introduced relative to training; online
describes when conversion occurs relative to loading and inference. Online
quantization can therefore be PTQ. Offline AWQ/GPTQ checkpoint production,
model calibration/reconstruction, checkpoint distribution and QAT belong to the
model-delivery workflow, outside this inference-engine optimization scope.
Serving an already quantized checkpoint alone does not satisfy
the online weight-conversion scope. Automatic per-request precision switching
and learning from live traffic are also separate proposals.

Model delivery and inference-engine responsibilities

For this RFC, offline quantized weights are a versioned model input. Producing
and accepting that artifact is model-delivery work. The inference engine owns
correct and efficient execution of the declared input and its runtime contract.
This boundary follows the changed artifact and behavior, even when both tools
ship in the same inference ecosystem.

The LLM Compressor deployment workflow
provides a concrete interface example: a compressed model is prepared before
deployment with vLLM. Supporting that model format is engine work; preparing a
different model artifact is a different delivery outcome.

Consequently, replacing the original checkpoint with offline INT4 weights must
be reported as a model/deployment comparison. It cannot, by itself, support a
claim that the inference framework became faster. An engine comparison holds
the model artifact and effective numerical contract fixed while changing the
engine. A quality-constrained online comparison may hold the original checkpoint
fixed and change runtime precision, but must identify that approximation in the
claim; equal source-checkpoint bytes do not establish equal effective precision.
If both model artifact and engine change, retain separate controlled comparisons
before attributing their effects.

This responsibility boundary is distinct from SIAS plane selection. Choosing a
different quantization/precision policy changes the effective Model contract,
including when the conversion is implemented online. Optimizing its runtime
implementation under a fixed numerical contract is an Infra change. Declaring
online conversion an engine feature does not permit mixing both planes in one
campaign step or bypassing the model-quality owner.

Problem and existing coverage

The inspected source snapshot contains 79 SKILL.md files. A broad scan for
quantization, algorithm names and low-precision dtypes matches 24 bodies, including
incidental mentions and explicit exclusions. Two catalog records focus directly
on quantization:

No dedicated online quantization SKILL.md was found. The existing H3 and KV
records are useful starting points for runtime mechanisms. The gap is a complete
procedure connecting bottleneck diagnosis, conversion/scale lifetime, effective
kernel dispatch, quality evaluation and end-to-end measurements.

These counts describe the inspected local checkout, not today's upstream main.
There is also a software-version difference:

RFC #128 accepted its bounded software/evidence delivery without accepting a
lower-cost, preserved-quality claim. This proposal adds quantization cases to
that work; it does not reopen its closeout or assume skill-selection efficacy.

Issue #220 reports an
exploratory failure mode relevant here: quantization material filed under a
diffusion owner was missed for an audio weight-traffic problem. It also reports
discarding procedures because a profiling tool was unavailable. These are
hypotheses for selection cases; the issue's exploratory setup and unavailable
raw evidence do not establish general effectiveness or authorize new runs.

Knowledge and skill additions

Organize by mechanism and preserve source ownership

Use the existing knowledge-base contract. Repository and
framework owners retain implementation facts; generic methods explain how to
reason across implementations. A reference to another owner permits investigation,
not transfer of its compatibility or validation status.

Keep algorithm, representation, artifact format and execution backend distinct.
INT4/FP8 describe representations with additional scheme details; an online
conversion setting does not identify the kernel that actually executes.
Weight-only, weight-and-activation, KV and transport quantization address different
bytes and quality contracts. Communication compression is a separate candidate;
quantizing linear weights does not by itself reduce collective traffic.

Seed the source review from the official
vLLM online quantization guide,
vLLM KV-cache contracts and
torchao inference workflows,
which distinguish weight-only and dynamic activation configurations. Prioritize
actual load/forward/cache code paths and the existing H3 source readsets over
offline checkpoint-production recipes. Record exact
implementation revisions and license/readset information during acquisition.
These sources identify candidate mechanisms, not an accuracy or speed ranking
for an InferRSI workload.

Deliver a bounded initial skill set

The following paths and IDs are proposed, not registered capabilities:

Reuse performance-ceiling-analysis, optimization-search-policy,
inference-benchmark-design, the existing evidence workflow and model validators.
The evaluation method should specify quantization-specific inputs and checks,
while linking common benchmarking and claim procedures instead of copying them.
Keep new library bodies and their resources in English.

Begin with one declared model/engine/backend combination, chosen in the
implementation Issue. Extend the H3/vLLM-Omni owner when the target task is a
repeated audio/video projection, and use torchao-owned guidance when its runtime
path fits the task. Supported engine configuration and a custom quantized kernel
are separate delivery slices; neither requires an offline conversion pipeline
as a prerequisite. Training, offline conversion and adaptive serving policies
remain separately scoped.

Record applicability and evidence once

Use existing JSON and skill sidecars for descriptions, tags, applicability,
sources, limitations and verification. Add mechanism-oriented discovery terms
such as weight bandwidth, model residency, online quantization, load-time conversion, dynamic activation, KV capacity, scale lifetime, activation outliers and quality-sensitive. Relevant English descriptions should
explain the trigger in ordinary language. Evaluate Chinese task prompts as well;
the caller can supply equivalent English mechanism cues without adding another
alias registry. Avoid keyword stuffing or making every skill a quantization hit.

Each acquired entry must preserve:

• Source identity, version/readset, attribution and license; paper versions and
  implementation commits remain distinct provenance.
• Model/component, engine/backend/device, scheme/granularity, scales/zero points,
  packing/layout, accumulator, shape and parallelism constraints; conversion timing,
  scale origin/update scope, source-buffer release, explicit fallback behavior and
  the evidence needed to confirm actual dispatch.
• Required initialization, calibration if applicable and evaluation assumptions,
  sensitive modules, negative results, failure modes and a minimal verification plan.
• Separate paper/upstream observations, source-derived guidance and qualified
  local experiments; import or green CPU CI does not promote validation.

Do not create a parallel quantization registry or duplicate per-record summaries.
Place source-specific records in their existing contribution/dependency/peer
owners. A new knowledge-only source does not require an executable repository
adapter. Register approved skills in skill-index.json, then regenerate projections
from committed source using the existing generator. Update the two matrices to
describe actual coverage and its evidence limits; do not add a third matrix.

When the Skill Indexer should select online quantization material

Separate discovery from experiment choice

The internal Skill Indexer selects relevant owners and whole resources through
collect. Its lexical score is a relevance signal. The host agent interprets
that material against observed task facts and chooses the next action.

flowchart TD
    A["Task goal and observed target facts"] --> B["Collect: relevant skills and complete resources"]
    B --> C["Host agent: contract, bottleneck and compatibility checks"]
    C --> D["Missing facts: diagnosis or contract clarification"]
    C --> E["Eligible mechanisms: compare next experiments"]
    E --> F["Existing runner: frozen campaign and validators"]
    F --> G["Retain outcomes; reassess the limiting stage"]
    G --> B

An explicit request about online quantization should retrieve its guidance even when
hardware or quality inputs are missing. Knowledge discovery needs no experiment
authorization. Missing inputs determine the next investigation or clarification;
they must not hide relevant knowledge or imply readiness to execute.

When the task is inference-framework optimization, keep offline checkpoint
production outside the default candidate set. If diagnosis suggests it as a
possible alternative, name it as a separately scoped model-delivery dependency.
An explicit offline-quantization request should retrieve or hand off to model
preparation guidance, with that different scope stated. An existing offline
quantized model may still need engine compatibility/kernel guidance; its origin
does not exclude optimization of its execution.

For a general optimization request, surface online quantization diagnosis when the task
facts suggest one of the following mechanisms:

Selection should happen at task intake for explicit quantization/fit requests,
after initial measurement for general speed requests, and again when a change
moves the bottleneck. A model that cannot start may use source-grounded memory
accounting and a bounded feasibility probe before profiling is possible.

Conditions before candidate experiments

The host agent should choose a quantization experiment only when the following
conditions have a concrete answer:

1. Allowed change. The task permits the relevant numerical approximation and
   effective model/runtime changes. Preserving original checkpoint bytes does not make
   runtime quantization an exact transformation. If the task requires exact
   parity or unchanged effective precision, route to compatible alternatives.
2. Relevant mechanism. The hypothesis names the component and bytes/work it
   can remove, and relates them to the actual objective. A capacity experiment
   and a latency experiment may have different objectives.
3. Executable path. The selected model, scheme, engine revision, device,
   shapes and parallelism have a compatible implementation or an explicitly
   scoped implementation task. Use the engine's actual support contracts, such
   as the vLLM hardware/implementation table,
   then check effective dispatch on the target. Branding alone is insufficient.
4. Frozen evaluation. The acceptance Issue defines the quality, memory,
   performance and stability requirements, required levels and stop conditions;
   plan/ and validator/ are frozen before candidate search. Unknown quality
   tolerance permits investigation, not an assumed default tolerance.
5. Affordable lifecycle. Initialization conversion, any required calibration,
   runtime scale/conversion overhead, engineering and validation fit the delegated
   resource budget and objective. Preserve a restartable incumbent and recovery
   path. A documented calibration-free path must not be blocked for lacking a
   calibration dataset; it still requires frozen quality evaluation.

Do not require every exact optimization to fail first. Compare eligible
quantization and exact mechanisms by objective relevance, plausible removable
time/bytes, uncertainty and total experiment cost. Use observed stage fractions
and appropriate attainable envelopes; no universal memory, utilization, quality
loss or gain percentage decides the choice. Quantization can change the work and
traffic, so the current high-precision roofline is not its performance ceiling.

For reused weights, compare the extra initialization cost against the useful
work over the declared worker lifetime. For activation and KV conversion, include
the recurring overhead in each measured path; it cannot be amortized as a one-time
load cost. Use matched timing boundaries and avoid summing overlapping profiler
durations as though they were exposed service time. A warm-service objective may
exclude initialization from its primary metric only when that boundary is frozen
in the plan; initialization time and peak memory must still be reported.

Explain selection and abstention

For each shortlisted mechanism, the agent should report its supporting
observation/source, applicability gaps, strongest counter-signal, next check and
reason for its relative priority. This can live in the existing task/campaign
plan; this RFC introduces no decision database or required response schema.

Distinguish missing evidence from known incompatibility. If a profiling tool is
unavailable, retain the skill's diagnosis and seek supported measurement or
accounting alternatives. Do not discard all of its guidance or fabricate a
profile. A failure under one engine/shape/device is a counter-signal for that
scope, not a universal ban on the quantization family.

These are planning recommendations, not new experiment verdicts. Campaigns keep
only PASS, FAIL and INCONCLUSIVE under the existing evidence contract.

Indexer integration and compatibility

Recommend content and selection cases first. Use the upstream #128
relevance/context_bytes opt-ins and exact-version reader instead of adding a
public quantization router, mandatory metadata schema or model-based reranker.
Better descriptions and mechanism cues are the first intervention. A structured
applicability filter is a separate follow-up only if labeled cases show that
content changes cannot address the selection errors.

• Preserve default query/filter/order behavior and explicit-ID membership/order.
  The existing token-AND query can exclude cross-domain owners before relevance
  ranking: the caller must deliberately broaden discovery rather than silently
  bypass its filters. Investigate a mechanism across owners, then qualify it for
  the target model and backend.
• Eligible generated skill projections retain kind=knowledge. Ordinary imported
  records, checkout skills and cases use their existing reads; do not relabel
  them to force entry into the packaged-skill ranking scope. Combine independent
  knowledge and skill discovery at the caller when needed.
• Preserve canonical Bundle closure, qualified refs, exact revisions and hashes.
  material retains its existing meaning; selected context is a separate view.
• Return complete core, mandatory prerequisites, provenance and licenses. If the
  mandatory set exceeds context_bytes, report the whole-owner omission with
  required/available bytes. Never emit a truncated procedure as ready to apply.
• Treat index_unavailable, context_limit, no-match and unsupported scope as
  explicit limitations. Resolve them through the existing read/budget choices;
  retrieval omissions do not establish technical incompatibility.

Evaluation and evidence

Validate discovery and decisions separately

Extend the existing Skill Eval and selection contract
tests. Freeze labeled cases before tuning descriptions or selection behavior;
separate development cases from held-out cases. Labels specify relevant owners,
required counter-signals and the appropriate next action, not one universally
best quantization algorithm.

Report relevant-owner recall, wrong-family selections, unsafe experiment
recommendations, correct abstentions, required-resource coverage and omission
handling. Numeric acceptance bars belong in the accepted evaluation Issue and
frozen plan, not reusable skills or CI defaults.

For task effects, compare the same client, task facts, available tools and
budgets with and without the new content; evaluate selection changes in a
separate contrast so content and selector effects are distinguishable. Retain
all outcomes, tokens, latency and missing observations. Offline labels establish
software/selection behavior; separately authorized agent trials establish only
their scoped task outcomes. Neither establishes an engine speedup.

Validate an actual online quantization campaign

Record <baseline_sha>, <candidate_sha>, <campaign-id>, the original model
manifest, tokenizer/preprocessing, conversion/kernel revision and configuration,
component exclusions, scale policy, initialization order and actual runtime
dispatch. Identify any calibration inputs or supplied scale metadata only when
the selected path requires them. The original checkpoint stays the common input;
a separately produced quantized checkpoint is not a prerequisite. When no runtime
artifact is serialized, bind the effective representation through its construction
code, original input hashes, complete settings and recorded numerical checks;
do not invent an artifact hash. Extend campaign-owned manifests first; add no
RunSpec field or generic configuration interface solely for this RFC.

Keep development/search evaluation and sealed final acceptance inputs distinct.
If calibration or warmup-derived scales are used, freeze their input identities,
update/reset policy and separation from final evaluation. Final measured requests
may compute per-input dynamic scales as defined by the frozen algorithm; do not
silently turn them into cross-request training/calibration state. Include the task's domains, lengths, languages and
modalities; PPL, tensor error or a single media example cannot replace complete
task quality. Hold sampling, geometry, steps and useful work constant except for
the declared approximation. Record sensitive data through approved identities
and access controls rather than publishing raw private prompts or paths.

Use L0 for quantize/dequantize and operator contracts, L1 for complete real-shape
operator paths, L2 for loading/dispatch/component integration, L3 for complete
model/modality quality and L4 for service claims. Select the required ladder in
the acceptance Issue. In particular, verify:

• Weights convert at the declared initialization point, source buffers are
  released as specified, and adapter/fusion changes invalidate or rebuild affected
  representations. Record transient and steady memory separately.
• Activation scale granularity, numerical edge cases, batch/shape dependence and
  complete conversion/GEMM cost match the declared policy, including compile/graph
  capture and warmup behavior where used.
• KV scales and storage stay paired through append, attention reads, prefix
  reuse, request reset and cancellation. Long-context quality and repeated-request
  state checks cover the selected cache implementation.

Report cold conversion/startup time and peak, steady residency, complete warm
execution, any calibration cost and stability separately. A smaller runtime
tensor or faster single GEMM does not establish service gains.

Separate a matched-workload comparison from a capacity experiment that increases
concurrency or context under the same memory budget. Retain conversion overhead,
scale/packing memory, failed requests and regressions. Quantization strategy
changes belong to the declared Model plane; runtime/kernel implementation belongs
to Infra when its numerical contract stays fixed; changing selection/evaluation
belongs to Harness. Use separate campaign
steps and frozen controls, following contributing.

Delivery plan and acceptance

Open or reuse an accepted scoped Issue before implementation. Assign its outcome
owner and independent reviewer for bounded slices; report PR readiness separately
from Issue completion. Protected knowledge changes follow the existing
authorship admission policy, with truthful model
and effort declarations. Campaign gates and research-source provenance remain
separate. Draft/ready/merge/nightly retain their existing roles; add no default
model downloads, paid calls, accelerator jobs or campaign thresholds to CI.

The RFC decision can accept a phased plan without accepting all proposed skills
at once. Knowledge delivery, improved selection and quantization performance are
three independently evidenced outcomes.

Alternatives and open decisions

Decisions for maintainers:

1. Confirm the initial scope: weight conversion at initialization, runtime
   activation conversion and runtime KV quantization. Offline checkpoint
   production and its quality acceptance belong to model delivery; support for
   loading/executing those artifacts remains an engine responsibility. Training
   and automatic per-request precision policies stay separately scoped.
2. Choose the first model/engine/backend and whether its first procedure belongs
   to vLLM, H3/vLLM-Omni or torchao. Confirm the bounded initial skill set and owner.
3. Agree that Skill Indexer discovery may surface conditional quantization
   guidance, while the host agent applies the experiment eligibility checks.
4. Freeze selection/task evaluation criteria and the target campaign's quality,
   resource and performance gates in their respective accepted Issues.

Ownership and completion

• Actual requester: the human requester in the authoring chat; RFC publication explicitly requested.
• Acting contributor: Codex, for drafting and publication.
• Status: Proposed.
• Accountable implementation owner: TBD.
• Goal-acceptance and closure owner: TBD; to be designated by maintainers.
• Authorized scope: prepare and publish this RFC. Implementation and campaign execution require their own accepted scope.
• RFC completion criteria: record decisions on the scope, initial skill set, first target and evaluation criteria; link accepted follow-up Issues and retain deferred or unresolved items. Publication alone does not satisfy those decisions.