Repository navigation
INT8 GEMM support #93
Description
The C++ matmul kernels (aie2/mm.cc, aie2p/mm.cc) already have vectorized INT8
matmul templates (i8→i8, i8→i16, i8→i32, MAC shape 8x8x8) and compile flags
(-Di8_i8_ONLY, etc.), but the Python GEMM operator only accepts bf16 input.
This would wire up INT8 through the Python layer:
- design.py: add "i8" to dtype_in, "i8"/"i16"/"i32" to dtype_out,
add i8 MAC dims (8,8,8) to microkernel_mac_dim_map - op.py: add i8 kernel flags, min tile sizes, skip bf16 emulation for int8
- reference.py: int8 golden reference
- test.py: int8 test cases
The NPU does ~50 TOPs INT8 vs ~3-5 TOPs bf16, so this would be a big
throughput gain for quantized inference.
If I'm not missing anything it should be trivial, I'm already on it.
Activity
Add INT8 GEMM support to the GEMM operator
Add dequant operator (int32 → bfloat16)
Add quantize operator (bfloat16 → int8)
hunhoffe commentedon Oct 28, 2026
Closing this out — INT8 GEMM support has landed.
gemm.py is now generic over input_dtype/output_dtype with explicit integer-dtype validation, and iron/operators/flm/dequant/ covers bringing the int32 accumulator back to bf16. This also means #99 (scale_i32_bf16) and #95 (dequantize) are no longer blocked on this landing — flagging there in case either of you wants to pick those back up.
A standalone quantize (bf16→int8) op isn't in yet — if that's still needed, let's track it as its own issue rather than reopening this one.