[vieduy]

Overview

On Metal, matmuls with batch > 8 use mul_mm only when K >= 64. Below that, mul_mv is used: _short for K < 32, and the generic kernel for K >= 32. The generic kernel is built for long rows, so at K = 32–63 its per-row reduction and dispatch overhead dominate. At n = 512, K = 32 runs 32x slower than K = 64, despite half the math.

LoRA matrices have K = rank, so rank 32–63 adapters hit this. On master, an r32 or r48 adapter roughly halves prefill speed; r16 and r64 don't (about 16% of popular LoRA adapters on Hugging Face are r32, also my research team choose r32 for good result enough).

This change sends 32 <= K < 64 to mul_mm, which already supports it: it steps K by 32 and zero-pads the tail.

• K < 32 is unchanged; mul_mm is slower there at small batch.

Additional information

All numbers are from an Apple M4 (10-core GPU), macOS 27.0.1.

Correctness: test-backend-ops on Metal passes (MUL_MAT 2475/2475, MUL_MAT_ID 867/867, full suite 17861/17861). Model output is unchanged: llama-perplexity KLD vs master on Qwen3.5 0.8B with r32 and r48 adapters is 2e-6, the same as rerunning master itself.

Op level (master vs PR, 30 shapes per K: F16/F32, m 1024–6144, n 16–2048):

K	speedup (geomean, min–max)
32	30.9x (5.8–54.6x)
48	20.5x (4.0–35.6x)
all other K (8–24, 64–128)	0.98–1.03x

e.g. F16 2048x512, K = 32: 1772 us → 36 us.

End to end: prefill tok/s, llama-batched-bench -npp 512 -ntg 64 -npl 1 -ngl 99, median of 3.

model / pack	master	PR
Qwen3.5 0.8B, no pack	2282	2239
Qwen3.5 0.8B, r16	1706	1733
Qwen3.5 0.8B, r32	799	1817
Qwen3.5 0.8B, r48	793	1792
Qwen3.5 0.8B, r64	1804	1772
Ternary Bonsai 2 2B, no pack	983	992
Ternary Bonsai 2 2B, my fine-tune r32 pack	489	859
Ternary Bonsai 2 2B, same pack zero-padded to r64	862	856

Reproduce:

test-backend-ops perf -b MTL0 -o MUL_MAT -p "type_a=f16,type_b=f32,m=2048,n=512,k=(16|32|48|64),bs"
llama-batched-bench -m <model> --lora <rank-32 adapter> -c 4096 -npp 512 -ntg 64 -npl 1 -ngl 99

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES. Claude Code was used to help develop and test the fix. I reviewed every line and take full responsibility for the changes.

[vieduy]

Hi @ggerganov ,a short ping on this PR. I just merged the latest master. Is there anything you'd like me to add or change? Thanks for your time.

[ggerganov]

I think we can simply add a BF16 x BF16 kernel as a stopgap. I'll be reworking the Metal backend to add more generic support for src1 BF16.

[vieduy]

@ggerganov Thanks for your suggestion about adding a BF16 x BF16 kernel.