[am17an]

Overview

Fixes #31531 ("--fit on" leaves one layer on the CPU, and performance halves).

Since #26605, common_get_device_memory_data_impl sets hp_ngl = n_layer when MTP is not loaded, so fit assigns n_layer + 1 layers across the devices. The model loader still counts n_layer_all + 1 for n_gpu_layers, including the NextN layers, which hold no weights without MTP (i_gpu_start = n_layer_all + 1 - n_gpu_layers). When fit offloads everything to more than one device, it sets n_gpu_layers = 65 for Qwen3.8-27B (n_layer = 64, n_layer_all = 65). The loader then starts at layer 1, which leaves layer 0 on the CPU. The tensor-buft-override indices computed from hp_ngl were off by the same amount.

In set_ngl_tensor_split_tbo, add the skipped NextN layers to n_gpu_layers and to the tensor split of the last device that has layers. That device also holds the output layer, and the NextN layers sit right before it. The layers the loader assigns then match the layers fit measured and the override indices. Nothing changes when MTP is loaded or when the model has no NextN layers.

Additional information

Reproduction (GB10, CUDA0 + local RPC0 as a second device, Qwen3.8-27B-UD-Q8_K_L):

ggml-rpc-server -d CUDA0 -p 50631 &
llama-server -m Qwen3.8-27B-UD-Q8_K_L.gguf -fit on --fit-target 100000 --rpc 127.0.0.1:50631 -lv 4

	fit result	load_tensors	CPU RS buffer	CUDA_Host model buffer
master 29c720e	34 + 31 layers	offloaded 65/66 layers	12.47 MiB (layer 0 on CPU)	1624 MiB
this PR	35 + 30 layers	offloaded 66/66 layers	none	1288 MiB

Single-device partial offload (--fit-target 100000, no RPC) stays consistent. Fit's test allocation for k layers now reports the same memory as master's k+1, because master's count included the empty NextN slot. The loaded layer count matches what fit printed.

Not tested: ROCm (the reporter's 2x Radeon VII setup), and models with more than one NextN layer.

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES, paired with Claude

[JohannesGaessler]

Please keep the handling of hyperparameters consistent. It's fine to make them optional via pointers but then please do this consistently for all of them.

[ggerganov]

@JohannesGaessler Would you like to take one more look at the changes?

[Eronex]

I can confirm this fix works on the ROCm setup that the PR lists as untested — 2× AMD Radeon VII (gfx906), Qwen3.8-27B Q6_K, --fit on, --device ROCm0,ROCm1, --split-mode layer.

Layer distribution (before the fix):

I common_params_fit_impl: set ngl_per_device[0].n_layer=34   # ROCm0
I common_params_fit_impl: set ngl_per_device[1].n_layer=31   # ROCm1
I load_tensors: offloaded 65/66 layers to GPU                # layer 0 stays on the host

After (this PR): offloaded 66/66 layers to GPU — no host-resident layer.

Why the throughput halves

It's more than just "one layer on the CPU". With layer 0 left on the host, its weights are copied to the GPU as split inputs on every graph evaluation, and the resulting buffer layout (many CPU↔GPU split boundaries) does not match the layout reserved at startup. The scheduler then re-reserves the compute buffers on (nearly) every step — the n_same counter just keeps climbing:

I sched_graph_realloc: graph reallocation: nodes = 4071 (changed = 0), splits = 20, n_same = 1, n_diff = 0
I sched_graph_realloc: graph reallocation: nodes = 4071 (changed = 0), splits = 20, n_same = 2, n_diff = 0
I sched_graph_realloc: graph reallocation: nodes = 4071 (changed = 0), splits = 20, n_same = 3, n_diff = 0
...

changed = 0 means the graph size is unchanged, yet a full re-reserve (ggml_gallocr_reserve_n) runs each step — and each one synchronizes all backends. That per-step churn is the direct mechanism behind the "performance drops by half" symptom.

The layout the scheduler sees is also dynamic — the input-embedding buffer resizes between a single generation token and a full prompt batch:

## SPLIT #1: ROCm0  inputs: [inp_mixed_embd ( 20K)] ...   # 1 token (generation step)
## SPLIT #1: ROCm0  inputs: [inp_mixed_embd ( 10M)] ...   # full prompt batch

The 20-split layout (GGML_SCHED_DEBUG=2)

Full split dump

## SPLIT #0  : CPU
## SPLIT #1  : ROCm0  inputs: [inp_mixed_embd] [node_1] [inp_mixed_slots] [model.input_embed]
## SPLIT #2  : ROCm0  inputs: [blk.0.attn_norm.weight]
## SPLIT #3  : CPU
## SPLIT #4  : ROCm0  inputs: [blk.0.attn_qkv.weight] [conv_states_reshaped]
## SPLIT #5  : CPU    inputs: [conv_state_last]
## SPLIT #6  : ROCm0  inputs: [blk.0.ssm_conv1d.weight]
## SPLIT #7  : ROCm0  inputs: [blk.0.ssm_alpha.weight]
## SPLIT #8  : ROCm0  inputs: [blk.0.ssm_dt.bias]
## SPLIT #9  : ROCm0  inputs: [blk.0.ssm_a]
## SPLIT #10 : ROCm0  inputs: [blk.0.ssm_beta.weight] [state_predelta]
## SPLIT #11 : CPU    inputs: [new_state]
## SPLIT #12 : ROCm0  inputs: [blk.0.ssm_norm.weight]
## SPLIT #13 : ROCm0  inputs: [blk.0.attn_gate.weight]
## SPLIT #14 : ROCm0  inputs: [blk.0.ssm_out.weight]
## SPLIT #15 : ROCm0  inputs: [blk.0.post_attention_norm.weight]
## SPLIT #16 : ROCm0  inputs: [blk.0.ffn_gate.weight]
## SPLIT #17 : ROCm0  inputs: [blk.0.ffn_up.weight]
## SPLIT #18 : ROCm0  inputs: [blk.0.ffn_down.weight] [attn_inp_k_rot] [attn_inp_v_rot] [attn_inp_k_idxs] [attn_inp_v_idxs] [attn_inp_kq_mask]
## SPLIT #19 : ROCm1  inputs: [norm-35] [l_out-34] [attn_inp_k_rot] [attn_inp_v_rot] [attn_inp_k_idxs] [attn_inp_v_idxs] [attn_inp_kq_mask] [out_ids]

The blk.*.weight entries listed as inputs to the ROCm splits are the host-resident weights copied to the GPU on every evaluation. The CPU splits (#0, #3, #5, #11) hold the SSM state (conv_state_last, new_state).

With this PR applied (66/66 layers, no host→GPU weight copies) the sched_graph_realloc events disappear entirely and throughput returns to the expected level.