Skip to content

server: trust a disk-restored slot on hybrid-recurrent (DSV4) models - #28074

Closed
SilasK wants to merge 1 commit into
ggml-org:masterfrom
SilasK:fix/dsv4-restore-checkpoint
Closed

server: trust a disk-restored slot on hybrid-recurrent (DSV4) models#28074
SilasK wants to merge 1 commit into
ggml-org:masterfrom
SilasK:fix/dsv4-restore-checkpoint

Conversation

@SilasK

@SilasK SilasK commented Aug 31, 2026 •
edited
Loading

Copy link
Copy Markdown

Overview

Fix POST /slots/:id?action=restore (--slot-save-path) being useless on hybrid-recurrent (DSV4) models, like qwen 3

llama_kv_cache_dsv4::seq_pos_min() intentionally returns seq_pos_max because DeltaNet recurrent state cannot be rewound, so server-context.cpp always requires a context checkpoint to reuse cached state. SLOT_RESTORE restored KV+recurrent via llama_state_seq_load_file but never populated slot.prompt.checkpoints, so the next request always logged forcing full prompt re-processing due to lack of cache data and re-prefilled from zero. Full-attention models unaffected.

Fix (17 lines, tools/server/server-context.cpp:SLOT_RESTORE): after successful load, register a PARTIAL_ONLY checkpoint covering [0, token_count-1] via update_tgt/update_dft/common_speculative_get_state. Next request finds it, restores recurrent state, and evaluates only the tail.

Additional information

  • Why not cache-ram alone: --cache-reuse is disabled for DSV4 (get_can_shift()==false) and cache-ram does not survive restart/idle unload; disk save/restore is the only mechanism that does.
  • Verified on Qwen3.8-27B-AD (DSV4): before = 561ms/317t full re-prefill; after = registered checkpoint over restored state (430 tokens, pos [0,429]) + cached n_tokens=430, 188ms/28t tail only, total 1103ms. 7112t/10-checkpoint context restores at cached n_tokens=7112.
  • Manual repro steps in description below. No small hybrid model available for automated test_slot_save.py coverage, so verification is via manual A/B logs and isolated build from 9de0fcf2b.

Reproduction:

  1. Build from this branch (cmake -B build-pr -S . && cmake --build build-pr --target llama-server)
  2. llama-server --slot-save-path /tmp/slots -np 1 -lv 4 -m <Qwen3.8-AD-Q5.gguf> --port 18082 --ctx-checkpoints 32
  3. Build conversation (~400t), POST /slots/0?action=save {"filename":"a.bin"}, overwrite slot with different prompt, POST /slots/0?action=restore {"filename":"a.bin"}, continue with original history+tail and check logs.

Requirements

  • I have read and agree with the contributing guidelines
  • AI usage disclosure: YES - used AI for the code writing. PR description double-cheked by me.

@SilasK
server: trust a disk-restored slot on hybrid-recurrent (DSV4) models
0f76cbb 
On DSV4 the slot operator requires a context checkpoint to reuse
restored KV (seq_pos_min returns seq_pos_max). SLOT_RESTORE never
populated checkpoints, so every restore forced a full re-prefill.
Register a PARTIAL_ONLY checkpoint over the restored range after
load_file.

Verified 317tok -> 28tok tail-only on Qwen3.8-27B-AD, 7112tok long
context resumes from restored state. Full-attention models unaffected.

Assisted-by: Muse Spark
@SilasK
SilasK requested a review from a team as a code owner August 31, 2026 08:17
@SilasK

SilasK commented Aug 31, 2026

Copy link
Copy Markdown
Author

Closing as duplicate of #26004 which correctly preserves real checkpoints. My synthetic full-range fallback only helps empty-checkpoint (430t) case - happy to add it as follow-up to #26004 if wanted.

@SilasK SilasK closed this Aug 31, 2026
dgabehar added a commit to dgabehar/llama.cpp that referenced this pull request Sep 10, 2026
@dgabehar @claude
server: fix slot save/restore losing checkpoint-based cache reuse
b0e57a8 
POST /slots/{id}?action=save|restore never actually delivers the
KV-cache-reuse benefit it's documented to, found while building this
fleet's disk-persistence feature (master.slotSavePath) against a
draft-mtp (speculative decoding) deployment.

REVISION NOTE: the first version of this fix seeded one synthetic
full-range checkpoint after restore. That worked for the common case
(resuming a saved conversation by appending a new turn -- verified live:
3.0s vs 70.5s cold, cache_n=4533) but failed for a non-monotonic
continuation (resending a prompt SHORTER than what was saved, e.g. a
divergent retry) -- confirmed live: same 71.6s either way, cache_n=0.
Root cause of that gap: a single synthetic checkpoint's own pos_max
becomes a hard ceiling on how much reuse the very next request can ever
be trusted for (`cur.pos_max > pos_next` rejects it outright once the new
request's own match falls short of that ceiling), and there's no one
fixed pos_max that works for every possible future request. Superseded by
the approach below -- persisting whatever REAL checkpoints already
existed at save time, at their own natural positions, rather than
inventing one -- which searches correctly against any future request.

UPSTREAM PRIOR ART (found after building and testing the superseded
synthetic-checkpoint version above, before writing this one): this exact
bug is well-documented upstream --

  - Issue ggml-org/llama.cpp#25913 ("/slots
    save/restore silently loses all prompt reuse on hybrid/recurrent
    models") -- open, matches our own symptom and log evidence exactly
    (f_sim_best = 1.000 at slot selection, cache_n = 0 on the very next
    request, "restore reports success and delivers zero benefit").
  - PR ggml-org/llama.cpp#20819 -- an early fix
    attempt, now stale/conflicting (unrebased since 2026-06), stalled on
    the project's AI-disclosure contribution policy rather than a
    technical objection.
  - PR ggml-org/llama.cpp#28074 -- an independent
    submission of the SAME synthetic-single-checkpoint approach this
    fix's own superseded first version used. Closed by the author as a
    duplicate of PR #26004 ("My synthetic full-range fallback only helps
    empty-checkpoint case").
  - PR ggml-org/llama.cpp#26004 -- OPEN,
    MERGEABLE (no conflicts as of this vendoring), with a proper
    regression test (tools/server/tests/unit/test_slot_save.py ::
    test_slot_restore_preserves_context_checkpoints), and multiple
    independent community confirmations across different hardware/model
    combinations (Mac/Metal/Qwen3.6-35B-A3B; Vulkan/RADV/gfx1151/
    Qwen3.8-Flash-Next; ROCm/Qwen3.6-27B). Not yet merged as of this
    vendoring, apparently on maintainer review bandwidth rather than any
    open technical objection. This commit's own save_slot_checkpoints/
    load_slot_checkpoints functions below are adapted from that PR
    (hand-ported to apply against our exact pinned commit -- PR #26004's
    own diff did not apply byte-for-byte due to unrelated line drift from
    its last rebase).

TWO DISTINCT FIXES, both needed for this fleet's draft-mtp deployments:

1. PR #26004's approach (save_slot_checkpoints/load_slot_checkpoints, and
   their two call sites in SLOT_SAVE/SLOT_RESTORE): persist the
   checkpoints that already existed in slot.prompt.checkpoints at save
   time -- each with its own data_tgt/data_dft/data_spec -- as a tagged
   appendix after the main llama-state payload in the same save file.
   Old files without the appendix (no "SCKP" magic at the expected
   offset) restore exactly as before -- silent no-op, not an error.

2. This fleet's own addition, NOT covered by PR #26004: neither
   SLOT_SAVE nor SLOT_RESTORE persist ctx_dft (the draft model's own,
   currently-live KV cache) directly -- only whatever data_dft happens
   to be attached to an existing checkpoint. On a short conversation
   that hasn't hit its first checkpoint yet, the restored slot's draft
   model would silently start from an empty KV cache with no live
   fallback. Implemented as a companion `<save-file>.dft` written
   alongside the main save file, via
   llama_state_seq_get_size_ext/get_data_ext (capture) and
   llama_state_seq_set_data_ext (restore) -- the same primitives
   common_prompt_checkpoint::update_dft/load_dft already use internally,
   just driven directly here since there's no single-call file-based API
   for a second context. A missing .dft file (an older save, or a
   non-speculative deployment with ctx_dft == nullptr) is not an error.

Both verified together live: save -> evict (different conversation on the
same slot) -> restore -> resend-with-extension shows cache_n = 4533 (the
full restored prefix reused, only 17 new tokens processed), 3.0s versus
70.5s cold.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017R3QNP4GznFWhhf11QY18j
dgabehar added a commit to dgabehar/llama.cpp that referenced this pull request Sep 14, 2026
@dgabehar @claude
server: fix slot save/restore losing checkpoint-based cache reuse
b8b5f74 
POST /slots/{id}?action=save|restore never actually delivers the
KV-cache-reuse benefit it's documented to, found while building this
fleet's disk-persistence feature (master.slotSavePath) against a
draft-mtp (speculative decoding) deployment.

REVISION NOTE: the first version of this fix seeded one synthetic
full-range checkpoint after restore. That worked for the common case
(resuming a saved conversation by appending a new turn -- verified live:
3.0s vs 70.5s cold, cache_n=4533) but failed for a non-monotonic
continuation (resending a prompt SHORTER than what was saved, e.g. a
divergent retry) -- confirmed live: same 71.6s either way, cache_n=0.
Root cause of that gap: a single synthetic checkpoint's own pos_max
becomes a hard ceiling on how much reuse the very next request can ever
be trusted for (`cur.pos_max > pos_next` rejects it outright once the new
request's own match falls short of that ceiling), and there's no one
fixed pos_max that works for every possible future request. Superseded by
the approach below -- persisting whatever REAL checkpoints already
existed at save time, at their own natural positions, rather than
inventing one -- which searches correctly against any future request.

UPSTREAM PRIOR ART (found after building and testing the superseded
synthetic-checkpoint version above, before writing this one): this exact
bug is well-documented upstream --

  - Issue ggml-org/llama.cpp#25913 ("/slots
    save/restore silently loses all prompt reuse on hybrid/recurrent
    models") -- open, matches our own symptom and log evidence exactly
    (f_sim_best = 1.000 at slot selection, cache_n = 0 on the very next
    request, "restore reports success and delivers zero benefit").
  - PR ggml-org/llama.cpp#20819 -- an early fix
    attempt, now stale/conflicting (unrebased since 2026-06), stalled on
    the project's AI-disclosure contribution policy rather than a
    technical objection.
  - PR ggml-org/llama.cpp#28074 -- an independent
    submission of the SAME synthetic-single-checkpoint approach this
    fix's own superseded first version used. Closed by the author as a
    duplicate of PR #26004 ("My synthetic full-range fallback only helps
    empty-checkpoint case").
  - PR ggml-org/llama.cpp#26004 -- OPEN,
    MERGEABLE (no conflicts as of this vendoring), with a proper
    regression test (tools/server/tests/unit/test_slot_save.py ::
    test_slot_restore_preserves_context_checkpoints), and multiple
    independent community confirmations across different hardware/model
    combinations (Mac/Metal/Qwen3.6-35B-A3B; Vulkan/RADV/gfx1151/
    Qwen3.8-Flash-Next; ROCm/Qwen3.6-27B). Not yet merged as of this
    vendoring, apparently on maintainer review bandwidth rather than any
    open technical objection. This commit's own save_slot_checkpoints/
    load_slot_checkpoints functions below are adapted from that PR
    (hand-ported to apply against our exact pinned commit -- PR #26004's
    own diff did not apply byte-for-byte due to unrelated line drift from
    its last rebase).

TWO DISTINCT FIXES, both needed for this fleet's draft-mtp deployments:

1. PR #26004's approach (save_slot_checkpoints/load_slot_checkpoints, and
   their two call sites in SLOT_SAVE/SLOT_RESTORE): persist the
   checkpoints that already existed in slot.prompt.checkpoints at save
   time -- each with its own data_tgt/data_dft/data_spec -- as a tagged
   appendix after the main llama-state payload in the same save file.
   Old files without the appendix (no "SCKP" magic at the expected
   offset) restore exactly as before -- silent no-op, not an error.

2. This fleet's own addition, NOT covered by PR #26004: neither
   SLOT_SAVE nor SLOT_RESTORE persist ctx_dft (the draft model's own,
   currently-live KV cache) directly -- only whatever data_dft happens
   to be attached to an existing checkpoint. On a short conversation
   that hasn't hit its first checkpoint yet, the restored slot's draft
   model would silently start from an empty KV cache with no live
   fallback. Implemented as a companion `<save-file>.dft` written
   alongside the main save file, via
   llama_state_seq_get_size_ext/get_data_ext (capture) and
   llama_state_seq_set_data_ext (restore) -- the same primitives
   common_prompt_checkpoint::update_dft/load_dft already use internally,
   just driven directly here since there's no single-call file-based API
   for a second context. A missing .dft file (an older save, or a
   non-speculative deployment with ctx_dft == nullptr) is not an error.

Both verified together live: save -> evict (different conversation on the
same slot) -> restore -> resend-with-extension shows cache_n = 4533 (the
full restored prefix reused, only 17 new tokens processed), 3.0s versus
70.5s cold.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017R3QNP4GznFWhhf11QY18j
dgabehar added a commit to dgabehar/llama.cpp that referenced this pull request Sep 14, 2026
@dgabehar @claude
server: fix slot save/restore losing checkpoint-based cache reuse
a1e9795 
POST /slots/{id}?action=save|restore never actually delivers the
KV-cache-reuse benefit it's documented to, found while building this
fleet's disk-persistence feature (master.slotSavePath) against a
draft-mtp (speculative decoding) deployment.

REVISION NOTE: the first version of this fix seeded one synthetic
full-range checkpoint after restore. That worked for the common case
(resuming a saved conversation by appending a new turn -- verified live:
3.0s vs 70.5s cold, cache_n=4533) but failed for a non-monotonic
continuation (resending a prompt SHORTER than what was saved, e.g. a
divergent retry) -- confirmed live: same 71.6s either way, cache_n=0.
Root cause of that gap: a single synthetic checkpoint's own pos_max
becomes a hard ceiling on how much reuse the very next request can ever
be trusted for (`cur.pos_max > pos_next` rejects it outright once the new
request's own match falls short of that ceiling), and there's no one
fixed pos_max that works for every possible future request. Superseded by
the approach below -- persisting whatever REAL checkpoints already
existed at save time, at their own natural positions, rather than
inventing one -- which searches correctly against any future request.

UPSTREAM PRIOR ART (found after building and testing the superseded
synthetic-checkpoint version above, before writing this one): this exact
bug is well-documented upstream --

  - Issue ggml-org/llama.cpp#25913 ("/slots
    save/restore silently loses all prompt reuse on hybrid/recurrent
    models") -- open, matches our own symptom and log evidence exactly
    (f_sim_best = 1.000 at slot selection, cache_n = 0 on the very next
    request, "restore reports success and delivers zero benefit").
  - PR ggml-org/llama.cpp#20819 -- an early fix
    attempt, now stale/conflicting (unrebased since 2026-06), stalled on
    the project's AI-disclosure contribution policy rather than a
    technical objection.
  - PR ggml-org/llama.cpp#28074 -- an independent
    submission of the SAME synthetic-single-checkpoint approach this
    fix's own superseded first version used. Closed by the author as a
    duplicate of PR #26004 ("My synthetic full-range fallback only helps
    empty-checkpoint case").
  - PR ggml-org/llama.cpp#26004 -- OPEN,
    MERGEABLE (no conflicts as of this vendoring), with a proper
    regression test (tools/server/tests/unit/test_slot_save.py ::
    test_slot_restore_preserves_context_checkpoints), and multiple
    independent community confirmations across different hardware/model
    combinations (Mac/Metal/Qwen3.6-35B-A3B; Vulkan/RADV/gfx1151/
    Qwen3.8-Flash-Next; ROCm/Qwen3.6-27B). Not yet merged as of this
    vendoring, apparently on maintainer review bandwidth rather than any
    open technical objection. This commit's own save_slot_checkpoints/
    load_slot_checkpoints functions below are adapted from that PR
    (hand-ported to apply against our exact pinned commit -- PR #26004's
    own diff did not apply byte-for-byte due to unrelated line drift from
    its last rebase).

TWO DISTINCT FIXES, both needed for this fleet's draft-mtp deployments:

1. PR #26004's approach (save_slot_checkpoints/load_slot_checkpoints, and
   their two call sites in SLOT_SAVE/SLOT_RESTORE): persist the
   checkpoints that already existed in slot.prompt.checkpoints at save
   time -- each with its own data_tgt/data_dft/data_spec -- as a tagged
   appendix after the main llama-state payload in the same save file.
   Old files without the appendix (no "SCKP" magic at the expected
   offset) restore exactly as before -- silent no-op, not an error.

2. This fleet's own addition, NOT covered by PR #26004: neither
   SLOT_SAVE nor SLOT_RESTORE persist ctx_dft (the draft model's own,
   currently-live KV cache) directly -- only whatever data_dft happens
   to be attached to an existing checkpoint. On a short conversation
   that hasn't hit its first checkpoint yet, the restored slot's draft
   model would silently start from an empty KV cache with no live
   fallback. Implemented as a companion `<save-file>.dft` written
   alongside the main save file, via
   llama_state_seq_get_size_ext/get_data_ext (capture) and
   llama_state_seq_set_data_ext (restore) -- the same primitives
   common_prompt_checkpoint::update_dft/load_dft already use internally,
   just driven directly here since there's no single-call file-based API
   for a second context. A missing .dft file (an older save, or a
   non-speculative deployment with ctx_dft == nullptr) is not an error.

Both verified together live: save -> evict (different conversation on the
same slot) -> restore -> resend-with-extension shows cache_n = 4533 (the
full restored prefix reused, only 17 new tokens processed), 3.0s versus
70.5s cold.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017R3QNP4GznFWhhf11QY18j
@Tough-Respawn Tough-Respawn mentioned this pull request Oct 3, 2026
Merged
1 task done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

server

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@SilasK