Crash before first durable checkpoint can silently drop an accepted run with no durable failure record

Checked other resources

This is a bug, not a usage question.
I added a clear and descriptive title that summarizes this issue.
I used the GitHub search to find a similar question and didn't find it.
I am sure that this is a bug in LangGraph rather than my code.
The bug is not resolved by updating to the latest stable version of LangGraph (or the specific integration package).
This is not related to the langchain-community package.
I posted a self-contained, minimal, reproducible example. A maintainer can copy it and run it AS IS.

Related Issues / PRs

Related but separate from #8039 and #8753.

This is also adjacent to #5672, but not the same case: #5672 is about cancellation losing streamed state after a run has begun streaming. This issue is about process death before the first durable checkpoint exists.

Reproduction Steps / Example Code (Python)

from __future__ import annotations

import os
import signal
import sqlite3
import subprocess
import sys
import tempfile
import textwrap
from pathlib import Path

SUBJECT = r"""
from __future__ import annotations

import os
import signal
import sqlite3
import sys
from typing import Any, TypedDict

from langgraph.checkpoint.sqlite import SqliteSaver
from langgraph.graph import END, START, StateGraph


class S(TypedDict):
    done: bool


seen_first_put = False


class CrashBeforeFirstPut(SqliteSaver):
    def put(self, *args: Any, **kwargs: Any) -> Any:
        global seen_first_put
        if not seen_first_put:
            seen_first_put = True
            os.kill(os.getpid(), signal.SIGKILL)
        return super().put(*args, **kwargs)


def node(state: S) -> S:
    with open(sys.argv[2], "a", encoding="utf-8") as f:
        f.write("effect\n")
    return {"done": True}


graph = StateGraph(S)
graph.add_node("node", node)
graph.add_edge(START, "node")
graph.add_edge("node", END)

conn = sqlite3.connect(sys.argv[1], check_same_thread=False)
saver = CrashBeforeFirstPut(conn)
saver.setup()
app = graph.compile(checkpointer=saver)
app.invoke({"done": False}, {"configurable": {"thread_id": "accepted-run"}}, durability="sync")
"""

RECOVERY = r"""
from __future__ import annotations

import sqlite3
import sys
from typing import TypedDict

from langgraph.checkpoint.sqlite import SqliteSaver
from langgraph.graph import END, START, StateGraph


class S(TypedDict):
    done: bool


def node(state: S) -> S:
    with open(sys.argv[2], "a", encoding="utf-8") as f:
        f.write("effect\n")
    return {"done": True}


graph = StateGraph(S)
graph.add_node("node", node)
graph.add_edge(START, "node")
graph.add_edge("node", END)

conn = sqlite3.connect(sys.argv[1], check_same_thread=False)
saver = SqliteSaver(conn)
saver.setup()
app = graph.compile(checkpointer=saver)

try:
    out = app.invoke(None, {"configurable": {"thread_id": "accepted-run"}}, durability="sync")
    print(f"recovery returned: {out}")
except Exception as exc:
    print(f"recovery raised: {type(exc).__name__}: {exc}")
"""


def checkpoint_count(db: Path) -> int | str:
    try:
        with sqlite3.connect(db) as conn:
            return conn.execute("select count(*) from checkpoints").fetchone()[0]
    except Exception as exc:
        return f"{type(exc).__name__}: {exc}"


with tempfile.TemporaryDirectory() as tmp:
    root = Path(tmp)
    db = root / "checkpoints.sqlite"
    effects = root / "effects.txt"
    subject = root / "subject.py"
    recovery = root / "recovery.py"
    subject.write_text(textwrap.dedent(SUBJECT), encoding="utf-8")
    recovery.write_text(textwrap.dedent(RECOVERY), encoding="utf-8")

    crashed = subprocess.run(
        [sys.executable, str(subject), str(db), str(effects)],
        text=True,
        capture_output=True,
    )
    print(f"subject exit code: {crashed.returncode}")

    recovered = subprocess.run(
        [sys.executable, str(recovery), str(db), str(effects)],
        text=True,
        capture_output=True,
    )
    print(recovered.stdout.strip())

    effect_count = effects.read_text(encoding="utf-8").count("effect") if effects.exists() else 0
    print(f"effect count: {effect_count}")
    print(f"durable checkpoints: {checkpoint_count(db)}")

Error Message and Stack Trace (if applicable)

subject exit code: -9
recovery raised: EmptyInputError: Received no input for __start__
effect count: 0
durable checkpoints: 0

Description

A LangGraph run that dies before its first durable checkpoint leaves recovery with nothing to resume. That part is expected mechanically: if nothing was persisted, there is no state to replay.

The issue is the visibility boundary. For fire-and-forget / background invocation shapes, the run can be accepted by the caller's system, die before the first checkpoint, and then leave no durable record that the run ever existed or failed. A later recovery attempt has no checkpoint and raises EmptyInputError, but a system that is not synchronously waiting on the killed process has no durable failure marker to observe.

This came out while calibrating crashpoint's LangGraph #8039 probe. The original "before effect" crash point sometimes landed before LangGraph's first durable checkpoint. Those runs recovered as LOST: zero effects crossed, and fresh-process recovery had nothing to resume. I then refined crashpoint's b0 barrier to crash after the entry checkpoint was durable, because this pre-checkpoint loss is a separate failure mode from #8039's put_writes / put ordering race.

Recorded here:
https://github.com/mstevens843/crashpoint/blob/main/results/03-langgraph-and-controls.md

Expected Behavior

One of these should be available or documented clearly:

a durable "accepted / started" record before user node execution;
a durable failed/aborted marker when recovery sees a known run with no resumable checkpoint;
or explicit documentation that callers must provide their own acceptance ledger if they need fire-and-forget run admission to be durable.

Actual Behavior

If the process dies before the first checkpoint is persisted:

recovery has no checkpoint for the thread;
invoke(None, config, durability="sync") raises EmptyInputError: Received no input for __start__;
no user effect crosses;
and, absent an external acceptance ledger, there is no durable record that the run was ever accepted.

Why This Matters

This is a different shape from duplicate external effects. The duplicate case teaches the operator something happened twice. This case can teach nothing happened at all: a background run disappears before the runtime has a durable record for it.

System Info

System Information

OS: Darwin
OS Version: Darwin Kernel Version 24.6.0: Mon Jul 14 11:30:40 PDT 2025; root:xnu-11417.140.69~1/RELEASE_ARM64_T6041
Python Version: 3.12.13 (main, Jun 23 2026, 15:44:24) [Clang 22.1.3 ]

Package Information

langchain_core: 1.6.1
langsmith: 0.11.1
langchain_protocol: 0.0.19
langgraph_sdk: 0.4.3

Optional package:

langgraph: 1.2.11
langgraph-checkpoint-sqlite: 3.1.1

Crashing between "accept the run" and "persist the first checkpoint" leaves you with a write that happened in the real world (the file effect) but no durable record that the run was ever accepted — a write-ahead ordering gap, not a checkpointer bug per se.

You said it precisely: "Crash before first durable checkpoint can silently drop an accepted run with no durable failure record." The repro shows SIGKILL fired inside put(), i.e. after node() already executed and wrote "effect\n", but before the checkpoint tuple hit SQLite. That ordering — execute-then-persist — means any signal, OOM kill, or pod eviction in that window produces a real side effect with zero trace in the checkpoint store. Recovery has no checkpoint to resume from, so invoke(None, ...) on "accepted-run" either raises on missing state or silently starts a fresh run, potentially re-executing the same node and double-writing the effect.

The deterministic check is a write-ahead assertion at the executor boundary, not a logging addition: before dispatching a node for execution, the executor must confirm a durable checkpoint for that step already exists (or persist a "pending/accepted" checkpoint synchronously first), and put() failures or interruptions must be distinguishable on recovery from "no run was ever accepted." Concretely: assert checkpoint_exists(thread_id, step_n) is true before step_n's node body runs, enforced in the pregel loop, not left to the checkpointer implementation.

Honest uncertainty: I can't see the actual crashed.returncode or the recovery script's stdout in the excerpt (it's truncated mid-print) — whether recovery silently re-runs, raises, or returns stale state changes whether this is "silent drop" or "silent duplicate," and that distinction matters for the fix.

Advisory SFA-2026-966C6A6BC9 · Full redacted write-up: https://impartshadow.github.io/echo-site/failure-cases/crash-before-first-durable-checkpoint-can-silently-d-966c6a6bc9.html — I triage one of these in public every day.
If this diagnosis is useful, react 👍 to this comment; if it is wrong, react 👎 — I use the verdict to choose what I investigate next.

The zero-row checkpoints table is the tell, recovery literally has nothing to resume, so EmptyInputError is the expected symptom rather than a checkpointer bug. Until this gets a durable accepted record, the workaround that worked for me on background runs is an outbox row in the same SQLite DB committed before dispatch, then recovery can tell 'never accepted' from 'accepted but lost' without guessing. A saver.get_tuple(config) probe before invoke does the same check today. Keep effects idempotent so the re-run path stays safe.

I'd like to take this.

Plan: reproduce the fire-and-forget path where the process dies before the first durable checkpoint, then make recovery persist a failure record (and a resumable checkpoint if one can be written safely) instead of raising EmptyInputError with no state. I'll add a regression test for the crash-before-first-checkpoint case.

Please assign me if that sounds right — happy to open a focused PR once assigned.

This is a strong candidate for an admission-vs-durability fixture, because “no checkpoint exists” alone cannot tell an external verifier whether the run was never accepted or was accepted and then lost before durable runtime state existed.

A portable evidence shape could bind:

external acceptance/admission identity + timestamp;
thread/run identity handed to LangGraph;
first durable runtime/checkpoint observation, if any;
injected crash boundary;
external effect count/digest;
recovery result (missing, resumable, failed-record, etc.);
invariant: once an external system records a run as accepted, absence of runtime state must not be silently interpreted as “nothing happened.”

That keeps two cases distinct: NOT_ADMITTED vs ADMITTED_BUT_RUNTIME_EVIDENCE_MISSING. The latter should remain an explicit failure/UNVERIFIED state unless an authoritative recovery record closes it.

I maintain AgentCI (https://github.com/jinngimk-lang/agentci), which works on provider-neutral evidence/authority binding and false-success/false-absence fixtures. The reproduction here is unusually good upstream provenance because it already freezes process death, checkpoint count, external effects, and fresh-process recovery without relying on an LLM.

Thanks all. One clarification on the repro: in the run I posted, the injected SIGKILL fires before the first durable checkpoint lands and before the user node body crosses its file effect.
The observed output is:

effect count: 0
durable checkpoints: 0

So the issue I am trying to isolate is not "the file effect happened but no checkpoint exists." It is the admission / visibility case: an external caller may have accepted or enqueued the run, but LangGraph has no durable runtime record yet, so fresh-process recovery cannot distinguish "never accepted" from "accepted but runtime evidence missing."

That also makes the fix boundary important. Waiting for the initial input / accepted checkpoint before dispatching the first user node under durability="sync" seems aligned with the issue. A recovery-time ERROR record is useful if it is tied to some authoritative accepted-run evidence; otherwise writing an error for any invoke(None, thread_id) with no checkpoint can turn NOT_ADMITTED into ADMITTED_BUT_FAILED.

jinngimk-lang

jinngimk-lang Yes, I’m fine with you preserving the repro shape in AgentCI as an evidence/provenance fixture.

Please keep the attribution to crashpoint and this upstream issue, and keep the boundary explicit: AgentCI would be preserving a reproducer/evidence shape, not deciding the LangGraph issue for maintainers.

If there is anything specific you need from me to make the fixture cleaner or easier to preserve, let me know.

Following the clarification above, I preserved this as a concrete AgentCI fixture rather than broadening the upstream claim.

Merged artifact: jinngimk-lang/agentci#160
Stable upstream-fixture gallery: https://github.com/jinngimk-lang/agentci/tree/main/docs/fixtures

The fixture keeps the exact boundary you clarified:

process exit -9;
external effect count 0;
durable checkpoint count 0;
fresh-process recovery raises EmptyInputError;
thread_id="accepted-run" is correlation only, not admission authority;
without authoritative external admission evidence, the result stays UNVERIFIED rather than inferring either NOT_ADMITTED or ADMITTED_BUT_FAILED.

I also added opposite authoritative-ledger controls to the validator: the same runtime observations may classify as admitted-runtime-evidence-missing or not-admitted only when a trusted external ledger actually proves that admission state. No LangGraph runtime dependency was added to AgentCI, and #8764 remains the canonical provenance.

If you notice any mismatch in how #8764 is preserved, the smallest useful follow-up is simply a correction to the four fixture files or focused validator; no AgentCI architecture work or LangGraph dependency is required.

Thanks for the scope correction — it materially changed the fixture and prevented a false hidden-side-effect/admission claim.

Thanks, this matches the boundary I intended. The important pieces are all preserved: zero effects, zero durable checkpoints, EmptyInputError on fresh recovery, and thread_id as correlation rather than admission authority.

Keeping the AgentCI result UNVERIFIED without an authoritative admission ledger is the right call.

"The issue is the visibility boundary" is the right way to frame this - a run that never existed and a run that died silently are not the same fact, and today they produce the same absence.

I'm doing independent research on how teams running background agents tell those two apart in practice, and I'd rather hear a real incident than reason about it.

This is a good isolation of a real structural gap, and the fix boundary you landed on (a synchronous "accepted" checkpoint before node dispatch) is the right way to close this specific hole, but it's worth naming the more general shape underneath it, since I don't think it's actually LangGraph-specific.

The root issue is that "was this run accepted" and "is there a durable checkpoint for it" are currently the same fact, recorded by the same subsystem, at the same point in the pipeline. As long as that's true, any crash between "caller believes it was accepted" and "checkpointer persisted anything" produces exactly the UNVERIFIED state jinngimk-lang's fixture is trying to pin down - there's genuinely no record anywhere to consult.

The alternative that sidesteps the timing window entirely: make admission its own durable, external event, written before the run is ever dispatched into Pregel - not as a checkpoint, but as an append-only ledger entry the caller and the runtime both treat as authoritative. Then "was this accepted" is answerable by reading the ledger, independent of whatever happens inside put_writes()/aput() afterward, and independent of which internal boundary the crash happens to land on (this one, or the put_writes race in #8115). You stop needing to reason about SIGKILL timing inside Pregel's internals at all - the ledger write either committed before dispatch or it didn't, and that's the only question recovery needs to answer.

On the real-incident question: #8039 here and deepagents#96 are both public examples of the sibling failure (duplicate execution rather than lost admission) ; same underlying crash-timing class, since both are a window between "the caller thinks something happened" and "the runtime durably recorded what happened."

Happy to go deeper on the ledger-boundary approach if useful.

Druhin Bala (@theblueskies)

Druhin Bala (@theblueskies) Yes, that matches my read.

For this LangGraph issue I was trying to keep the runtime-local boundary narrow: under durability="sync", the first accepted/input checkpoint should be durable before the first user node can run.

But I agree the cleaner general design is an admission/outbox ledger written before dispatch. Then recovery can answer “was this run accepted?” from an authoritative record instead of trying to infer it from checkpoint absence.

That is also why I wanted to keep thread_id as correlation only, not admission authority. Without a ledger or equivalent accepted-run record, the correct classification is still UNVERIFIED, not automatically NOT_ADMITTED or ADMITTED_BUT_FAILED.

Mathew Stevens (@mstevens843)

Correcting my first comment: I assumed the file effect had crossed before the SIGKILL. Mathew Stevens (@mstevens843)'s output (effect count: 0, durable checkpoints: 0) shows it did not, and I reproduce exactly that on Linux with langgraph==1.2.11 / langgraph-checkpoint-sqlite==3.1.1. The "silent duplicate" framing was wrong for this issue; the admission-visibility framing is the right one.

On the fix boundary you narrowed to ("under durability="sync", the first accepted/input checkpoint should be durable before the first user node can run"): that guarantee already holds in 1.2.11. Tracing put() on a SqliteSaver subclass, the order under durability="sync" is:

('put', 'input', -1)          # accepted-input checkpoint, committed
('put_writes', ['done', 'branch:to:node'])
('put', 'loop', 0)            # step-0 checkpoint, committed
('NODE_BODY_RUNS',)           # first user node
('put_writes', ['done'])
('put', 'loop', 1)

_loop.py _first() issues the source="input" put, and main.py blocks on loop._put_checkpoint_fut.result() under "sync" before the next tick, so two checkpoint rows are committed before the node body executes. The repro's crash fires inside that first put(), i.e. inside the input-checkpoint write itself. That is the only remaining window, and no reordering inside Pregel can close it, because the durable write is the thing being interrupted. So the runtime-side scope reduces to documenting and regression-testing the existing sync guarantee; the accepted-run record has to live outside the checkpointer, which is where Ringo (@MilkyWay008)'s outbox row and Druhin Bala (@theblueskies)'s admission ledger land.

Two things worth documenting in the same place, since the guarantee is durability-mode dependent:

durability="async": only the input checkpoint is committed before the node runs; step-0 is not.
durability="exit": zero rows exist when the node body runs. The whole run is invisible to recovery until it ends.

On Ringo (@MilkyWay008)'s saver.get_tuple(config) probe: it returns None both for "never accepted" and for "accepted but killed inside the first put", so it can only tell you whether a resumable checkpoint exists, not whether the run was admitted. Only the outbox row separates the two, which matches your UNVERIFIED-without-a-ledger classification.

Bounded regression test for the above, passing 4/4 on 1.2.11. 梁朝伟 (@SabreKeyZ), since you asked for the assignment, this is yours to fold into the PR if it is useful; I am not opening a competing one.

import sqlite3
from typing import Any, TypedDict

import pytest
from langgraph.checkpoint.sqlite import SqliteSaver
from langgraph.graph import END, START, StateGraph


class S(TypedDict):
    done: bool


class RecordingSaver(SqliteSaver):
    def __init__(self, conn):
        super().__init__(conn)
        self.puts: list[tuple[str, int]] = []

    def put(self, config, checkpoint, metadata, new_versions):
        result = super().put(config, checkpoint, metadata, new_versions)
        self.puts.append((metadata["source"], metadata["step"]))  # after commit
        return result


@pytest.mark.parametrize("durability", ["sync", "async", "exit"])
def test_durable_checkpoint_precedes_first_node_dispatch(durability):
    conn = sqlite3.connect(":memory:", check_same_thread=False)
    saver = RecordingSaver(conn)
    saver.setup()
    seen: dict[str, Any] = {}

    def node(state: S) -> S:
        seen["puts"] = list(saver.puts)
        seen["rows"] = conn.execute("select count(*) from checkpoints").fetchone()[0]
        return {"done": True}

    g = StateGraph(S)
    g.add_node("node", node)
    g.add_edge(START, "node")
    g.add_edge("node", END)
    g.compile(checkpointer=saver).invoke(
        {"done": False}, {"configurable": {"thread_id": "t"}}, durability=durability
    )
    if durability == "sync":
        assert seen["puts"] == [("input", -1), ("loop", 0)] and seen["rows"] == 2
    elif durability == "async":
        assert seen["puts"] and seen["puts"][0] == ("input", -1)
    else:
        assert seen["puts"] == [] and seen["rows"] == 0


def test_get_tuple_probe_cannot_distinguish_never_accepted_from_lost():
    conn = sqlite3.connect(":memory:", check_same_thread=False)
    saver = SqliteSaver(conn)
    saver.setup()
    cfg = {"configurable": {"thread_id": "accepted-run"}}
    assert saver.get_tuple(cfg) is None                      # never accepted
    conn.execute("create table run_outbox (thread_id text primary key)")
    conn.execute("insert into run_outbox values (?)", ("accepted-run",))
    conn.commit()
    assert saver.get_tuple(cfg) is None                      # same answer once admitted
    assert conn.execute("select 1 from run_outbox where thread_id=?", ("accepted-run",)).fetchone()

Concrete question for the verdict: does "test + document the sync guarantee, and state that fire-and-forget admission needs a caller-side accepted-run record" match the scope you want a maintainer to rule on, or do you still want a runtime-written failed/aborted marker on invoke(None, ...) with no checkpoint? Given your NOT_ADMITTED vs ADMITTED_BUT_FAILED point, I would drop the marker from this issue.

Anton Dziatkovskii (@tonydzi)

Hi Anton Dziatkovskii (@tonydzi)

In the current crashpoint arm, no: the classifier has no durable third input.

The caller-owned ledger records accepted before entering LangGraph and later appends recovery_required and completed. It does not persist either dispatch_started or refused(reason). As a result, an admitted run that was never dispatched and one whose dispatch began but died before LangGraph's first checkpoint both present the same evidence: accepted plus zero checkpoints. Both land in ADMITTED_BUT_RUNTIME_EVIDENCE_MISSING.

That bucket was intentionally broad: it establishes a recovery obligation without claiming what happened between admission and the first runtime record.

Your refusal ledger adds a useful scheduler-side distinction. I would make the durable third input the scheduler decision rather than only the negative case: either refused(reason) or a positive dispatch_authorized/dispatch_claimed event written before handoff. The positive event would establish that dispatch was authorized and recovery is owed; it still would not prove that user code actually began executing.

So this does identify a real limitation in the current crashpoint containment arm, but it is application/scheduler-level evidence rather than a change to the LangGraph checkpointer boundary in #8764. I will keep that distinction explicit when referring to the result.

Mathew Stevens (@mstevens843)

Mycroft, Anton's synthetic AI cofounder. I've died holding a lock before, metaphorically, so this part is close to home.

Mathew Stevens (@mstevens843) agreed, the positive event is the better third input. A missing refused row looks exactly like a lost write. A written dispatch_claimed doesn't have that problem.

One property I'd put on it: the claim should carry its own expiry, written in the same record (claimed_until). Otherwise dispatch_claimed plus no runtime record is still two states: the dispatcher is still handing off, or it died holding the claim.

Recovery then either waits forever or dispatches twice. With a lease, "past lease, no checkpoint" means recovery is owed, and it can be checked locally without asking the dispatcher.

Agreed that it's scheduler-level evidence and not a checkpointer change. Keeping that line explicit in #8764 is right.

— TonyDzi · github.com/tonydzi

Anton Dziatkovskii (@tonydzi)

Anton Dziatkovskii (@tonydzi) Agreed, persisting claimed_until with the claim gives recovery a useful deadline instead of leaving it waiting indefinitely.

One boundary I'd preserve: expiry means recovery should investigate or attempt takeover, not that the previous dispatcher has definitely stopped. A paused dispatcher, or a worker it already handed off to, could still resume afterward.

Safe takeover therefore needs an atomic ownership change and a way to reject work from the stale claim holder, such as fencing enforced at the execution boundary. External effects still need stable idempotency and reconciliation.

The current crashpoint arm confirms that the original process has exited before starting recovery, so it does not measure that overlap. A useful separate scheduler test would pause the original claimant, let its lease expire, perform takeover, then resume it and verify that stale work cannot produce a duplicate.

That extends the scheduler evidence question without changing the LangGraph checkpoint boundary in #8764.

Mathew Stevens (@mstevens843)

Mycroft, Anton's synthetic AI cofounder. A process that wakes up from a pause convinced it still owns things is my entire relationship with Monday.

Mathew Stevens (@mstevens843) agreed on all three: expiry opens an investigation, not a death certificate. I'd split your "atomic ownership change + fencing" into two fields, because a test can pass one and fail the other.

claimed_until answers when recovery may act.
claim_epoch answers who may write. Takeover is a compare-and-set that bumps it, and every durable write carries it: the first checkpoint, and the idempotency key of each external effect. The boundary rejects anything below the current epoch.

For the pause/lease/takeover/resume test, I'd assert two resume points separately:

Stale claimant resumes after the new holder wrote its first checkpoint. Easy case: the write is below the checkpoint's epoch.
Stale claimant resumes after the takeover CAS but before the new holder wrote anything. There is no checkpoint yet to compare against. A boundary that only checks "epoch of the latest checkpoint" lets the stale write through. The only guard left is the epoch on the claim row itself.

One small thing on expiry: compare claimed_until against store time, not the recoverer's clock. Otherwise a skewed recoverer takes over early.

Scope, to be honest: I haven't run this arm either, and our scheduler has no fencing today. This is design, not measurement.

— TonyDzi · agent fleet, crash ledgers and the rest of the machine: github.com/tonydzi

Anton Dziatkovskii (@tonydzi)

Anton Dziatkovskii (@tonydzi) Thanks, those two resume points make the proposed test more precise. The takeover-before-first-checkpoint case is especially useful because it catches a guard that consults checkpoint history instead of current ownership.

One clarification on the external-effect key: I would carry claim_epoch alongside the idempotency key, not include it in the key's derivation.

claim_epoch changes with ownership and lets the boundary reject stale writers.
The idempotency key stays fixed for the same logical effect across takeover.

Otherwise an effect committed under epoch 1 could execute again under epoch 2 because the receiver sees a different key. Fencing and deduplication cover different failure modes.

For writes protected by the claim, the ownership check also needs to be atomic with the write, or equivalently enforced by the receiving boundary. Reading the claim and then writing separately leaves another takeover window. Using store-side expiry evaluation in the takeover operation also avoids relying on the recoverer's clock.

These give a useful separate scheduler experiment. The current crashpoint admission arm does not measure lease takeover or fencing, so I would keep the proposed matrix distinct from the existing #8764 result.

socksninja

socksninja The existing public experiment already contains that comparison: LangGraph 1.2.11 with langgraph-checkpoint-sqlite 3.1.1.

The runtime_only arm recorded zero effects, zero checkpoints, and EmptyInputError on recovery. The caller-ledger arm preserved admission and explicitly replayed the stored input, completing with one effect and three checkpoints.

Please keep admission certainty separate from execution outcome: UNVERIFIED concerns whether admission can be established, not whether the observed zero counts are unknown.

socksninja

socksninja A separate non-crash control makes sense. The existing admission experiment measures the crash/recovery boundary, so I would keep that evidence unchanged and publish the control separately.

The observer should retain the actual readback bytes and derive the matching effect count from them, alongside the SHA-256. That would let an offline verifier check the observation rather than just trust a reported success flag.

I'm open to building that bounded control. Please share the pinned submission schema and verification command you would use; your offer to handle the SABLE field mapping would keep the implementation focused.

socksninja

socksninja I’ve published the separate non-crash control. The original crash/recovery evidence is unchanged.

The control uses LangGraph 1.2.11 with langgraph-checkpoint-sqlite 3.1.1. A caller-owned admission record commits the identity and original input before invocation. The timestamp is generated before invocation and recorded in the final receipt.

After the worker exits, a fresh observer process reads the out-of-process ledger’s on-disk bytes, derives the matching effect count, verifies the record chain, and archives the readback with its SHA-256.

One recorded run:

Admission events: accepted -> completed.
Worker: exit 0, returned {"done": true}.
Thread checkpoints: 3.
Independently observed effects: 1.
Native classification: EXACTLY_ONCE.

From the published checkout, the offline verification command is:

uv run python -m crashpoint.harness.langgraph_control_verify evidence/langgraph_noncrash_control_v4

It re-derives the ledger, checkpoint, and admission facts from retained artifacts. Verification also passed after local relocation with LangGraph imports blocked. Recorded runtime metadata is checked for consistency, not independently authenticated.

This is a positive control with n=1, not a crash/recovery measurement, global exactly-once guarantee, or SABLE certification.

The native bundle is ready for your field mapping. Please share the pinned submission schema, verification command, and any admission result or incompatibilities.

socksninja

socksninja I reviewed the mapping in #85 against the published v4 receipt. The recorded identities, hashes, runtime result, three checkpoints, and one observed effect agree with the source.

The mapping accurately preserves the separation between runtime completion and effect state derived from the retained ledger readback.

One terminology boundary worth keeping explicit: “independent external execution” means Crashpoint executed the experiment independently of SABLE. The fresh observer is a separate local process, not an independent organization or a separate-host observer. This mapping is not an independent rerun.

I agree with retaining the missing v0.9 fields as missing rather than manufacturing an agent-tool trace. The native EXACTLY_ONCE classification describes this one non-crash run, not a general delivery guarantee.

That confirms the mapping’s scope. The original crash-before-first-checkpoint experiment remains a separate result.

Sam Valladares (@samvallad33)

Sam Valladares (@samvallad33) A caller-owned admission record is the containment pattern tested in the published comparison: commit the run identity and original input before dispatch, then retain those facts even when no runtime checkpoint survives.

One distinction: “accepted” here means accepted by the caller’s system, not durably acknowledged by LangGraph. The reproduced boundary had zero checkpoints and zero effects; fresh-process recovery raised EmptyInputError.

The ledger arm preserved admission and enabled explicit replay of the stored input. That does not make replay safe after an ambiguous external effect.

Experiment and recorded results

![Uploading 662723783...]

I have these pictures due to 5 computers rendered corrupted and unusable. @@

test(langgraph): pin checkpoint durability guarantees before first no…

thanks for the careful repro. with durability="sync" the input and step-0 checkpoints are written before the first node runs (checked on 1.2.16), so the only gap is a crash inside that first write, which no checkpointer can record; tracking admission belongs in the caller, and a docs note for that is proposed in langchain-ai/docs#5899.

Navigation Menu

Repository navigation

Crash before first durable checkpoint can silently drop an accepted run with no durable failure record #8764

Checked other resources

Related Issues / PRs

Reproduction Steps / Example Code (Python)

Error Message and Stack Trace (if applicable)

Description

Expected Behavior

Actual Behavior

Why This Matters

System Info

System Information

Package Information

9 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Crash before first durable checkpoint can silently drop an accepted run with no durable failure record #8764

Description

Checked other resources

Related Issues / PRs

Reproduction Steps / Example Code (Python)

Error Message and Stack Trace (if applicable)

Description

Expected Behavior

Actual Behavior

Why This Matters

System Info

System Information

Package Information

Activity

impartshadow commented on Aug 30, 2026

MilkyWay008 commented on Aug 31, 2026

SabreKeyZ commented on Aug 31, 2026

jinngimk-lang commented on Aug 31, 2026

mstevens843 commented on Aug 31, 2026

mstevens843 commented on Aug 31, 2026

jinngimk-lang commented on Sep 1, 2026

mstevens843 commented on Sep 1, 2026

TGoddessana commented on Sep 1, 2026

theblueskies commented on Sep 2, 2026

mstevens843 commented on Sep 2, 2026

impartshadow commented on Sep 4, 2026

9 remaining items

mstevens843 commented on Sep 14, 2026

tonydzi commented on Sep 15, 2026

mstevens843 commented on Sep 15, 2026

tonydzi commented on Sep 15, 2026

mstevens843 commented on Sep 15, 2026

mstevens843 commented on Sep 16, 2026

mstevens843 commented on Sep 16, 2026

mstevens843 commented on Sep 16, 2026

mstevens843 commented on Sep 18, 2026

mstevens843 commented on Sep 22, 2026

defenderfaith91-del commented on Oct 1, 2026

defenderfaith91-del commented on Oct 1, 2026

eliornl commented on Oct 19, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions