Durable Runs and Recovery Guide
When to use it
Use this recovery contract when a task writes files, runs commands, verifies across turns, or must continue after approval denial, terminal closure, or a network interruption. A one-shot read-only answer still receives an operation, but normally needs no manual resume.
Inspect the authoritative result
coremind run coremind.yaml --prompt "run the task" --json-eventsThe final run_result.snapshot is the authoritative pure-JSON result for runId, operation, outcome, metrics, evaluation, trace, checkpoints, artifacts, extension receipts, and resumability. Compatibility fields at the top level must match it. Do not infer success from the natural-language summary alone.
Verify correlation invariants
The internal checkInvariantFacts(facts, { mode }) seam reads Run, Session, and Checkpoint facts without modifying them. Production defaults to off; debugging and evaluation use eval; release acceptance uses gate. Gate fixtures must live under the tracked packages/coremind-runtime/test-fixtures/invariant-checker/ directory and must not depend on .coremind files generated by local test runs.
Legacy 0.3.0 facts have no TurnId, so the checker degrades according to the correlation known when each fact occurred. A new TurnId appended when the same Run resumes under 0.3.1 must not retroactively change legacy semantics.
Continue a paused or interrupted run
coremind run coremind.yaml --resume <runId> --json-eventsAutomatic resume stops for a changed configuration fingerprint, a terminal operation, a started or unknown effect receipt, a committed effect outside a stable completed step, or corrupt RunState. A tool denied before execution uses not_started and is not mistaken for an uncertain side effect.
Config 3 deferred tools retain the same resume gate: the same Run must match its original configuration fingerprint, and an old selection cannot authorize a changed configuration. Catalog snapshots, successful searches, and declaration generations remain private append-only facts; public context_prefix and provider_request events carry only safe summaries. Each new Run builds declarations from its current configuration. Sessions, parents, and previous Runs grant no activation. Runs without discovery facts follow the existing recovery path without inventing a historical tool set.
Executions within one Run remain distinct. A quality retry starts a new execution even for the same Agent and Step, without inheriting activation. Only an exact continuation binding in the latest authoritative pause may restore that execution's catalog and selection generation; completed executions are excluded. A crash without that binding starts with fresh deferred declarations if the existing admission gates allow resume, rather than guessing the latest state for that Agent and Step. The binding stays private and is stripped from public projections and Worker events. It does not prove complete conversation-message recovery.
Reading and reprojecting the same Facts in a cold Worker does not authorize continued execution. Resume still checks the original input, stable step boundary, Effect Receipts, and ownership. Successful discovery never replaces subsequent tool approval.
Critical journal cost and limits
The 1.2 development line reuses historical JSON text from successfully published critical prefixes. Under the existing writer lock, each critical commit still reads and exactly compares the complete UTF-8 text before reuse; an idempotent critical commit still synchronizes its barrier. At most 64 recently updated cursors retain a combined 4 MiB of UTF-8 prefix payload. Prefix eviction drops text, not the tail cursor.
This is not a Store heap limit. Explicit reads, query polling, failed commits, or external changes can require fresh validation. Caches are not shared across Store instances or processes. Hits reduce repeated parsing and serialization only: complete reads, full temporary rewrites, synchronization, and atomic publication remain. The journal format, resume gates, and critical acknowledgement contract do not change. Fixed-history JSON call counts do not establish production throughput, latency, or physical I/O gains.
Legacy sessions
The framework creates .v3.backup before migration. Supported messages, model changes, active tool changes, compactions, branch summaries, labels, and names migrate. An entry that cannot be represented losslessly returns an explicit error and preserves the source. Follow the SOP to retain the old version or export its effective context.
Explicit atomic task commits (1.4 release line, internal)
A trusted host can use TaskCommitSession from coremind-runtime/internal to persist a task checkpoint, business documents and related text together. This is an internal storage foundation: it does not add a regular ChatSession mode, execute tasks, schedule wakeups or expose Python/Worker commands. See ADR 0025.
open({ workspace, sessionId, documentSchemas })canonicalizes an existing workspace and obtains exclusive task-session ownership. A second writer is rejected.commit({ schemaVersion: 1, commitId, expectedRevision, taskId, task, documents, entries })validates everything before one atomic write. The receipt contains commitId, the new revision anddurabilityBoundary.snapshot()returns a copy of{ revision, tasks, documents, entries }. Readers see the old projection during the write and the complete new projection after success. Reopening rebuilds every view from commit facts.close()rejects new submissions, drains accepted submissions and releases ownership. Reads and writes are rejected after closing.
task is { type, version, phase, checkpoint }. This layer validates identifiers, a positive integer version and JSON, without executing or migrating task phases. Documents are { id, type, version, value }, replaced in full by id without deletion. Entries are { id, role: "user" | "assistant", text }, append-only with unique ids.
Register documents as { type, version, schema } with synchronous JSON Schema draft-07. Each type/version must be registered; used schema fingerprints are persisted. Reopening rejects missing or changed schemas. Async validation and unresolved remote references are rejected. Validation never coerces values, removes fields or inserts defaults.
Identifiers, types and phases contain 1–128 ASCII letters, digits, _ or -, starting with a letter or digit. Both the input and the fact payload including identity/schema fingerprints are limited to 1 MiB UTF-8. Checkpoints and document values must be finite JSON, without accessors, cycles, Date, functions, BigInt or undefined. Revision belongs to the whole task session. The same commitId with the same complete content returns its original receipt without appending; changed content conflicts. JSON object key order does not change idempotency.
The default file is <workspace>/.coremind/tasks/<sha256(sessionId)>/commits.jsonl, bound to the canonical workspace identity. Do not copy it into another workspace and reopen it, edit it, or duplicate its entries into the legacy Session log. Task data is neither automatically redacted nor encrypted. Standard tool authorization protects .coremind/tasks under the config directory and cwd; hosts choosing other workspaces must protect those paths too.
| Error | Action |
|---|---|
task_commit_invalid | Correct the complete submission or document schema; this attempt has not written |
task_commit_conflict | Inspect revision, commitId content and entry ids; do not automatically choose another key |
session_write_locked / policy_denied | Check ownership and workspace/link identity; never remove a live lock or bypass validation |
task_store_corrupt / run_state_corrupt | Preserve the file and inspect schema, version, identity and sequence; task tails are not repaired automatically |
durability_unsupported | The custom Store cannot enable this mode without the atomic critical/durability contract |
durability_barrier_failed / task_store_unavailable | Stop using the holder, close and reopen, then check the original commitId; failure may follow a complete durable write and does not imply rollback |
A custom store(directory) is a trusted host interface. It must declare atomicCriticalCommit: true, support critical, guarantee at least process_crash, commit one record atomically, and return the matching durability receipt. Reads must not silently repair task facts. A FileRunStore implementation must set repairTornTail: false. Any exposed pathFor must match the current lease. Capability declarations do not certify an arbitrary Adapter.
The built-in Store covers process crashes only, not power loss or remote payment/order transactions. Recovery reads the whole history; critical commits still rewrite the entire journal. This targets bounded local tasks, not a distributed queue or unbounded history. Run the model-free transaction example to check visibility, reopening and idempotency.
The new shared error codes change the Protocol v2 schema fingerprint: update SDK, Worker and manifest together. Hosts exhaustively matching the public RunStateKind must handle task_commit; ordinary Runs never emit it. To roll back, stop writers, retain the complete task directory and restore matching old SDK/Worker versions. Legacy fact domains require no migration. Older versions cannot read task logs; copying them into legacy Session or Run files is not a migration.
Common beginner mistakes
- Treating
pausedas permission to retry blindly. - Assuming an external effect succeeded merely because a local file changed.
- Deleting a lock without proving that its writer is gone.
- Backing up only the new file and losing the automatic legacy backup.
- Rerunning a new prompt after terminal closure and duplicating a non-idempotent tool call.
- Treating local
.coremindoutput as a reproducible Gate fixture.
Run the module example before adapting recovery to a business workflow.
Public durable task phases (1.4 release line)
TaskSession from coremind-ai adds start/query/resume/cancel and explicit version migration over the storage foundation. Each resume executes one phase; models/tools use the existing Runtime. Preserve and inspect the original Run when pendingRunId exists; do not resend automatically. See the public task guide and ADR 0026. This guide covers the 1.4.0 release line; public status depends on the corresponding Release, Registry artifacts and final qualification. TS and sync/async Python already access the same Worker; see the cross-language guide. Keyed calls are described below.
Durable waits (1.4 release line)
Returning sleepUntil atomically saves the next phase and absolute deadline. nextWake queries the earliest durable wait; wake(taskId, wakeId, sleepId) idempotently makes it ready, followed by explicit resume. An old alarm cannot advance a later sleep. Hosts own process restart, known ready-task recovery and timer reconstruction. Cancellation invalidates alarms; closing an idle session preserves waits. Waiting never extends Run, tool, phase or Worker timeouts. See the task guide and ADR 0027 for uncertain acknowledgements and older-SDK rollback limits.
Keyed calls (1.4 release line)
runCalls/executeTool bind stable keys to arguments, authority and order within one governed Run, sharing its budgets, permissions and ChildRun limits. calls queries history; unknown effects pause, while committed results reconcile from the original Run. Terminal parents execute no more calls; history retains the original 1 MiB limit without automatic cleanup. For compatibility, rollback and host versioning responsibility see the task guide and ADR 0028.