Building LLM workflows that can fail and resume

Checkpoints, idempotency, and recovery for a Spring Boot application.

Twenty rules need to be generated from a specification. Thirteen succeed. The request for rule 14 times out. Before the application saves the failure, the server crashes.

The server restarts. What happens next?

If the answer is “start the whole thing again,” the application has lost track of useful work. If it is “continue with rule 15,” it may have skipped a rule whose result was never saved. If two workers both decide to recover the job, they may repeat the same model calls.

A workflow is reliable when failure does not destroy the progress already made. For this article, progress means results committed to durable storage. A successful model response still sitting in memory is not a checkpoint.

I use a fictional expense-policy pipeline throughout:

Upload specification → Parse → Generate rules → Generate tests → Validate → Persist the final result

Persistence also happens between those stages. The final step publishes the validated result; it is not the first database write.

The companion example runs the 20-rule generation stage with a fake provider and a SQLite file. It includes a real process exit and restart, so the recovery claim can be tested. The Spring snippets below show how I would arrange the same transaction boundaries in a Java application. They are design sketches, separate from the executable demo.

Why doing everything in one request breaks

A synchronous endpoint is a reasonable first implementation. It reads the upload, calls the model, generates tests, and returns a result. The trouble begins when the work outlives the request that started it.

A client disconnect does not tell the application whether to abandon the job. A gateway timeout does not tell the client whether generation finished. A browser retry can submit the same work again. Keeping all intermediate results in a list leaves the restart story dependent on that process staying alive.

I separate accepting work from executing it. The upload is stored durably first. A short database transaction records the job and its initial work items. Only after that commit does the endpoint return 202 Accepted, a job ID, and a status URL.

For this design, a database poller discovers runnable jobs. A queue can wake workers sooner, but it is not the only record that work exists. If I later make queue delivery essential, I need to address the gap between committing the job and publishing its message—for example with a transactional outbox.

Why CompletableFuture is not workflow durability

This Java sketch makes stages asynchronous:

CompletableFuture.supplyAsync(() -> parse(spec), executor)
    .thenApplyAsync(this::generateRules, executor)
    .thenApplyAsync(this::generateTests, executor);

It does not save a job identity, checkpoint a rule, or tell another server where to resume. Those responsibilities are absent from the chain.

CompletableFuture provides completion and composition mechanisms, with asynchronous work dispatched through an executor. Those are useful building blocks inside a running process. They do not constitute a persisted workflow journal. Cancellation also does not guarantee that the underlying computation stops. Java 21 API documentation

I can still use futures to run independent model calls concurrently. I build recovery around persisted state, then use an executor to do the work that state makes eligible.

Give every execution a durable job identity

I distinguish a submission from an attempt.

The client supplies an idempotency key for the submission. The server associates that key with one job and an immutable input description:

job: job-482
submission key: expense-upload-37
source: expense-policy-v1
pipeline revision: rules-and-tests-v1
generation configuration: prompt-v1 / model-config-v1 / schema-v1

In a multi-tenant application, the uniqueness boundary includes the tenant. Repeating the same key and input returns the existing job. Reusing the key with different input is a conflict. An intentional rerun gets a new submission key.

Each rule also has a stable work identity, such as (job-482, generate-rule, rule-14). Attempt numbers and worker claims belong to that identity; they do not create a new logical rule each time.

I freeze the source and generation configuration for the job. A deployment that changes a prompt must not quietly resume half of an old job with the new prompt. It must retain the required configuration, explicitly migrate the job, or stop it for a decision.

Persist checkpoints, not just final results

A useful checkpoint says both what completed and where its output lives. A counter saying “13 done” cannot identify which 13 rules are safe to reuse, especially once work runs in parallel.

I store one work record per independently retryable unit:

For a small result, saving the output and changing its state to succeeded in the same transaction is straightforward. The reference example stores both in one row. If outputs live in object storage, I need an immutable object reference and a policy for uploads that succeed before the database commit fails.

In the fictional pipeline, parsing commits the extracted input and creates 20 rule items atomically. This version uses a stage barrier: test generation becomes eligible only when all required rules succeed. Test items are created once, under a uniqueness constraint, in the transaction that advances the stage.

If rule generation succeeds and test generation fails, the rules remain saved. The job has unfinished test work; it does not need new rules merely because a later stage failed.

Design steps to be idempotent

Idempotency is about the effect of repeating an operation. A repeated completion for rule 14 must not produce two saved rule-14 records, even if the model returns different text on each call.

I avoid “check whether a result exists, then insert it” as two unprotected operations. Two workers can both pass that check. Instead, a uniqueness constraint identifies the logical output, and a conditional write accepts only the current attempt’s result.

The application database and the model provider are separate systems. Consider this sequence:

1. The provider finishes generating rule 14.
2. The connection fails before the application receives the response.
3. The application retries.

The application cannot infer from the timeout that the provider did no work. It may pay for two calls while storing one accepted result.

If the provider supports idempotency keys or retrieving an existing operation, I can use that capability within its documented limits. A local job ID alone does not create it. I would keep a provider operation key stable across retries of the same logical request, while issuing a fresh worker claim token for each local attempt.

The guarantee in this design is narrow: one accepted local result per work identity, with stale attempts rejected. It does not promise exactly-once external execution.

Separate retryable failures from permanent ones

A retry policy needs a reason, a schedule, and a limit.

I persist next_attempt_at. Sleeping a thread for a minute wastes a worker slot and loses the retry schedule on restart.

A practical delay policy uses exponential backoff with jitter, a cap, and an overall job deadline. The demo deliberately uses a fixed delay and a three-attempt budget so its tests remain predictable. A worker crash consumes an attempt too; otherwise a repeatedly crashing item can avoid the retry limit forever.

A permanent failure leaves the existing successful outputs available. For this pipeline’s all-required stage barrier, it also blocks downstream work. “Nothing running” is not the same state as “job completed.”

Use bounded concurrency

Rules can run independently after parsing. Creating 20 futures at once is easy; deciding how many requests the system can afford at once is the more useful question.

I give a worker a small number of slots and claim an item only when a slot is available. Claiming a hundred items before queueing them behind three threads lets leases expire while work is still waiting locally.

The runnable example has three slots. Each slot claims one item, awaits its provider result, saves an outcome, and then claims another. The tests assert that no more than three fake provider calls are active at once.

That is a per-process bound. Four replicas with three slots can have twelve calls in flight. A production limit must also consider provider quotas, tokens per minute, connection pools, job fairness, and costs across replicas. Limiting concurrency alone does not enforce a request-per-minute quota.

The sequential rule-14 walkthrough is intentionally easier to follow. With parallel workers, some of rules 15–20 may already be complete when rule 14 fails. Recovery selects unfinished records; it never resumes from a numeric offset.

Prevent duplicate execution

A job can be triggered twice without creating two jobs. Two workers can also discover the same unfinished item. These are separate races.

I use an atomic claim to address the second race. A worker receives a fresh token and a lease, commits that claim, and then calls the provider outside the database transaction.

PostgreSQL can support queue-like consumers with row locking and SKIP LOCKED: another consumer skips a locked candidate rather than waiting for it. The claim update must occur in the same short transaction as selection. PostgreSQL cautions that skipped rows give an inconsistent view, so this is a queue-consumer technique, not a general reporting query. PostgreSQL 17 SELECT documentation

The SQLite demo uses BEGIN IMMEDIATE to serialize its short claim transactions. It is a local teaching example, not a demonstration of PostgreSQL locking or a multi-host queue.

A lease lets another worker recover abandoned work. It cannot prove that the old worker has stopped. A paused worker may return after its lease expires while a replacement is already calling the provider.

For that reason, completion is conditional on the token and an unexpired lease. The old result cannot overwrite the replacement’s result. For calls that may outlive the lease, I would add renewal under the same ownership condition and set request deadlines deliberately. Neither mechanism can force a remote provider to cancel work it has already accepted.

Resume from persisted state

Now return to the crash.

The recovery loop is the ordinary worker loop. It queries eligible pending items, due retries, and expired claims. There is no separate reconstruction from log messages.

Download the reference example ZIP, extract it, and open a terminal in its folder:

node --test store.test.mjs
node worker.mjs /tmp/expense-workflow-demo.db crash 10000
node worker.mjs /tmp/expense-workflow-demo.db resume 12000

Use a fresh database filename for each walkthrough. The crash command intentionally exits with code 14. The last argument is an injected clock in milliseconds: advancing it from 10000 to 12000 puts the second process beyond the demo’s one-second lease, without a sleep.

The restarted process makes seven fake provider calls, for rules 14–20. The first 13 rows remain unchanged. This is an injected timeout and abrupt process exit, not a live provider timeout or a machine power-loss test.

The reference example uses Node.js 22.23.3 and its experimental built-in SQLite API. It needs no provider account. It implements the rule-generation stage and recovery mechanics, not the entire upload-to-publication application.

Apply state transitions safely

A state diagram only helps if storage enforces it. These are the transitions I allow:

Here is the actual completion guard from the demo, with positional parameters:

UPDATE work
SET state = 'succeeded', output = ?, error = NULL,
    token = NULL, lease_until = NULL
WHERE job = ? AND item = ? AND state = 'running'
  AND token = ? AND lease_until > ?;

One updated row means the completion was accepted. Zero means the claim is no longer valid or the item already completed. I discard that stale result instead of retrying an unconditional write.

In a Spring application, I would separate the short claim and completion transactions from the provider call. This sketch uses TransactionTemplate explicitly; repository methods and result validation are omitted:

var claim = transactionTemplate.execute(status -> repository.claimNext(jobId));
if (claim == null) return;

var result = modelClient.generate(claim.input()); // Outside the transaction.
validateShape(result);

var accepted = transactionTemplate.execute(status -> repository.completeIfOwned(claim, result));
if (!Boolean.TRUE.equals(accepted)) {
    discardStaleResult(claim);
}

Spring documents TransactionTemplate for programmatic transaction management. If I use @Transactional instead, I keep the proxy boundary in mind: a call from one method to another on the same instance bypasses the usual proxy interception. Merely annotating a helper does not establish the transaction I need. Spring transaction management, Spring proxy behavior

In a multi-host database implementation, I use a shared database clock for lease comparisons. The demo passes time explicitly for repeatable tests. I also treat database errors separately from provider errors: failing to save a result must not be misreported as a failed model call.

Know when this architecture is overkill

For a short, cheap, disposable operation, a synchronous request with a timeout may be enough. If rerunning the whole task is harmless and no reviewed intermediate work exists, item-level checkpoints may cost more complexity than they save.

Once jobs are expensive, long-running, or reviewed between stages, I want the recovery behavior written down and tested. Before building more orchestration machinery, I would compare a simple persisted queue with an existing workflow engine, especially if I need timers, human pauses, cancellation, or version migrations across many stages.

My minimum failure test is the opening scenario: commit 13 outputs, lose the process during the next attempt, and restart against the same storage. Then I test duplicate delivery, expired claims, retry exhaustion, and the gap between remote success and local commit.

The result I want is specific: the system knows which work is saved, which work is unfinished, and which attempt is still allowed to change it. That is what turns a sequence of model calls into something I can recover.

The previous article covers which outputs need regeneration after a specification changes. This one covers carrying out that work without throwing away committed progress. For the wider engineering context, see SESCA.