Relationships
#4101 swamp serve (format 2, S3): the shutdown push of interrupted runs can resolve having sent nothing, so no unpushed-interrupts marker is written and the run stays `running` (follow-up to #3958)
Opened by vcjdeboer · 10/29/2026· Shipped 10/30/2026
Follow-up to #3958 (shipped in 20261027.021509.0-sha.f96841b5). Its fix works when the shutdown push fails visibly. This issue is the case it does not cover: a push that resolves without sending the interrupted record. Related: #3252 (the original missing flush), #3961 and #3962 (the format 3 versions, found while triaging #3958). This supersedes our two ripples on #3958 (2026-10-28 and 2026-10-29). They say the shutdown push "never commits", and that reading was wrong (see "Correction").
Setup. swamp 20261027.021509.0-sha.f96841b5 (linux/aarch64, systemd unit, default TimeoutStopSec). @swamp/s3-datastore 2026.10.24.2 (also seen on 2026.10.01.1). Datastore format 2 (synced) on S3, bucket versioning on, no namespace. Remote-only, one remote worker, default 30 s drain, log level info.
Steps.
- Start a multi-step workflow run whose steps execute on the remote worker (steps of about 20-30 s).
- While a step is running:
systemctl restart <serve unit>(SIGTERM). - Once the new serve is ready:
swamp workflow history get <workflow> --run <id>, and list the S3 object versions of the run's record.
Expected. The run reads interrupted, either pushed at shutdown or, if that push failed, pushed at the next boot from .swamp/serve-unpushed-interrupts.json.
Actual, in 2 of 3 restarts. The run stays running indefinitely, and only an admin workflow cancel settles it. The bucket holds no version with status: interrupted. The old serve logs Interrupted workflow run "<id>" (server shutdown) and then no Failed to push interrupted workflow run(s) ... line. No marker is written, and the new serve logs no Boot: pushed ... and no skip or reap line for the run. The trimmed journal is in our 2026-10-28 ripple on #3958.
The third restart worked. It had two runs in flight. The shutdown push logged Failed to push interrupted workflow run(s) ..., the next boot reaped both runs, and both settled interrupted. So the marker and boot path is fine. Only the silent case is left.
Correction (from reading the f96841b5 source). Preparing push (uploading files)... / Committing index update... come from flushTwoPhasePush (src/cli/repo_context.ts:3760, 3806), not from the shutdown flush. The shutdown flush (src/serve/shutdown_interrupt.ts:157-245) calls pushNamespaceCounted and logs nothing when it resolves. In the journal, a two-phase push started at 08:47:45, after the interrupt, and the process exited at 08:47:46 before that push logged its commit.
Where it can go silent.
pushNamespaceCountedreturns the number of files sent (src/infrastructure/persistence/push_paths.ts:80-88), but the shutdown flush discards it.- The marker is written only when
pushErroris set or a save or load failed (shutdown_interrupt.ts:241-289). A push that resolves with 0 files therefore counts as success.
Question. Can the shutdown push see nothing to send because the concurrent two-phase push (a lock release during the abort) had already taken the dirty files into its prepare manifest, and was then dropped at exit before its commit?
Possible fix. Treat a shutdown push that sends no files while runs were interrupted as unpushed, and write the marker. The managed lockfile publish already fails a push that sent nothing (#3192).
We can repeat the restart with --log-level debug and send the [s3-sync] lines.
Shipped
Click a lifecycle step above to view its details.
zack commented 10/30/2026, 3:33:26 PM
Thanks @vcjdeboer for reporting this! We shipped: An aborted run whose executor outlives serve's abort grace keeps saving its in-memory running copy of the run after shutdown has saved it interrupted, so the shutdown push uploads running and the datastore keeps the run running. Add a run write fence: the executor's saveRun skips saves of a run whose fence is closed, and shutdown closes it for the runs it interrupts (waiting briefly for saves in flight) before loading and saving them. The shutdown push's 0-count check stays as insurance; the boot re-save from the earlier attempt is dropped.. The fix has been merged and a release is on its way. We appreciate your contribution to swamp.
Sign in to post a ripple.