← Back to list
01Issue
BugShippedSwamp CLIPublic
Assigneesstack72
Relationships
#4512 A member's behavior run rewrote its read model for 19 minutes (6.9 GB) and ignored SIGTERM (F37 from #4374)
Opened by stack72 · 11/8/2026· Shipped 11/8/2026
Found by the team-mode dogfood on real GCS (swamp-club#4374), in the soak after run 4, release 20261106.094050.0-sha.fcdd7e60. Seen once. It runs in parallel with group 5 (span exporter and telemetry costs). It doesn't block dogfood run 5, which will show whether it recurs, but if the fix is ready in time it goes into run 5's release.
What happened
On member-1 (a team member checkout writing through serve, no storage credentials), the load loop's swamp behavior run ops-writer execute --json ran for over 19 minutes:
| What | Observation |
|---|---|
| Signals | It ran under timeout 900 and didn't exit on SIGTERM. SIGTERM was caught (SigCgt), but the process carried on |
| CPU and state | 62% CPU, 646 s of main-thread CPU time; state D/R, with the main thread in jbd2_log_wait_commit (fsync) |
| Writes | 6.86 GB written in 6.69 M write syscalls, with no growth over the last 10 s |
| Open files | _read_model_v3.db-wal and .db-shm, read_model.lock (twice), and one socket to serve |
| Read model afterwards | _read_model_v3.db 323 MB and its WAL 73 MB; _catalog_v3.db 31 MB and its WAL 28 MB; .swamp/ 1.4 GB in total |
| Effect on other commands | Other commands in the checkout waited or failed while it ran. One reported the checkout offline ("fetch failed") for 4+ min with serve healthy |
Context:
- The checkout had about 25 hours of history, including a 1,000-step
forEachthat stored about 2,000 data items. - The last good
write-opson member-1 was at 21:38:22 UTC on 2026-11-07; the runaway started about 21:42. - It was killed with SIGKILL at about 22:02, after the evidence was saved.
Evidence, on the dogfood VM member-1:
~/stuck-ops-writer/: ps, threads, fds, sockets, the read-model listing,last.json;- the checkout as left:
~/team/.swamp/.
Questions to answer
- What wrote 6.9 GB into a 323 MB read model? Repeated rewriting, either a reset-and-replay loop or one very large catch-up with a sync per row.
- First check the member's logs for the warning #4387 added when the read model is reset from outside, "every commit is applied again". If it's there, find what reset it: this would be an F1-style replay on the member side.
- Then reproduce it on a copy of member-1's
.swamp/and trace the read model's catch-up and commit pattern.
- Why SIGTERM didn't end it. Most likely synchronous SQLite work on the main thread: the signal handler can't run until the event loop is free.
Fix direction
- Remove the cause: no full replay or repeated rewrite of a member's read model during normal commands. A catch-up applies only new commits (#4387's invariant, on the member side). Large catch-ups commit in bounded transactions, not one per row.
- Signals: SIGTERM must end a member command within a bounded time during read-model work. Break long synchronous work into steps that yield, and check for cancellation between them. Leave the read model consistent at a step boundary, so the next command carries on.
- Other commands in the same checkout shouldn't report the checkout as offline while another command holds the read model. They should wait with a clear message, or get a coded busy error.
Tests
- Replaying the evidence: on a copy of member-1's
.swamp/(or a fixture built to match: a long history and a largeforEach), abehavior runfinishes with bounded read-model writes.- Assert on work done: commits applied, transactions, bytes written relative to the read model's size.
- SIGTERM during a large catch-up ends the command, and the next command completes the catch-up from where it stopped.
- A second command during that work gets a clear wait or a coded busy error, not "offline".
Must not regress
- Catch-ups apply only new commits (#4387's invariant).
- No acknowledged write is lost if a catch-up is interrupted.
- Offline queuing under an explicit lease still works.
Done when
- The cause is found and fixed, with a test that fails on current main.
- SIGTERM is effective during read-model work.
- Dogfood run 5's soak shows no runaway.
02Bog Flow
Shipped
Click a lifecycle step above to view its details.
03Sludge Pulse
Sign in to post a ripple.