← Back to list
01Issue
BugTriagedSwamp CLIPublic
Assigneesstack72
Relationships
#4981 Team mode: a follower serve's commits.db WAL is never checkpointed and grows about 1 GB a day (1.7 GB after 2 days), and filled the disk
Opened by stack72 · 11/21/2026
Problem
On the dogfood team's leading serve, the replicated commits.db WAL is not being checkpointed. It grows without bound, and a crash then costs a cold restart over a large WAL.
Evidence (M, dogfood, 2026-11-20, read-only)
| Node | commits.db | WAL |
|---|---|---|
| serve-a (team store) | 352 KB, last written 2026-11-19 21:42 | 767 MB, last written 12:44 the next day, growing |
| serve-b | 189 MB | 63 MB |
| members | — | no WAL |
So serve-a has had no checkpoint for about 15 h.
Context:
- The serve's main open sets
wal_autocheckpoint=0when the store is replicated, because Litestream owns checkpoints. - Other openers use SQLite's default.
- Litestream 0.5.17 ships a full database image after a WAL salt reset, so a checkpoint by anyone other than Litestream also costs a full upload.
Scope (timeboxed, about one day)
- Find the cause first, read-only: is Litestream's checkpoint failing or being skipped (its logs and metrics, its checkpoint thresholds), or is a long-lived read transaction on a serve connection holding the WAL? Which connection? Which serve led at the time?
- Fix only that, with tests. Any change to who checkpoints a replicated store is escalated before it's built.
- One full verify, then one VM run: a soak with both serves on the build, sampling each serve's commits.db and WAL size every 15 min, plus quick.sh for safety.
- At most one fix round. Review suggestions are listed in the PR, not folded.
Stop rule: if the fix needs a new mechanism (for example swamp taking over checkpointing from Litestream), stop and report.
Done when
- The leader's WAL stays bounded over a 4 h soak, below a stated cap tied to Litestream's checkpoint thresholds.
- No full-image re-upload is caused by the fix.
- quick.sh safety unchanged: 0 lost, 0 diverged, 0 failed recoveries.
Correction and a strong lead (M, 2026-11-21)
- serve-a is the follower, not the leader. serve-b leads; this report originally said "leading serve" in error.
- The growing WAL is on the follower's copy: 1,727,025,752 B on 21 Nov 01:5x, against 767 MB a day earlier, so about 1 GB a day. commits.db is still 352 KB, last written 19 Nov 21:42. No checkpoint for about 2 days.
- Lead (I): a follower doesn't run a Litestream sidecar, but its copy may still be opened as a replicated store (
wal_autocheckpoint=0, "Litestream owns checkpoints"). If so, nothing ever checkpoints the follower's WAL. Check the follower's open path and its checkpoint ownership first. - Consequence seen: on 21 Nov the disk filled (with a bench copy alongside). The follower's feed apply failed with "database or disk is full" 62 times over about 13 minutes, and its restores failed until space was freed. Three member commands failed in the last minute of that window (rc 75 ×2, rc 1 ×1), plausibly forwarded through the failing follower (not traced).
02Bog Flow
Triaged
Click a lifecycle step above to view its details.
03Sludge Pulse
Sign in to post a ripple.