Skip to main content
← Back to list
01Issue
BugTriagedSwamp CLIPublic
Assigneesstack72

Relationships

#4981 Team mode: a follower serve's commits.db WAL is never checkpointed and grows about 1 GB a day (1.7 GB after 2 days), and filled the disk

Opened by stack72 · 11/21/2026

Problem

On the dogfood team's leading serve, the replicated commits.db WAL is not being checkpointed. It grows without bound, and a crash then costs a cold restart over a large WAL.

Evidence (M, dogfood, 2026-11-20, read-only)

Node commits.db WAL
serve-a (team store) 352 KB, last written 2026-11-19 21:42 767 MB, last written 12:44 the next day, growing
serve-b 189 MB 63 MB
members — no WAL

So serve-a has had no checkpoint for about 15 h.

Context:

  • The serve's main open sets wal_autocheckpoint=0 when the store is replicated, because Litestream owns checkpoints.
  • Other openers use SQLite's default.
  • Litestream 0.5.17 ships a full database image after a WAL salt reset, so a checkpoint by anyone other than Litestream also costs a full upload.

Scope (timeboxed, about one day)

  1. Find the cause first, read-only: is Litestream's checkpoint failing or being skipped (its logs and metrics, its checkpoint thresholds), or is a long-lived read transaction on a serve connection holding the WAL? Which connection? Which serve led at the time?
  2. Fix only that, with tests. Any change to who checkpoints a replicated store is escalated before it's built.
  3. One full verify, then one VM run: a soak with both serves on the build, sampling each serve's commits.db and WAL size every 15 min, plus quick.sh for safety.
  4. At most one fix round. Review suggestions are listed in the PR, not folded.

Stop rule: if the fix needs a new mechanism (for example swamp taking over checkpointing from Litestream), stop and report.

Done when

  • The leader's WAL stays bounded over a 4 h soak, below a stated cap tied to Litestream's checkpoint thresholds.
  • No full-image re-upload is caused by the fix.
  • quick.sh safety unchanged: 0 lost, 0 diverged, 0 failed recoveries.

Correction and a strong lead (M, 2026-11-21)

  • serve-a is the follower, not the leader. serve-b leads; this report originally said "leading serve" in error.
  • The growing WAL is on the follower's copy: 1,727,025,752 B on 21 Nov 01:5x, against 767 MB a day earlier, so about 1 GB a day. commits.db is still 352 KB, last written 19 Nov 21:42. No checkpoint for about 2 days.
  • Lead (I): a follower doesn't run a Litestream sidecar, but its copy may still be opened as a replicated store (wal_autocheckpoint=0, "Litestream owns checkpoints"). If so, nothing ever checkpoints the follower's WAL. Check the follower's open path and its checkpoint ownership first.
  • Consequence seen: on 21 Nov the disk filled (with a bench copy alongside). The follower's feed apply failed with "database or disk is full" 62 times over about 13 minutes, and its restores failed until space was freed. Three member commands failed in the last minute of that window (rc 75 ×2, rc 1 ×1), plausibly forwarded through the failing follower (not traced).
02Bog Flow
✓OPEN◉TRIAGED○IN PROGRESS○SHIPPED+ 1 MOREASSIGNEDCLASSIFICATION

Triaged

11/21/2026, 2:29:02 AM

Click a lifecycle step above to view its details.

03Sludge Pulse
stack72 assigned stack7211/21/2026, 2:27:03 AM

Sign in to post a ripple.