Skip to main content
← Back to list
01Issue
BugClosedSwamp CLIPublic
Assigneesstack72

Relationships

#4854 Stopped: Team mode: a member's v3 copy keeps growing under steady load even with follow-serve's-compaction (about 16 MB/h, no plateau)

Opened by stack72 · 11/18/2026

Problem

A member's copy of a v3 team datastore (its local commits.db) grows without bound under steady load, even with "follow serve's compaction" shipped (the base-install-by-partition change, section "v3 team-mode copies: following serve's compaction, catch-up by difference and bounded recovery" in the datastore design document). That change exists to keep a member's copy bounded.

Evidence (M, dogfood 4 h soak, 2026-11-16 18:33–22:35 UTC)

Setup: three members and both serves on the build, with load on all members plus serve-driven runs. Serve folds hourly.

commits.db per member, MB:

Minutes in 0 15 61 122 183 241
member-1 0.2 19 32 51 73 84
member-2 0.2 19 32 57 72 87
member-far 0.2 19 32 47 63 77
  • Growth is about 15–17 MB/h per member, close to linear, with no flattening after any of the three serve folds.
  • Serve folds: 5.3 s, 27.4 s and 26.4 s (604, 1,972 and 1,939 commits).
  • The follow step (swamp.datastore.v3.follow_compaction, 2,630 commands): outcomes recently-idle 2,044, below-threshold 486, followed 100; refused 0.
    • Installs: 511 partitions (506 whole, 1 by difference). Deferred: 5,948. Rows written: 22,059.
    • Budget used: p50 2 ms, max 1,045 ms.

Likely causes to check (I, from the design as merged)

  1. Hot partitions switch to install-by-difference only after a copy's next full base load. These copies never took one, so their hot partitions are deferred.
  2. Deferred installs were meant to run in a "first command after absence" batch, which was moved out of scope. Nothing drains the deferrals under steady load.
  3. A copy keeps each installed partition's commits down to its latest snapshot (up to about 1,000 commits per partition), so deletion may lag far behind serve's fold.
  4. "recently-idle" skips most commands; the idle recheck may run too rarely under load.

Scope (timeboxed, about one day)

  1. Find the cause first, from code and the soak's span data. No code before it's named with evidence. Answer: which of 1–4 (or something else) keeps the copy growing, and by how much each contributes.
  2. Fix only that, with tests. One full verify.
  3. One VM run: a 4 h soak with members and serves on the build, sampling each member's commits.db every 15 min, with the follow and compact spans; plus quick.sh for safety.
  4. At most one fix round, only for a miss against the done criteria. Review suggestions are listed in the PR, not folded.

Stop rule: if the cause needs a new mechanism (for example the full first-command batch), stop and report before building it.

Done when

  • Each member's commits.db plateaus within the 4 h soak (growth in the last 2 h under 10 % of the first 2 h's).
  • No command exit over 1 s, and the follow step stays inside its 1 s budget.
  • quick.sh safety unchanged: 0 lost, 0 diverged, 0 failed recoveries.

Stopped by the stop rule: outcome not met

Every fix we tried made command exit slower and left the copy no smaller. Nothing from this issue ships. The branch 4854 (head d7ca8731, not pushed) stays as a reference.

Cause (M, from the 2026-11-16 soak's own copy and serve-a's store, read-only)

On member-1, commits made up 65 MB of the 84 MB commits.db. Of that, 40.5 MB sat between the copy's compacted seq and serve's, in about 7 hot command/shell and workflow partitions at about 17 KB a commit. For example, partition 4910cbd6 was compacted at seq 15 on the copy and at 454 on serve. Three things kept the follow step from reaching those partitions:

  • (a) Order. Installs ran smallest partition first. Each fold leaves 150-184 tiny run partitions, and they used up every 1 s budget.
  • (b) A follower's index. A follower serve answered GET base/index from its own copy, which folds behind the leader. A member that read it found nothing to do and waited 10 minutes. About 19 of each member's 33 runs that read the index did this. This only delays installs; it never produces wrong data.
  • (c) History kept to the snapshot. Installs by difference and by staging kept commits down to the latest snapshot. Snapshots come every 1,000 commits, so that is up to about 17 MB per hot partition.

What we tried, and the 4 h soaks (serves and members on the build; quick.sh safety 0 lost / 0 diverged / 0 failed recoveries every time)

The first fix:

  • installed partitions by the commit bytes each frees;
  • had a follower serve pass base reads to the leader;
  • never marked the copy idle from an index behind its own sources;
  • made difference and staged installs snapshot at the installed seq and fold through it.

The fix round:

  • cut a difference read at half a budget and fell back to staging, remembering the fallback;
  • started the replay estimate at 5 ms a commit;
  • let a bounded install snapshot part of the way.
baseline (3ff743ab) first fix (2b1d1959) fix round (d7ca8731)
commands spending > 100 ms following 2.4 % 16.4 % 17.5 %
commands over 1 s 28 233 135
follow step p95 5.6 ms 1,029 ms 1,006 ms
follow time over about 2,600 commands 53 s 416 s 404 s
members' commits.db at the end 77-87 MB 70-77 MB 90-91.5 MB
serve-a's own store at the end 103 MB 84.8 MB 86.5 MB

The first fix also livelocked member-far: every command read the same hot partition's difference over the far link and installed nothing for two hours. The fix round removed that (279 installs). It still left 716-839 history-only commits per copy, because a 1 s budget replays about 190 commits and hot partitions get 400-600 between serve's folds.

The goal, re-baselined

"Plateau in 4 h" cannot hold while the data itself grows. Under this load serve-a's own store grew too: every command adds run and output records, and serve keeps a 2,500-commit tail and folded only twice in 4 h. In every soak the copy ended at or below serve's store. The invariant to hold is that a member's copy ≈ serve's base + a bounded tail, never a multiple of serve's store. The metrics are:

  • the copy's bytes against serve's base plus its tail;
  • the commit bytes between the copy's compacted seq and serve's for each partition, which is the part that grew without bound (the 40.5 MB above).

Why command exit is the wrong place, and the direction for later

At the same total size, every variant cost more exit time. Keeping hot partitions caught up does not fit in a command's 1 s exit budget. For the next round of planning, not filed:

  • do the follow work outside the command path: the deferred first-command batch, or background work in a long-lived process where one exists (serve-driven runs, workers);
  • serve-side, so there is less to follow: ship ready-made per-partition snapshots with base/partition, and fold by tail bytes rather than commit count;
  • record (a)-(c) as design input. The follower-index and idle findings cost latency only.

Evidence on dogfood-ctl:

  • soak results: ~/dogfood/results/soak/soak-20261116T183300Z, soak-20261118T051557Z, soak-20261118T153325Z;
  • member copies on member-1 and member-far: ~/team-old-31, -36, -39;
  • serve stores on serve-a: ~/team-old-31, -36, -39.
02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS◉CLOSED+ 1 MOREASSIGNED+ 5 MOREREVIEW+ 12 MORECODE_CONFORMANCE_REVIEW

Closed

11/19/2026, 5:47:22 PM

No activity in this phase yet.

03Sludge Pulse
stack72 assigned stack7211/18/2026, 2:46:48 AM
Editable. Press Enter to edit.

stack72 commented 11/19/2026, 5:47:21 PM

Closing: stopped by the stop rule, outcome not met. Every fix we tried made command exit slower (follow p95 5.6 ms to about 1 s; 2.4 % to 17.5 % of commands spending over 100 ms) without making the copy smaller, so nothing ships. The cause, the three-soak table, the re-baselined goal (copy ≈ serve's base + a bounded tail) and the direction for next round (follow work outside command exit; serve-side partition snapshots; folds by tail bytes) are in the body. Branch 4854 (d7ca8731, unpushed) is kept as a reference.

Sign in to post a ripple.