Skip to main content
← Back to list
01Issue
BugClosedSwamp CLIPublicTeam
AssigneesNone

Relationships

#1928 S3 datastore: model method run & datastore sync deadlock in commit/lock phase (anon_pipe_read), no self-heal

Opened by kneel · 9/1/2026

Summary

On an S3 datastore (@swamp/s3-datastore, shard-first index), every swamp model method run and swamp datastore sync hangs indefinitely in the post-method datastore commit / lock-release phase. The method's actual work completes — the query/method returns and its output is written to the data artifact — but the process never exits and must be killed at a timeout. This makes the datastore effectively unusable for any write path (all model methods, workflows).

Environment

  • swamp 20260831.202137.0-sha.7401da9d (also reproduced on 20260822.011157.0-sha.06b61472 — updating did not fix it)
  • @swamp/s3-datastore@2026.08.12.1
  • Backend: Hetzner Object Storage (S3-compatible, region hel1), shard-first index (_index/, 311 shards, actively written — newest today). Monolithic .datastore-index.json present but stale (superseded; migration already done, so migrate-index is N/A).

Symptom / process inspection

A hung PID (caught mid-hang, after the query had already written 13.5 KB of correct output):

State: S (sleeping)   Threads: 1
open TCP sockets: (none)
thread wait-channel (wchan): anon_pipe_read

So the process is blocked on an internal pipe (IPC to the datastore sync coordinator) — not on any network socket. swamp datastore sync hangs identically, so the repair path is wedged by the same deadlock and it cannot self-heal.

Stack from the --json lock-timeout envelope seen on concurrent invocations:

S3Lock.acquire (datastore-bundles/.../s3.js)
  -> registerDatastoreSyncNamed (datastore_sync_coordinator.ts:312)
  -> acquireModelLocks (repo_context.ts:1144)
  -> model_method_run.ts:410

Everything downstream is healthy (ruled out)

  • S3 GET / LIST / PUT / DELETE via aws-cli against the same bucket: all ~1s
  • The S3 endpoint and the model's target APIs (Grafana, Coolify) respond in <0.5s over both IPv4 and IPv6
  • swamp doctor install / datastores / extensions: all green (✓ S3 bucket accessible, ✓ Catalog index covers all 0 record(s) on disk, extensions PASS incl. @swamp/s3-datastore)
  • The query itself always succeeds and produces correct output before the hang

Secondary bug: expired locks are not reclaimed

Locks carry ttlMs: 30000. When a process is killed mid-hang it does not release its .lock object, and swamp's acquire does not reclaim these expired locks even though the holder PID is dead and the lock is well past its 30s TTL — the next acquire blocks 60s then errors Lock … held by pid <dead> — timed out after 60000ms. This wedged 7 different models; the only recovery was manually aws s3 rm-ing the orphaned .lock objects. Expected: acquire should reclaim a lock whose acquiredAt + ttlMs < now (and/or whose holder PID is dead).

Trigger / timeline

Shard-first commits worked all day until ~18:24 (last successful shard write). The deadlock began ~20:20 immediately after a model method run was killed while hung mid-commit (a transient network blip — a "telemetry flush failed (network error)" warning appeared around then). This strongly suggests: a commit interrupted mid-write leaves shard/coordinator state that every subsequent commit then deadlocks reconciling, and there is no timeout/recovery on the anon_pipe_read coordinator wait.

Not fixed by

binary update (fresh 7401da9d reproduces), telemetry disable (DO_NOT_TRACK=1 etc.), swamp repo upgrade, sweeping orphaned locks, swamp datastore sync (itself hangs). migrate-index is inapplicable (already shard-first).

Requests

  1. Reclaim expired locks (acquiredAt + ttlMs < now and/or dead holder PID) instead of blocking/erroring.
  2. Add a timeout + recovery to the commit/sync-coordinator anon_pipe_read wait so an interrupted-mid-commit datastore state does not permanently deadlock all subsequent commits.
  3. Ensure swamp datastore sync (the repair path) cannot itself be blocked by the same deadlock.
02Bog Flow
✓OPEN○TRIAGED○IN PROGRESS◉CLOSED

Closed

9/1/2026, 3:27:38 AM

No activity in this phase yet.

03Sludge Pulse
kneel moved this from the private collective Elowish to public9/1/2026, 2:58:53 AM
Editable. Press Enter to edit.

kneel commented 9/1/2026, 3:27:38 AM

CORRECTION — please disregard the "deadlock" framing; closing as not-the-bug-I-described. Root cause found, and it's mostly configuration on my end.

This is not a commit deadlock. The datastore is shared by multiple repos without namespaces, so all writes serialize behind a single global lock — swamp told me directly:

datastore·lock: Lock acquisition took 16612ms — multiple repos sharing this datastore without namespaces serialize all writes behind a single global lock. Run 'swamp datastore namespace set <name>' to scope each repo to its own lock and index

Combined with a large shared index (~14k objects, mostly accumulated report-swamp-method-summary versions), a cold full sync takes ~165s+. My swamp model method run invocations were wrapped in 60–120s timeouts, so I was SIGKILLing the sync before it finished — which produced the apparent "hang," and the SIGKILL is why lock objects were left orphaned. Running swamp datastore sync to completion (as another repo on the same datastore did — 2900 pulled / 11 pushed in 165s) works fine. The anon_pipe_read/no-open-sockets observation is just the main process waiting on the sync-coordinator child doing the slow-but-progressing pull — not a deadlock. swamp datastore namespace set + swamp data gc resolve it.

Two smaller things may still be worth a look (feel free to split/close):

  1. Expired-lock reclaim: lock objects carry ttlMs: 30000, but an acquire against a lock whose holder PID is dead and whose acquiredAt + ttlMs is well past does not reclaim it — it blocks ~60s then errors held by pid <dead>. I had to manually delete orphaned .lock objects to unblock. Reclaiming expired/dead-holder locks would make interrupted runs self-heal.

  2. Timeout/UX asymmetry: swamp datastore sync waits patiently (300s) and streams progress ("Still pulling… elapsed 60s of 300s"), but the sync sub-step inside model method run gives no progress and appears to hang — so a slow cold pull is easily mistaken for a deadlock (as I did). A progress line + the same patient timeout there would prevent that.

Apologies for the noise — thanks.

Sign in to post a ripple.