Skip to main content
← Back to list
01Issue
BugOpenSwamp CLIPublicTeam
AssigneesNone

Relationships

#4673 datastore push: a dropped push after a reported-success run isn't persisted (run still exits 0); a wedged push holds the datastore lock until the process is killed

Opened by kneel · 11/12/2026

Running the @webframp/azure-blob-datastore backend on a busy repo, we're seeing two core-side behaviors around the trailing datastore push. The extension itself looks correct on these points (it propagates the push rejection, and its lock has a working lease + heartbeat); the behaviors below are in swamp's push orchestration.

(1) A dropped push after a reported-success run is silently not persisted. At the end of an otherwise-succeeded method run, the push fails (push failed: fetch failed + Failed to release locks during cleanup), but the CLI has already reported success and exits 0. The write is not persisted: an immediately following read shows the record missing; re-running persists it. The extension does surface the failure (its withSpan rethrows and the push path has no swallowing catch), so the rejection reaches core — but core reports run success before/independently of the push outcome. Expected: a failed datastore push should fail the run (non-zero exit or a structured error), since the local state is fine but the remote record is missing and the caller otherwise has no signal.

(2) A wedged push holds the per-instance datastore lock until the process is killed. Separately, a push wedged for 15+ minutes: the process sat idle in ep_poll, ~0 CPU, with zero open TCP sockets — i.e. it was not waiting on a network request, but on some non-socket await in the push/sync orchestration. While wedged it held the per-instance datastore lock: concurrent writers failed with Lock timeout after 60000ms on key: data/<type>/<instance>/.lock, and it only recovered on SIGTERM (after which the lease auto-expired in ~19 s and the next writer proceeded). The reason the lock stayed held: the acquire → push → release sequence is orchestrated in core, and release runs in a finally after the push promise settles — which never happens while the push is wedged — while the extension's lock heartbeat keeps the lease alive for the live process. Expected: an overall deadline on the push leg; on expiry, release/force-release the lock and fail the run, so a wedged push frees the lock without a manual kill.

Scope note. The extension (webframp/swamp-extensions) separately lacks a per-request timeout on its fetch (reported there), which explains a different occurrence that stalled ~18 min and then self-terminated via the kernel TCP timeout. But occurrence (2) above was not on a socket — it is a non-settling await at the orchestration level — so a fetch timeout in the extension would not cover it. Hence this separate report.

Environment. swamp 20261111.001635.0-sha.34b6477f (occurrences above) and 20261111.234122.0-sha.cd35f58c, @webframp/azure-blob-datastore v2026.11.10.3, Linux x86_64, Azure Standard_ZRS. Not deterministically reproducible — looks connection-drop-triggered; a few occurrences in ~a day under frequent small pushes. Happy to add logging or run a patched build.

02Bog Flow
◉OPEN○TRIAGED○IN PROGRESS○SHIPPED

Open

11/12/2026, 6:15:19 AM

No activity in this phase yet.

03Sludge Pulse
kneel moved this from a private collective to public11/12/2026, 6:29:31 AM
Editable. Press Enter to edit.

kneel commented 11/12/2026, 6:30:16 AM

Neil thinking out loud: this may be happy-eyeballs related, if my IPv6 path is going bad for some reason. I will also investigate.

Sign in to post a ripple.