Lab #2553: model method run holds the per-model lock across the datastore pull, so a 40ms method holds it ~70s and any concurrent call hits the 60s LOCK

Defect

swamp model method run acquires the per-model lock before the datastore pull and releases it after the push. On an S3 datastore where the pull takes ~66s, a 40ms method holds the lock for ~70s, which is longer than the 60s lock wait. So any two overlapping calls to the same model deterministically fail one with LOCK_TIMEOUT. The backoff from #520 cannot help once hold time > wait time.

Evidence

swamp 20260925.193456.0-sha.752d3384, @swamp/s3-datastore 2026.09.24.1, one namespace (~98k catalog rows), macOS. Single run, --log-level trace, seconds since start:

t	event
0.8s	`datastore·lock: Acquiring lock`
1.0s	`Syncing model … from datastore...`
67.2s	`Executing method "investigate"`
67.3s	`Completed investigate … succeeded in 40ms`
68.3s	`Wrote catalog export (98407 row(s))`
70.3s	`Committed 14 file(s) to datastore index`

Two concurrent runs of the same model method (both inputs valid):

build	pairs	stagger	result
20260925.193456	1	0s	one `succeeded` (exit 0, 71s), one `LOCK_TIMEOUT … timed out after 63537ms` (exit 1)
20260924.181834	3	0s	same in every pair (timeouts 60.7–65.6s, exit 1)
20260924.181834	1	15s	second run `LOCK_TIMEOUT`
20260924.181834	2	30 and 45 seconds	both succeed

Serial runs: 5/5 exit 0.

Expected

The lock's hold time should be bounded by the method, not by the datastore sync. Either pull before acquiring the per-model lock (re-checking under the lock if needed), or scope the pull to the model being run. Alternatively, make the lock wait configurable from the CLI, or derive it from the observed hold time. The #520 burn-in showed 300s eliminated errors, but I can't find a user-facing knob for model method run.

Also: exit code does not match help

swamp model method run --help says lock_timeout … (exit 75) and 75 = lock contention (temporary — retry with backoff). The losing run above exits 1 with "code": "LOCK_TIMEOUT" in the JSON error, on both builds above, so callers that retry on 75 never retry.

Related (shipped/closed): #520, #812, #829, #1932.

Thanks @sntxrr for reporting this! We shipped: Core-only fix for swamp-club#2553. Map extension-datastore lock timeouts (code LOCK_TIMEOUT from S3/GCS) to the core LockTimeoutError at the single point where extension datastore providers enter core, so model method run and every other lock-taking command exits 75 with code lock_timeout, and serve's LockTimeoutError handling fires. Stop invalidating the whole catalog after a pull that pulled nothing, which removes a full backfill that otherwise runs while the per-model lock is held. Document SWAMP_LOCK_TIMEOUT_MS in help. The unlocked pre-pull the issue suggests is rejected because it can overwrite a concurrent run's unpushed local writes. Bounding the pull itself is extension work (swamp-club#2590).. The fix has been merged and a release is on its way. We appreciate your contribution to swamp.

Terminal title should be conversation summary

dns-records: data schema only has CAA fields, so SRV and other data-carrying records lose their data

Docs: usage-telemetry reference should describe the telemetry stats empty state and the last-flush record

run doctor --fix --server cannot settle damaged runs left by a crashed serve after restart

@swamp/issue-lifecycle: collapse the verification_passed, post_attestation, link_pr sequence

@swamp/issue-lifecycle: one-call verification

audit: register legitimate non-swamp commands as approved bypasses

workflow run: show artifact attributes for runs through serve, and let a run leave out the "Data produced" box

Docs: what serve instances must not share, flat AWS Secrets Manager names, and where OAuth values are stored

Let a serve:* collective key look up usernames, so a recreated swamp serve does not need oauth:manage at runtime

aws-sm vault: optional secret_prefix to scope secrets within an AWS account and region

@swamp/cve http2shell, php-rce-2026-10 and tailworm: scan fails with 'Duplicate data instance name' (log and status share a name)

swamp skill: add a workspace guide and routing-table entry

stagecraft tracker: relate tickets across trackers (another repo's tracker)

serve: a capability probe answered by a follower does not speak for the leader that handles the forwarded request

serve OAuth client reuse: remaining review findings from swamp-club#4264

Agents.md: consider not using H1 'Project' inside managed block

serve: the cron-fire dedup claim has no deadline, so a hung control plane stalls every scheduled fire

serve: define what swamp serve does when its OAuth client is deleted at swamp-club

Docs: schedule catch-up and missed-fire tracking for scheduled workflows

Work-item commands run from the wrong repo report the work item as not started

Docs: step concurrency caps only that step's forEach iterations, not its whole dependency level

Package everything on ops.swamp-club.com; stop direct server edits

workflow run: let a run skip the "Data produced" box, or make it show what each artifact holds

Shorten swamp serve's token session re-check so revocation takes effect sooner

@swamp/cve/emudhra: Ubuntu's pip vendored certifi reported as exposed although pip uses the OS store

cfgmgmt firewall (ufw): an input for rule position (insert before/after)

@swamp/video: informal demos — join multiple takes into one video, and a full-length background music bed

agent-sessions: lockfile test helper silently depends on the captured @swamp/git key

agent-sessions: low-severity edge cases from the Codex importer reviews (#4158)

Team mode manual pages: swamp-serve and remote-execution for leader/follower, member checkouts, offline work, run control and known limits

Agent session record: four questions from the Codex importer (#4158)

@swamp/video: upload workflow to publish a take to YouTube (video, description, chapters, captions, thumbnail)

@swamp/video/brand: customizable outro card (next steps, headline, per-video CTA)

data prune through serve deletes data of an auto-definition whose declared type is invalid

serve: give the definition index a change signal (file watcher with a sync barrier) so a write to a definitions directory costs O(1), not a stat of every file in it

Warn when a grant with a methods list is a deny: it also refuses requests that name no method

serve reload reports 0 types reloaded when it replaced edited repo-local behaviors

@swamp/video/brand: cap each Chromium stderr line kept for errors

serve resolves realpath for thousands of files under .swamp/outputs/swamp/worker/set_status

Retention or GC for auto-definitions whose runs and data are gone

S3 and GCS: after a real commit by another writer, a serve stays off the pull fast path until it restarts

Licence header check only compares the first line of a file

CI: an accepted attestation approves the PR, and anything else fails the attestation job

artifacts.swamp-club.com unreachable in Spain during LaLiga matches (ISP blocking of Cloudflare IPs)

Supported binary file ingestion for remote members of swamp serve, without the bytes landing in run history

Worker queryData fails at the fixed 30 s RPC limit while it waits for a full catalog rebuild

Add expiring service-account token requests and authorization checks to RBAC

Reads other than findById still treat a lapsed temporary ban as active until the operative next signs in

verify-build upgrade gate silently skips models whose version is a constant

@swamp/agent-sessions: explore a decision model (TypeSafe Jev) for call labelling

agent-sessions: follow-ups from the 4211 reviews (repo resolution cache and edge cases)

agent-sessions: revisit the import's remaining review findings (redaction edge cases, import robustness)

Swamp Club collective GUI: show the swamp serve instances currently connected to a collective

A malformed --ca-cert file is reported as an unreadable OS certificate store

TLS hint blames a missing root CA and macOS for every invalid peer certificate failure

workflow cancel --server does not send SWAMP_SERVE_EXTRA_HEADERS

S3/GCS repairUnpushedFiles: the pull it asks for after a lost last-synced index does not stop it re-uploading copies a peer deleted

swamp-uat private-registry tests leave npm/<port> dirs in the shared Deno cache

A resume runs settled work after the teardown gated on it has already run, and nothing tears that work down

serve (token mode): --admins accepts idp-group entries that can never match

build-swamp-extension example: code-review isn't given the quality evidence, so the reviewer reports quality as missing

deployment: getLogs method (deployment-aware pod log tailing)

model.output.search response phase takes 3.9s for 5,000 items with no attribution (redaction vs serialization vs send)

Skill/docs: document registry listing limits, static-extraction rules, and splitting large APIs by domain

Check registry listing limits in extension quality/push and suggest splitting into per-domain extensions

Over-limit contentMetadata publishes a blank extension page; archive fallback extracts nothing

@swamp/video/hyprland: recording hotkeys, retake marker and phone-mirror window rules

@swamp/video/obs: optional OBS recording with per-person hardware config for @swamp/video

serve daemon enable: add Windows support (service or startup task)

swamp skill docs: publishing.md contradicts itself on dynamic imports, overstates where git dependencies are listed, and never says consumers keep the publisher's bundle

extension push: the dynamic import() refusal tells authors to run swamp extension fmt, and the archive-project-file fix names the kind directory push never filters

extension push: file:, link:, workspace: and git package.json dependencies are left out of the unaudited list when the project also has a deno.json

push (Apple Containers) makes an index whose platform manifest is unpullable on Forgejo

Streak projector: incremental runs instead of a full scan every minute

Registry: detect code from outside npm and JSR in a v2 archive itself, not only from the author's dependency-audit.json

Scorecard: code past the analyzer's read caps, or unreadable, is skipped silently instead of counting as not audited

access can-i: evaluate a local grants file without --server

Support search and filters as URL query parameters on /extensions

bundleExtension same-cache test can pass with a broken first-bundle gate under CPU load