#146

feat: add support for Nix via a flake

s12d ago

← Back to list

01Issue

FeatureShippedSwamp CLI

Assigneesstack72

Relationships

#911 Add swamp datastore sync --migrate-index CLI command to trigger shard-first migration

Opened by stack72 · 7/1/2026· Shipped 7/1/2026

Summary

swamp-club#906 added shard-first indexing to @swamp/s3-datastore and @swamp/gcs-datastore. Both extensions expose a public migrateMonolithToShards() method on their sync service, but nothing in the CLI calls it.

When commitPush detects no v2 _meta.json, it falls back to the monolith path and logs:

Index is using monolithic format. Run 'swamp datastore sync --migrate-index' to enable shard-first commits for improved concurrency.

That command does not exist yet. Users see the hint but cannot act on it.

What needs to happen

Add a --migrate-index flag to swamp datastore sync (or a dedicated swamp datastore migrate-index subcommand) that:

Acquires the distributed lock
Calls syncService.migrateMonolithToShards() on the active datastore extension
Reports the result (the method logs "Migrated N entries into M shards in Xms")
Releases the lock

The method is already implemented and tested in both extensions. This is purely CLI plumbing.

Interface

The sync service already advertises twoPhaseSync: true via capabilities(). The migrateMonolithToShards method is public on both S3CacheSyncService and GcsCacheSyncService. Core needs to:

Check if the sync service has a migrateMonolithToShards method (duck-type or add to the DatastoreSyncService interface in the extension)
Call it under the lock
Surface the console.info log lines the method emits

#906 - shard-first index extension changes (merged)
#812 - original lock contention issue
#829 - two-phase sync

02Bog Flow

Shipped

7/1/2026, 11:35:44 PM

Click a lifecycle step above to view its details.

03Sludge Pulse

stack72 assigned stack727/1/2026, 9:29:47 PM

Scorecard: beta/rc-only extensions show "not scored" despite successful scoring (GET /score resolves only latestVersion)

`swamp extension search --channel beta|rc|stable` ignores the filter — all channels + the no-flag default return an identical result set

Beta-channel extensions are never scored — perpetual 'Scoring is taking longer than expected'

feat: allow models to call other models mid-execution via context.runModel()

swamp-club: enable OAuth provider for swamp serve collective-based admission

extension update/install reports new version while on-disk pulled files stay old

command/shell: step with timeout + a surviving child process reports exit -1 despite command exiting 0

swamp serve startup/shutdown is very slow on large repo and lacks diagnostics

Vault push log overstates what left the machine: Pushed N vault file(s) to datastore counts every dirty file, not just vault files

serve-auth: mode: oauth — device grant client + collective-based admission

serve-auth: enable OIDC Provider + Device Authorization plugins in swamp-club

Orphaned data (migrated-away/deleted model) can't be GC'd or deleted

Docs: document swamp data prune (orphaned-data reclamation)

UAT: CLI coverage for swamp data prune (orphaned-data reclamation)

Session credential revocation scans entire credential map on every dispatch completion

Docs: update worker-commands.md for --all flag and JSON output change

datastore migrate-index aborts on hardcoded 30s S3 requestTimeout for large shards

swamp worker list: hide disconnected workers by default, add --all flag

swamp serve should survive bad extension failures

Worker fleets phase 7: how-to guides and explanation docs

swamp serve: catch unhandled promise rejections from extensions

[@adam/cfgmgmt] SSH helper can crash swamp serve via unhandled finally() rejection

Docs: add --server-token flag to worker connect reference

Add --server-token to worker connect for authenticated orchestrators

Docs: document --trusted-hosts flag for swamp serve

Document SWAMP_SERVE_EXTRA_HEADERS for reverse proxy/tunnel setups

Add --trusted-hosts flag to swamp serve for Docker/Kubernetes worker connections

Allow swamp clients to send custom headers when connecting to swamp serve

Docs: update serve-flags.md default queue timeout from 60s to 10m

Worker fleets phase 6: raise default queue timeout to 10 minutes

Worker fleets phase 4b: capacity slots and concurrent dispatch

MethodContext docs/comments mention context.putSecret but runtime exposes vaultService.put

Docs: add --concurrency flag and SLOTS column to worker-commands reference

install.sh still points users to systeminit/swamp

Support user systemd units for 'swamp serve daemon enable'

swamp serve scheduled execution runs steps in an isolated data scope: readResource sees empty state, writeResource never reaches the model-scope chain — stateful scheduled workflows duplicate instead of deduplicating

Support installing swamp via mise GitHub backend

Worker fleets phase 4a: dispatch runner extraction — child process per dispatch

Docs: add swamp worker daemon enable/disable/status to worker-commands reference

Worker fleets phase 3 PR2: swamp worker daemon enable

Docs: add drain, lifecycle policies, and exit-code contract to references

Worker fleets phase 3 PR1: drain state machine, lifecycle policies, signal handling

Leaderboard search excludes ghosts — a visible ghost can't be found by its alias

issue-lifecycle: plan lifecycle entry silently dropped when summary exceeds 1000 chars

Docs: add worker verify, --verify-on-enroll, and env-var config to references

Labs - Filters do not apply to Blocked status and Billing Tier category

Worker fleets phase 2 PR2: env-var config, swamp worker verify, --verify-on-enroll

Docs: add --type flag to issue edit reference and how-to

Docs: update manual with whole-field expression syntax for workflow task inputs/globalArgs

Allow changing an existing issue to security/confidential after submission

Allow dynamic object expressions for workflow task inputs/globalArgs

Docs: update enrollment token references for fleet tokens (--max-enrollments)

Worker fleets phase 2 PR1: fleet tokens (--max-enrollments, bindings, worker naming)

$identify events are never consumed — anonymous web history never binds to the account

Worker fleets phase 1 PR3: boot reconciliation and integration test

Login success still shows login tip

Support confidential visibility for non-security issues

Fine-grained scopes for swamp.club access tokens

Add associative info to user profiles: collective membership + collaborative contributions

Remove comments from /feed and redesign the page

feed_post_score_events queue grows unboundedly (no TTL, no delete-on-delivery)

Make paid subscribers visually stand out (badge, profile popup summary)

Relax the profanity filter

Giphy integration via /giphy command

@-mention autocomplete for users when writing lab ripples

Worker fleets phase 1 PR2: step_queued events, pending-dispatch model, queue introspection

Docs: add swamp worker queue to worker-commands reference

/lab shipped list is sorted by issue number, not ship date

Docs: update manual for ephemeral data and step-output dependency deferral

Docs: update remote execution references for elastic queueing

Support dynamic host discovery from external sources

Worker fleets phase 1: elastic queueing

catalog backfill should support scoped population for targeted lookups

Docs: document global skill sync behavior for custom tools in manual

swamp update syncs global skills to default agent directories instead of configured custom Pi skillsDir

coerceToSuffix does not sanitize path-unsafe characters, causing forEach items and vary dimensions with / to produce invalid data instance names

Badge for activity on other people's issues

swamp serve: add @keychain=<service>/<account> indirection for secret args on macOS

swamp serve: eliminate wrapper scripts via @vault= secret refs and native startup hooks

Automate mechanical frontmatter fixes in the fixing stage