Lab: Redundant push after pullChanged: localMtime clobbered by pullIndex(forceRemote)

Problem

The first explicit `pushChanged` call after a `pullChanged` against a populated remote re-uploads every just-downloaded file with byte-identical content. Discovered during verification of swamp lab #220 / PR https://github.com/systeminit/swamp/pull/1290 — was previously masked by the coordinator's implicit pull+push fast-pathing past the explicit ones, became visible after #220 decoupled `swamp datastore sync` from the coordinator's implicit phase.

Not a correctness issue (data is byte-identical and the system reaches steady state after the first redundant push), but a bandwidth waste of N S3 PUTs once per fresh-cache bootstrap, and visible to users as a confusing `filesPushed: 17` in sync output immediately after a successful `filesPulled: 17`.

Reproduction

Run a workflow N times against an S3 datastore from machine A, populating the bucket.
On a fresh machine B, run `swamp datastore setup extension @swamp/s3-datastore --skip-migration` against the same bucket. After PR #1290 this hydrates the cache and reports `filesPulled: 17`.
Run `swamp datastore sync --push` immediately. Reports `filesPushed: 17` even though the cache already matches the remote — those exact files were just downloaded.
Run `swamp datastore sync` (default) again. Now correctly reports `0/0` — steady state reached after the redundant push completes and rewrites the index with proper localMtime values.

Verified end-to-end against MinIO.

Root cause (traced through extensions/datastores/_lib/s3_cache_sync.ts)

`pullChanged` (around line 808-816) sets `localMtime` on each downloaded entry IN MEMORY only. When `pulled > 0`, the sidecar isn't updated (line 843 only calls `markSynced` for the zero-diff case), and the local `.datastore-index.json` file isn't rewritten — so those `localMtime` values live exclusively in the `S3CacheSyncService` instance's in-memory state.

The next `pushChanged` (around line 920) calls `pullIndex({ forceRemote: true })`, which inside `pullIndex` (line 691-692) does `this.index = JSON.parse(text)`. The remote payload doesn't carry `localMtime` (those are local-only state), so the assignment overwrites the in-memory index and wipes the values `pullChanged` just set.

The walk in `pushChanged` (line 941-955) then sees `existing.size === stat.size` true and `existing.localMtime === undefined` — which SHOULD hit the `continue` at line 951 and skip the file. Empirically it doesn't, and the file ends up in `toPush`. I haven't fully resolved why; possibilities include a control-flow read I'm missing, or an interaction with `scrubIndex`/`indexMutated` that I haven't traced. Either way the observable effect is N redundant uploads.

Suggested fixes (pick one)

Persist `localMtime` after `pullChanged` — write the in-memory index back to disk at the end of a non-zero pull, so the values survive across `pullIndex(forceRemote)` calls. Cleanest semantically.
Merge `localMtime` values when `pullIndex(forceRemote)` overwrites the index — preserve the in-memory `localMtime` map across the refresh, since it's local-only state the remote can't authoritatively replace.
Track downloaded files in a separate just-pulled set within the service instance and have `pushChanged` skip them on the first call after a pull.

Options 1 and 2 fit the existing fast-path / sidecar design more cleanly.

Context

Discovery: swamp lab #220 / PR https://github.com/systeminit/swamp/pull/1290 (now shipped).
Cross-link: `design/datastores.md` already documents the markDirty + sidecar contract; this issue is consistent with that design but reveals a gap in the pull→push handoff.
Repro harness from #220 (MinIO + scratch repos A/B, populate bucket, observe `filesPushed` mismatch) is reusable.

Upstream repository: https://github.com/systeminit/swamp-extensions

Environment

Extension: @swamp/[email protected]
swamp: 20260501.234710.0-sha.f1687b62
OS: darwin (aarch64)
Deno: 2.7.14
Shell: /bin/zsh

extension unyank needs -y/--yes flag for non-interactive use

Step key parsing in execution_service uses naive split(":") and silently truncates colon-containing step names

Workflow-scope report artifacts unreachable via `swamp data get --workflow`

Add --stdin support to method run and workflow run for Unix pipe composition

Docs: add run namespace to CEL expressions reference

Fix invalidate-then-reconcile sequencing in doctor extensions; failure-mode RowStates unreachable in sourceDetails

doctor extensions invalidateAll does not trigger fingerprint recheck for existing Indexed rows

Extension failure recording has dual write paths (legacy buildIndex vs W3 reconcile)

Expose workflow run ID in CEL so resource keys can be run-scoped

Add `swamp doctor workflows` subcommand to surface workflow YAML parse errors during preflight

Add 'swamp issue comment' command for updating existing issues

Close #327 — fixed in 20260511.160514.0-sha.9d03b09a

Nested workflow task fails with 'Bad resource ID' when workflowIdOrName uses @collective/ prefix

Detect stale skill directories and prompt for repo upgrade

Improve skill trigger routing for cross-model edge cases

Add type search for driver, datastore, and report kinds

Surface ReconcileFromDisk dryRun transitions in swamp doctor extensions

Docs: update doctor extensions reference for W6 aggregate-state rendering + repair flags

swamp issue: check if reporter is on an outdated binary before opening

Implement W6: swamp doctor extensions aggregate-state rendering + repair surface (extension catalog rearchitecture)

extension pull fails referencing removed source after extension source rm

Investigate whether allExtensionMethodsAttached guard can be removed via registration-path consolidation

Trace context is lost: CLI ignores inbound TRACEPARENT and raw driver doesn't propagate active context into in-process methods

swamp model create --global-arg KEY=VALUE doesn't coerce strings to z.number() schemas

swamp model create --global-arg KEY=VALUE doesn't coerce strings to z.number() schemas

swamp issue get should rate-limit unauthenticated users instead of blocking

swamp issue get should not require authentication

telemetry: emit child entries for follow-up action method invocations

Publish release-candidate / unstable extension versions

extension source rm leaves stale @local/. rows in catalog, blocking later pulls of registry types

extension push drops binaries: field from re-emitted archive manifest.yaml

macOS launchd autoupdate (club.swamp.autoupdate) silently fails — binary stays stale

Clicking @hivemq/honeycomb extension card on /extensions shows 'Something went wrong'

Docs: Update model-definitions.md and workflows.md for direct type execution

Docs: document binaries manifest field in extension-manifest.md

Add an official @swamp/ssh extension for general-purpose SSH (brownfield-friendly)

Accept and display binaries field from extension push metadata

Direct type execution: collapse model create + method run into one command

Per-method telemetry events for workflow runs

Workflow run liveness: orphaned 'running' records when originating CLI process dies mid-run

Provide a CLI-shape primer for AI agents to reduce rediscovery overhead

quality rubric: don't penalize extensions whose upstream constrains them to a single platform

Extension update rejects multiple .ts files extending the same target type within one local extension (regression)

swamp extension push deadlocks when invoked from inside a swamp workflow step

Collective-scoped auth keys + OIDC federation for CI publishing

forEach self.* in modelIdOrName not resolved in runtime execution path

Award leaderboard points for referrals and collective invites

Add agent harness detection and AiTool to telemetry

Workflow-level runtime expressions (env.*, vault.*) not resolved in driverConfig — docker driver receives literal ${{ ... }} strings

Implement W5: Per-fingerprint import URLs + subprocess test harness (extension catalog rearchitecture)

swamp config set crashes with YAML serialization error

datastore compact: VACUUM fails in compiled binary (SQLITE_LIMIT_ATTACHED=0)

Repo-level version gating: minSwampVersion high-water mark for team consistency

Docs: document self.* expressions in modelIdOrName during forEach

Docs: How-to guide for background autoupdating

Manifest version bumps silently ignored for existing local extension aggregates

materialiseExtensions misclassifies pulled rows when manifest name collides with a pulled extension

Local extension edits don't reliably trigger rebundle

Missing unique indexes on user.email and user.username allow duplicate users

Resolve self.* expressions in modelIdOrName during forEach expansion

discord-bot double-sends sign_up notifications

Discord bot sends duplicate signup notifications

Add 'swamp workflow list' as alias for 'swamp workflow search'

Add 'swamp auth status' as alias for 'swamp auth whoami'

Extension bundle cache does not invalidate on source edits

extension pull fails on @local/[email protected] phantom-claim collision when local repo has its own extension

Lab search by numeric issue ID returns no results

W3 sourceToRow writes empty source_mtime — should carry filesystem mtime through Source entity

Warm-start rebundleAndUpdateCatalog should respect terminal RowStates set by reconcile

Implement W4: KindAdapter + unified loader (extension catalog rearchitecture)

Docs: add vault read-secret command to reference manual

Extension layer garbage collection: prune catalog rows + evict orphaned bundles

Docs: document workflow concurrency limits in reference manual

Locally-sourced extension: source_mtime updates without regenerating stale bundle

Extension push: allow shipping executable host helpers (bin/mudroom blocker)

Vault expressions silently deliver __SWAMP_VSEC__ sentinels under the docker driver

First-class shell-shim support for extensions, with registry-level visibility

Local extension model bundles don't rebuild when source changes (no rebuild CLI; manual cache delete breaks the runner)

Configurable concurrency limits for workflow fan-out (forEach, parallel jobs/steps)

feat(security): redact sensitive method arg values from audit log

Workflow-level runtime expressions (env., vault.) not resolved in driverConfig — docker driver receives literal ${{ ... }} strings