Lab #3955: Filesystem datastore: an unclean shutdown leaves zero-byte data versions as the latest version, and `doctor datastores --repair` does not detect them

Description

The filesystem datastore does not write data versions atomically. An unclean shutdown during a write leaves the version directory with an empty raw file and an empty metadata.yaml file. The latest pointer file and the catalog row (is_latest = 1) already point to the empty version.

After this:

swamp data query returns the record with content: {}. CEL that reads a field fails, for example No such key: stageId. Every workflow that queries the spec fails.
swamp data get <model> <name> returns not_found.
swamp data versions does not list the empty versions, but the catalog keeps them.
swamp doctor datastores reports pass. --repair reports "Catalog index matches data on disk". The check only compares disk to catalog. It does not check for empty content, and it does not check for catalog rows whose files have no content.

No CLI command can recover the record. swamp data delete removes all versions of a record, not one version. No command rebuilds the catalog from disk.

Observed impact

One unclean shutdown made 168 files zero bytes. All files had the same mtime. 26 records had an empty latest version. These records included the state, journal, status and evidence records of three active @swamp/software-factory work items. The work items stopped for two days with no error until a workflow evaluated a field of the empty record.

The @swamp/workflow-summary report for the run that wrote the empty version shows succeeded.

Steps to reproduce

Run a method that writes a new version of a resource. Stop the VM or host before the OS flushes the file data to disk, for example wsl --shutdown on WSL2. (This step is not verified. The observed failure came from an unplanned stop at an unknown point in the write.)
Run swamp data query '<predicate that matches the record>' --json. Result: content: {}.
Run swamp data get <model> <name>. Result: not_found.
Run swamp doctor datastores --repair. Result: no findings.

Manual recovery that worked

Move the empty version directories out of the record directory.
Write the last good version number into the latest file.
In .swamp/data/_catalog.db, delete the rows for the moved versions. Set is_latest = 1 on the last good version. For each (workflow, job, step) group that lost its is_step_latest row, set is_step_latest = 1 on the highest remaining version of the group.

Suggested changes

Write raw and metadata.yaml to a temporary file, call fsync, then rename. Update latest and the catalog only after the rename.
Make swamp doctor datastores find versions with zero-byte or unparseable content, and catalog rows that have no readable content on disk.
Make --repair roll a record back to its last readable version and correct is_latest and is_step_latest in the catalog.

Environment

swamp 20261022.021804.0-sha.983e2929
Linux 6.18.33.2-microsoft-standard-WSL2 (WSL2), filesystem datastore
Writes came from swamp serve scheduled workflows

Extension dependencies: the shared import scan reads Babel syntax, not tokens (follow-up to #3969)

Rescore every extension's current version under rubric v5

Extension scoring queue: two replicas can claim the same pending event

Scheduled flaky-test detection is failing

Manual: the supported extension dependency categories, the canonical form and the lock rule, rewritten once against the final design

Registry audits v2 extension archives as written, and a range costs the loose point only without a valid lock

Extension consumers keep the publisher's bundle: pull, update and auto-resolve stop rebuilding (direction A for #3634)

Extension push: inputs that open up (workspace root lock, ancestor config refused by name, JSR dist-tags) and the skill docs rewritten once

Extension push ships the canonical form: every npm:/jsr: import pinned exact, and a v2 dependency-audit.json

stagecraft studio ticket columns: round-5 review lows from #3828 (source watch cost, relationText lookup, cross-instance claim)

A long-lived serve client for extensions, and a stable rate_limited error code

swamp serve on a v3 datastore never settles a workflow run left running by a stopped instance: boot reaper skips it as belonging to a remote instance

swamp serve on a v3 datastore: a run interrupted at a clean shutdown is never committed when the bucket is slow, and stays running for every other checkout

Support creating 1Password vaults for new clients

Filesystem datastore: an unclean shutdown leaves zero-byte data versions as the latest version, and `doctor datastores --repair` does not detect them

datastore_v3_replica_verify_test stall case flakes under the full integration run: restore.pid is read before the stub writes it

Generated state schemas require every child of nested objects, so normal resources warn on create/get

Add LinkedIn as a login provider

Showcase reward backfill can double-pay 5,000,000 points when replicas start together

Showcase editor: caption counter is not announced to screen readers

Profile showcase read: approved list sorts on review.reviewedAt in memory

Showcase countByStatus: $match live statuses before the $group

Showcase admin queues: index-serve the updatedAt sort on approved/rejected

Annotate and retain past workflow runs and data after the fact

Docs: --trusted-hosts accepts host:port for public addresses on non-default ports

A workflow step's method output is invisible to `data query` and `data list`, and the method summary's retrieval command returns an older run's data

Skill propulsion rule is silent when no transition is satisfied because stage work is outstanding

package.json mode: no loose-specifier warning for a range push ships as the exact installed version

extension fmt (and any command) creates or rewrites deno.lock when the extension lives in the repo's loaded extensions/ dir

serve shutdown does not wait for worker status writes, and its listener accepts workers after the gateway is disposed

Registry rescore: secret-safe-docs passes a secret README line on versions whose stored metadata predates the sensitive flag (follow-up to #3783)

stagecraft: rename work mode `method` to `behavior`, matching swamp's rename of models to behaviors

extension quality --json reports percentage but not the grade that extension info and the registry show

auth token create --json fields key and signinToken are undocumented; no token-mode service setup page

trigger.schedule is evaluated in the serve host's local time zone, with no zone field and no doc saying so

workflow validate/run say "Workflow not found" for a file that doctor workflows explains is misnamed

telemetry stats says "No telemetry data found" right after successful sends; say the spool is empty because it flushed

Docs: publish typed-secrets Phase 4.6 expression and delivery semantics

Extension registry page: a supported way to include media (images) in the README

CEL: data.latest(...).version is a double, but data.version() only accepts int, so "version - 1" fails at run time

workflow resume --run refuses workflows with the same step name in two jobs, though validate and run accept them

Update public OpenTelemetry manual for remote waterfalls and execution metrics (#3786)

Server-driven warning and info messages to CLI users, targeted by swamp version (exploration)

Serve: delegate namespace-scoped extension pulls and reload without granting administrator access

workflow cancel --server ignores --ca-cert and SWAMP_CA_CERT

Preserve active worker execution across serve restarts and upgrades; reconnect and reattach dispatches

v3: chunk durability barrier costs a full drive-cache flush per command on macOS; use an ordering barrier

vary/forEach object values with no key/name/id produce data names Windows refuses

issue-lifecycle start drops issue comments: context.comments is always empty

swamp core: a non-string sensitive value reads back as a JSON string

Relationships

#3955 Filesystem datastore: an unclean shutdown leaves zero-byte data versions as the latest version, and `doctor datastores --repair` does not detect them