Skip to main content
← Back to list
01Issue
BugOpenSwamp CLIPublic
AssigneesNone

Relationships

#3955 Filesystem datastore: an unclean shutdown leaves zero-byte data versions as the latest version, and `doctor datastores --repair` does not detect them

Opened by anthony · 10/26/2026

Description

The filesystem datastore does not write data versions atomically. An unclean shutdown during a write leaves the version directory with an empty raw file and an empty metadata.yaml file. The latest pointer file and the catalog row (is_latest = 1) already point to the empty version.

After this:

  • swamp data query returns the record with content: {}. CEL that reads a field fails, for example No such key: stageId. Every workflow that queries the spec fails.
  • swamp data get <model> <name> returns not_found.
  • swamp data versions does not list the empty versions, but the catalog keeps them.
  • swamp doctor datastores reports pass. --repair reports "Catalog index matches data on disk". The check only compares disk to catalog. It does not check for empty content, and it does not check for catalog rows whose files have no content.

No CLI command can recover the record. swamp data delete removes all versions of a record, not one version. No command rebuilds the catalog from disk.

Observed impact

One unclean shutdown made 168 files zero bytes. All files had the same mtime. 26 records had an empty latest version. These records included the state, journal, status and evidence records of three active @swamp/software-factory work items. The work items stopped for two days with no error until a workflow evaluated a field of the empty record.

The @swamp/workflow-summary report for the run that wrote the empty version shows succeeded.

Steps to reproduce

  1. Run a method that writes a new version of a resource. Stop the VM or host before the OS flushes the file data to disk, for example wsl --shutdown on WSL2. (This step is not verified. The observed failure came from an unplanned stop at an unknown point in the write.)
  2. Run swamp data query '<predicate that matches the record>' --json. Result: content: {}.
  3. Run swamp data get <model> <name>. Result: not_found.
  4. Run swamp doctor datastores --repair. Result: no findings.

Manual recovery that worked

  1. Move the empty version directories out of the record directory.
  2. Write the last good version number into the latest file.
  3. In .swamp/data/_catalog.db, delete the rows for the moved versions. Set is_latest = 1 on the last good version. For each (workflow, job, step) group that lost its is_step_latest row, set is_step_latest = 1 on the highest remaining version of the group.

Suggested changes

  1. Write raw and metadata.yaml to a temporary file, call fsync, then rename. Update latest and the catalog only after the rename.
  2. Make swamp doctor datastores find versions with zero-byte or unparseable content, and catalog rows that have no readable content on disk.
  3. Make --repair roll a record back to its last readable version and correct is_latest and is_step_latest in the catalog.

Environment

  • swamp 20261022.021804.0-sha.983e2929
  • Linux 6.18.33.2-microsoft-standard-WSL2 (WSL2), filesystem datastore
  • Writes came from swamp serve scheduled workflows
02Bog Flow
◉OPEN○TRIAGED○IN PROGRESS○SHIPPED

Open

10/26/2026, 7:31:03 AM

No activity in this phase yet.

03Sludge Pulse

Sign in to post a ripple.