Skip to main content
← Back to list
01Issue
FeatureShippedExtensionsPublic
AssigneesNone

Relationships

↔ sibling #3395

#3492 Datastore rework Phase 4 G: S3 and GCS extensions implement storage/1 chunk storage for v3

Opened by stack72 · 10/15/2026· Shipped 10/15/2026

Background

swamp is reworking its datastore layer (tracking: swamp-club#2865). In the new v3 format, content is stored as immutable, content-addressed chunks. On S3 and GCS, swamp reaches them through the datastore extension's storage/1 contract. Phase 4E puts v3 on S3 and GCS and needs both first-party extensions to implement it.

This is Phase 4 G, the extensions' side. swamp-extensions has GitHub issues disabled, so it is tracked here.

The contract is DatastoreStorage in @swamp-club/swamp-testing (packages/testing/datastore_types.ts; the core side is swamp-club#3237). The provider's optional storage() returns it, scoped to the datastore root:

  • chunks live at <prefix>/_blobs/<namespace>/sha256/<first two hex>/<sha256>;
  • namespace is the repo's namespace slug, or _default.
Method Contract
capabilities() maxChunkBytes, and touch support
has(namespace, hashes, signal) the subset stored
put(namespace, sha256, bytes, signal) replaces; sets lastModified to now; a reader never sees a partial chunk; concurrent puts of one hash leave one complete chunk
get(namespace, sha256, signal) bytes, or null only when absent
touch?(namespace, hashes, signal) refreshes lastModified; returns the ones that are missing
list(namespace, signal) async iterable of {sha256, lastModified}
delete(namespace, sha256, signal) removing a missing chunk succeeds

Every method rejects when signal aborts. Swamp verifies every chunk it reads, so the storage need not hash.

What is needed (S3 and GCS)

  1. Implement storage() on both providers, per the contract.
  2. Batch for real:
    • has on many hashes should not be one HEAD per hash where a list or batch is cheaper; choose per backend and say why;
    • touch refreshes lastModified without re-uploading where the backend allows it. S3 copy-in-place with metadata replacement; GCS rewrite or metadata patch. Measure what each costs.
    • If a backend can't touch cheaply, set capabilities().touch to false, and swamp puts the chunk again instead.
  3. Honour aborts during uploads, multipart included: an aborted put leaves no partial object and no orphaned multipart upload.
  4. maxChunkBytes: swamp cuts chunks of min(16 MiB, maxChunkBytes). Choose a value that avoids a multipart upload per chunk where possible, and say why.
  5. lastModified comes from the server's clock, never the client's.

Tests

  • assertStorageConformance from @swamp-club/swamp-testing against ministack (S3) and fake-gcs-server (GCS). Run it without the suite's setLastModified helper where the emulator can't set times, and say which cases that skips.
  • An abort mid-upload leaves nothing behind.
  • Batching: count the requests for has and touch on 1,000 hashes.

Acceptance

  • storage() on both providers passes the conformance suite.
  • The release notes name the swamp version that first calls storage(): Phase 4E, not yet released.

Blocked by: nothing. Phase 4E (swamp-club#2865's plan) waits for it, as it does for swamp-club#3395.

02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPEDLINKED+ 5 MOREPR_MERGED+ 1 MORESESSION_SUMMARIZED

Shipped

10/15/2026, 6:52:48 PM

Click a lifecycle step above to view its details.

03Sludge Pulse
stack72 linked sibling of #339510/15/2026, 4:53:20 PM
Editable. Press Enter to edit.

system commented 10/15/2026, 12:26:45 AM

Classified automatically when this issue was filed.

  • Source: Extensions

If you feel this classification is incorrect, add a ripple to tell us so.

Sign in to post a ripple.