Relationships
↔ sibling #3395#3492 Datastore rework Phase 4 G: S3 and GCS extensions implement storage/1 chunk storage for v3
Opened by stack72 · 10/15/2026· Shipped 10/15/2026
Background
swamp is reworking its datastore layer (tracking: swamp-club#2865). In the new v3 format, content is stored as immutable, content-addressed chunks. On S3 and GCS, swamp reaches them through the datastore extension's storage/1 contract. Phase 4E puts v3 on S3 and GCS and needs both first-party extensions to implement it.
This is Phase 4 G, the extensions' side. swamp-extensions has GitHub issues disabled, so it is tracked here.
The contract is DatastoreStorage in @swamp-club/swamp-testing (packages/testing/datastore_types.ts; the core side is swamp-club#3237). The provider's optional storage() returns it, scoped to the datastore root:
- chunks live at
<prefix>/_blobs/<namespace>/sha256/<first two hex>/<sha256>; namespaceis the repo's namespace slug, or_default.
| Method | Contract |
|---|---|
capabilities() |
maxChunkBytes, and touch support |
has(namespace, hashes, signal) |
the subset stored |
put(namespace, sha256, bytes, signal) |
replaces; sets lastModified to now; a reader never sees a partial chunk; concurrent puts of one hash leave one complete chunk |
get(namespace, sha256, signal) |
bytes, or null only when absent |
touch?(namespace, hashes, signal) |
refreshes lastModified; returns the ones that are missing |
list(namespace, signal) |
async iterable of {sha256, lastModified} |
delete(namespace, sha256, signal) |
removing a missing chunk succeeds |
Every method rejects when signal aborts. Swamp verifies every chunk it reads, so the storage need not hash.
What is needed (S3 and GCS)
- Implement
storage()on both providers, per the contract. - Batch for real:
hason many hashes should not be one HEAD per hash where a list or batch is cheaper; choose per backend and say why;touchrefresheslastModifiedwithout re-uploading where the backend allows it. S3 copy-in-place with metadata replacement; GCS rewrite or metadata patch. Measure what each costs.- If a backend can't touch cheaply, set
capabilities().touchto false, and swamp puts the chunk again instead.
- Honour aborts during uploads, multipart included: an aborted put leaves no partial object and no orphaned multipart upload.
maxChunkBytes: swamp cuts chunks ofmin(16 MiB, maxChunkBytes). Choose a value that avoids a multipart upload per chunk where possible, and say why.lastModifiedcomes from the server's clock, never the client's.
Tests
assertStorageConformancefrom@swamp-club/swamp-testingagainst ministack (S3) and fake-gcs-server (GCS). Run it without the suite'ssetLastModifiedhelper where the emulator can't set times, and say which cases that skips.- An abort mid-upload leaves nothing behind.
- Batching: count the requests for
hasandtouchon 1,000 hashes.
Acceptance
storage()on both providers passes the conformance suite.- The release notes name the swamp version that first calls
storage(): Phase 4E, not yet released.
Blocked by: nothing. Phase 4E (swamp-club#2865's plan) waits for it, as it does for swamp-club#3395.
Shipped
Click a lifecycle step above to view its details.
system commented 10/15/2026, 12:26:45 AM
Classified automatically when this issue was filed.
- Source: Extensions
If you feel this classification is incorrect, add a ripple to tell us so.
Sign in to post a ripple.