Relationships
#4233 Supported binary file ingestion for remote members of swamp serve, without the bytes landing in run history
Opened by vcjdeboer · 11/1/2026
Use case. Authenticated members of a swamp serve deployment submit a binary file (in our case a 0.6–32 MB instrument file) for a remote worker to analyse. The result must bind back to the exact bytes submitted (version + sha256), and only members allowed to read the file may use it.
What we do today, and why it hurts. At a2f815f3 the only route is a base64 string input, which a workflow step then writes as a file record. The file record itself is fine. But the same bytes are also persisted, with no opt-out, in:
- the run record (
execution_service.ts:4093-4094); - every entry of
.runs-index.json, which is rewritten in full on every run save; - the per-run evaluated workflow (
execution_service.ts:4144, inputs spliced into step arguments,persisted_workflow.ts:54-78).
Workflow inputs have no sensitive flag or size bound (.claude/skills/swamp/references/workflow/reference.md:560). At our scale this is what drives #4224: 709 runs gave a 433 MB index and serve heap exhaustion. At our declared 32 MiB maximum (≈44.7 MB of base64), about ten submissions would give the same index size.
What we found when looking for an alternative (a2f815f3):
- No upload command. There is no
data put/import/uploadcommand (src/cli/commands/data.ts:36-44).--input-fileandkey=@pathread the file into an argument string (src/cli/input_parser.ts:97-117). - Webhooks don't help. Webhook bodies (10 MB cap,
src/serve/webhook.ts:618) become workflow inputs, so they are persisted the same way. - The upload endpoint needs v3. Serve's only byte-upload endpoint is the v3 datastore API (
src/serve/datastore_api.ts). It needs a v3 datastore with serve holding the writer lease, which an@swamp/s3-datastoredeployment doesn't have. The other serve data verbs only read, delete or gc. - The closest workaround is fragile. That is a direct
model method run --server … --stdinwhose base64 is a plain method argument.--inputoverrides are applied after the evaluated definition is snapshotted (src/libswamp/models/run.ts:1106-1121), so the argument isn't persisted there. But:- it breaks as soon as the value goes through a definition
inputskey or a workflow; - marking it
sensitivewould push the payload into the vault; - we haven't verified whether serve's audit detail or the WebSocket transport handle a 45 MB request;
- a worker can read file bytes only for its own dispatch's model type (
src/serve/capability_service.ts:182-201), which ties the upload method and the analysis to one type.
- it breaks as soon as the value goes through a definition
What we'd like.
- A member-authorized upload through serve that stores a binary file record under a model (streamed or chunked, not one giant argument). It returns
{model, dataName, version, sha256}and persists nothing in run, audit or evaluated-definition records beyond that reference. - A documented way for a workflow input to be such a reference. Serve should check, at submission time, that the submitting member may read that version. It should pin the exact version for replay (never "latest"). And a worker step should be able to read those bytes with their sha256.
- Docs steering applications away from embedding files in workflow inputs, plus an example of the pattern above. Until then, a warning when one input exceeds, say, 1 MB would make the cost visible.
Item 1 alone would fix new submissions for us. Existing history still needs the index change asked for in #4224.
Related: #4224 (heap exhaustion from large inputs in .runs-index.json), #3407 (history size), #2435 (serve boot hydration).
Open
No activity in this phase yet.
Sign in to post a ripple.