Skip to main content
← Back to list
01Issue
FeatureOpenSwamp CLIPublic
AssigneesNone

Relationships

#4233 Supported binary file ingestion for remote members of swamp serve, without the bytes landing in run history

Opened by vcjdeboer · 11/1/2026

Use case. Authenticated members of a swamp serve deployment submit a binary file (in our case a 0.6–32 MB instrument file) for a remote worker to analyse. The result must bind back to the exact bytes submitted (version + sha256), and only members allowed to read the file may use it.

What we do today, and why it hurts. At a2f815f3 the only route is a base64 string input, which a workflow step then writes as a file record. The file record itself is fine. But the same bytes are also persisted, with no opt-out, in:

  • the run record (execution_service.ts:4093-4094);
  • every entry of .runs-index.json, which is rewritten in full on every run save;
  • the per-run evaluated workflow (execution_service.ts:4144, inputs spliced into step arguments, persisted_workflow.ts:54-78).

Workflow inputs have no sensitive flag or size bound (.claude/skills/swamp/references/workflow/reference.md:560). At our scale this is what drives #4224: 709 runs gave a 433 MB index and serve heap exhaustion. At our declared 32 MiB maximum (≈44.7 MB of base64), about ten submissions would give the same index size.

What we found when looking for an alternative (a2f815f3):

  • No upload command. There is no data put/import/upload command (src/cli/commands/data.ts:36-44). --input-file and key=@path read the file into an argument string (src/cli/input_parser.ts:97-117).
  • Webhooks don't help. Webhook bodies (10 MB cap, src/serve/webhook.ts:618) become workflow inputs, so they are persisted the same way.
  • The upload endpoint needs v3. Serve's only byte-upload endpoint is the v3 datastore API (src/serve/datastore_api.ts). It needs a v3 datastore with serve holding the writer lease, which an @swamp/s3-datastore deployment doesn't have. The other serve data verbs only read, delete or gc.
  • The closest workaround is fragile. That is a direct model method run --server … --stdin whose base64 is a plain method argument. --input overrides are applied after the evaluated definition is snapshotted (src/libswamp/models/run.ts:1106-1121), so the argument isn't persisted there. But:
    • it breaks as soon as the value goes through a definition inputs key or a workflow;
    • marking it sensitive would push the payload into the vault;
    • we haven't verified whether serve's audit detail or the WebSocket transport handle a 45 MB request;
    • a worker can read file bytes only for its own dispatch's model type (src/serve/capability_service.ts:182-201), which ties the upload method and the analysis to one type.

What we'd like.

  1. A member-authorized upload through serve that stores a binary file record under a model (streamed or chunked, not one giant argument). It returns {model, dataName, version, sha256} and persists nothing in run, audit or evaluated-definition records beyond that reference.
  2. A documented way for a workflow input to be such a reference. Serve should check, at submission time, that the submitting member may read that version. It should pin the exact version for replay (never "latest"). And a worker step should be able to read those bytes with their sha256.
  3. Docs steering applications away from embedding files in workflow inputs, plus an example of the pattern above. Until then, a warning when one input exceeds, say, 1 MB would make the cost visible.

Item 1 alone would fix new submissions for us. Existing history still needs the index change asked for in #4224.

Related: #4224 (heap exhaustion from large inputs in .runs-index.json), #3407 (history size), #2435 (serve boot hydration).

02Bog Flow
◉OPEN○TRIAGED○IN PROGRESS○SHIPPED

Open

11/1/2026, 9:07:39 AM

No activity in this phase yet.

03Sludge Pulse

Sign in to post a ripple.