Rclone Archive

Inject --s3-no-check-bucket, or every packed upload dies 403 on CreateBucket.

The pack strategy became reachable in 2026.08.19.3. The first time it ever ran against S3 it failed on every object:

ERROR : _root.tar: Post request rcat error: failed to prepare upload: operation error S3: CreateBucket, StatusCode: 403 ... is not authorized to perform: s3:CreateBucket

rclone checks the destination bucket exists and tries to CREATE it when that check is inconclusive, and rcat -- how every packed upload is written -- takes that path per object. An archive user that deliberately cannot create buckets therefore cannot write a single pack. Adding the permission is the wrong fix: the bucket is provisioned out of band and nothing in the archive path should be able to make one, so the denial is the design working.

Verified against a live bucket before the fix was written: the identical rcat without the flag returns 403 and writes nothing; with the flag the object lands in DEEP_ARCHIVE.

The flag is injected on the same terms as --s3-storage-class, and the shell-entrypoint guard now requires BOTH by name -- appending a flag to sh -c '<script>' makes it a positional parameter of the shell that rclone never sees. The extract script carries it too; it is a no-op for a read, but uniform means the next path that writes cannot be added without it.

Why this surfaced so late: copy never triggers the bucket check, so the direct strategy ran green against real hardware for days, and the packed path was unreachable until the previous release.

131 tests. One existing test changed: the storage-class guard test matched the error's prose rather than the flag name, so it broke when the message changed to cover both flags. Adversarial review 14 pass / 1 na against content hash 7fa53f6f.

push resolves the documented auto strategy instead of falling through to direct.

The strategy global argument has always described itself as "auto (default) picks from mean file size at scan time". push never implemented it: chooseStrategy was not called there, so with strategy unset the answer was always direct. Present since the first release.

That is the worst shape a cost bug takes. scan computes the same decision correctly and reports strategy=pack, so the operator is told the share will be packed and push then silently does the opposite, with both rungs reporting success. Packing exists only to amortise the 40 KB per-object billing minimum, so a small-file share paid that minimum per file. It also meant the pack path was unreachable in production -- every pack fix shipped so far was found by measurement and tests, never by a live run.

push now measures with rclone size on the SOURCE only (no credentials, no S3 request, nothing billed) and calls chooseStrategy, logging the file count, the mean and the threshold it compared against. If the measurement fails it REFUSES, recording strategy-undetermined and inconclusive: defaulting to direct would be the original bug wearing a comment, and defaulting to pack would tar a share whose size we just failed to establish.

Pinning strategy explicitly skips the measurement, which is a full source walk. That is the right choice for a scheduled incremental run where the answer is known and stable, and the argument description now says so.

Note for existing archives: objects written before this version went up object-per-file whatever scan reported. They are not wrong, only more expensive, and this suite cannot remove them -- --immutable, no sync, and the IAM user has no s3:DeleteObject -- so re-pushing a share as a pack stores it twice for the 180-day minimum.

128 tests. Six pre-existing push tests had to be pinned to strategy=direct: they assert copy-path flags while feeding a fixture whose mean selects pack, so they were passing because of the bug. Adversarial review 14 pass / 1 na against content hash dfc378cf.

dryRun now prints the exact command the seed is pasted from, and the env file that command reads from stdin.

SETUP §7.3 tells the operator to generate the seed command rather than write one, because a hand-written invocation that drops --s3-storage-class archives at S3 Standard -- roughly 23x the Deep Archive price -- and rclone reports success either way. That instruction was unbacked: dryRun logged what it WOULD pack, and the assembled docker command was never logged at any level.

Command assembly now lives in an exported buildRcloneInvocation that both the real run and the emission call, so the printed string cannot drift from the executed one. On the direct path the emitter filters the real argv for --dry-run rather than rebuilding it, so the two differ in that one flag and no other. The pack path emits one command per pack, matching how it actually runs.

The env file is printed with the command. Without it the paste does not fail, it BLOCKS in cat reading the terminal -- the command reads its credentials from its own stdin, because docker run -e leaks into ps and DSM cannot open /dev/stdin. The two secret values print as placeholders, produced by passing placeholder credentials to the real buildEnvFile, so the true values are never in the string.

Safe to print by construction: the credential-in-argv check refuses rather than redacts. Note that the command does carry the bucket, prefix, ssh host and mount path, and dryRun logs persist.

Also corrects every documented swamp CLI invocation in SETUP.md and the README. The form is swamp model method run <model> <method>; model and method were reversed throughout, so the documented workflow -- including the seed command -- failed with "Model not found".

124 tests. Adversarial review 14 pass / 1 na against content hash c4712ca5.

First release.

Archive a Synology share to S3 Glacier Deep Archive over SSH into a container on the NAS, so no binary is installed on DSM. A cost-ordered ladder: scan, push, verify, restoreRequest, restoreDrill. Every rung writes its own resource, and a rung that has never run is itself a finding.

All five rungs are proven against real hardware, including a completed Deep Archive retrieval whose SHA-256 matched its source.

It never deletes. sync, move, purge and delete are refused at the runner: a source that fails to mount presents to rclone as empty, and sync would then empty the destination unrecoverably while still billing the 180-day minimum.

It reports what recovery costs, not just what storage costs, because an archive whose recovery price is first discovered during a recovery is one nobody can afford to use.

Some things this encodes that the documentation will not tell you:

• DEEP_ARCHIVE is the storage class. GLACIER_DEEP_ARCHIVE reads like the obvious name and is not one; S3 rejects every object carrying it.
• --no-traverse without a --max-age window is a cost trap. It skips listing the destination and HEADs every source file considered instead, roughly 80x the cost of one listing. The two are coupled in code so they cannot be separated.
• A tar streamed into rcat has unknown size, and rclone only auto-scales --s3-chunk-size for files whose size it knows. At the 5 MiB default that caps a streamed object at 48 GiB, and it fails 48 GiB into the upload rather than at the start. Pack sizing derives the chunk size from the planned size and refuses, before any bytes move, what it cannot stream or buffer.
• Size a transfer by apparent bytes, never by df or du defaults. Sparse and reflink-shared files read far larger than they store, and the gap can be hundreds of times.
• rclone has md5sum and sha1sum and no sha256sum. Hashing is hashsum sha256 --download, and --download is what makes a restore drill mean anything: without it the hash is requested from the remote, S3 does not serve SHA-256, and the drill reports success over bytes that never moved.
• backend restore does not signal per-object outcomes through its exit code. A per-object error exits 0 with the failure only in the JSON, and a filter matching nothing also exits 0.

Credentials never appear in ps on either host. They travel as a docker --env-file streamed down the SSH stdin pipe into a FIFO, because docker run -e leaks into the process table and DSM refuses to open /dev/stdin.