UNDERSTANDING AMAZON S3 AUTOMATION IN SWAMP

S3 has two roles in Swamp. In the first, an S3 bucket, policy or access point is a resource that a behavior creates, reads, changes and deletes, in the same way Swamp manages any other cloud resource. In the second, a bucket holds the repository's own data, with every behavior's definitions, run results and resource state stored under a prefix in that bucket. The two roles come from different extensions and answer different questions. @swamp/aws/s3 is about what exists in your AWS account. @swamp/s3-datastore is about where Swamp keeps its records. A team can use one without the other, and the decisions behind each are mostly separate. Datastore Architecture covers the second role in general terms.

Resources as behavior types

The @swamp/aws/s3 extension maps each kind of S3 resource to one behavior type. The types are bucket, bucket-policy, access-point, multi-region-access-point and multi-region-access-point-policy. The S3 Access Grants trio is access-grants-instance, access-grants-location and access-grant, and the Storage Lens pair is storage-lens and storage-lens-group. The split follows the way AWS divides its resources rather than a list of tasks. A bucket and its policy are separate resources in AWS, so they are separate behaviors in Swamp. Each can be created, changed and removed without touching the other.

Each type has the same set of methods: create, get, update, delete, sync and list. The exception is multi-region-access-point, which has no update because AWS does not allow a Multi-Region Access Point to be changed once it exists. Because the methods are the same across types, a workflow that manages a bucket and one that manages an access point look alike. Behaviors, Types, and Methods covers the general design.

Configuration as global arguments

A resource's settings are the behavior's global arguments. Top-level properties such as VersioningConfiguration, PublicAccessBlockConfiguration, BucketEncryption and LifecycleConfiguration are each one global argument whose value is an object. On the command line they are passed as JSON, and in a definition file they are written as nested YAML. The bucket type has no required global arguments. A definition can be as short as a name, or it can hold the whole configuration of a production bucket.

Most other types need one or two required arguments. A bucket policy needs Bucket and PolicyDocument, and an access point needs Bucket. The access grants and Storage Lens types also need a name argument, which is the instance name in the factory pattern. With that pattern, one definition can stand for many similar resources, each with its own name. Reuse a Behavior with the Factory Pattern describes how to use it.

Resource state as data

Every method writes the resource state it reads or changes into the data layer. After a create, the repository holds a record of what AWS returned for the new resource. sync matters because people also change buckets in the console, with the AWS CLI or with other tools. Running sync brings those changes into Swamp's records, so the stored state matches the account rather than the last thing Swamp did.

Once the state is stored as data, other parts of Swamp can read it without calling AWS. The @jentz/aws-s3-bucket-audit report reads bucket and bucket-policy state produced earlier in a workflow. It checks that state for versioning, encryption, blocked public access, enforced object ownership, TLS-only policies, lifecycle expiry of old versions and access logging. If a step's data is missing, the report records a skip and does not fail. A missing finding therefore usually means the workflow has no step that writes that state, such as a sync of the bucket policy.

Credentials

The S3 behavior types read credentials from the default AWS credential chain: environment variables, a profile in ~/.aws, or a role attached to the instance or task. They also take accessKeyId, secretAccessKey and sessionToken as global arguments, and these override the environment. Those arguments are meant to be wired to a vault with an expression such as ${{ vault.get('aws-creds', 'AWS_ACCESS_KEY_ID') }}, never written inline.

Each approach has its trade-offs. The default chain keeps credentials out of definitions completely and works with SSO sessions and instance roles. It does mean the definition does not say which identity it runs as, so the outcome depends on where Swamp is running. Vault-wired arguments make the identity explicit and let different behaviors in one repository use different accounts. They also tie the definition to a vault and to long-lived keys unless the vault refreshes them. For people working at a desk, SSO is often the cleaner choice (see Authenticate to AWS with SSO). Vault-wired credentials suit unattended runs that need a specific identity.

Task-focused extensions

Next to the resource types are extensions built around a single job. @mgreten/aws/s3-site-deploy does not manage a bucket as a resource at all. It treats the bucket as a deployment target, compares MD5 hashes to decide what to upload, and deletes objects that are no longer in the site. @mgreten/s3-object-read is about least privilege. It verifies read access, writes a listing of object metadata within a prefix and a size limit, or downloads one named object. It never pulls a whole bucket. @mgreten/s3-backup-retention separates deciding from deleting. It first creates a retention plan that deletes nothing, and only applies a plan that has been confirmed exactly.

These extensions show a different way to automate S3. The resource types describe what should exist. The task extensions describe an operation and record its outcome. Many teams use both: buckets and policies as behaviors, plus a deploy or retention behavior that acts on those buckets.

Decisions for a team

Not every bucket needs to be a behavior. Buckets that other tools create, such as CloudFormation stacks or log delivery, are often better left to those tools. If Swamp should know about them, read them with get or list. Managing the same bucket from two places leads to each tool overwriting the other's changes.

Keeping Swamp's own data in S3 is a separate choice. A local datastore is enough for one person on one machine. Once several people, CI runners or a swamp serve deployment need the same history, a shared datastore is needed. @swamp/s3-datastore keeps a local cache and syncs it with the bucket in both directions. It uses S3 conditional writes (If-None-Match and If-Match) as a distributed lock and for compare-and-swap merges of its index, so concurrent writers do not drop each other's entries. The same mechanism explains why S3-compatible endpoints are checked during setup. An endpoint that ignores conditional headers would make the locking unsafe, so setup fails on such an endpoint instead of reporting it healthy.

A single S3 datastore can also be shared by several repositories, each in its own namespace, so their data stays separate but can be queried together. Giga-Swamp and Namespaces covers when that makes sense and when separate datastores are the better choice.