UNDERSTANDING GOOGLE CLOUD STORAGE AUTOMATION IN SWAMP

Google Cloud Storage plays two separate roles in Swamp. In the first, Cloud Storage is something Swamp manages. Buckets, objects, access controls and notifications are resources, and behaviors from the @swamp/gcp/storage extension create, read, change and delete them. In the second, Cloud Storage is where Swamp keeps its own records. The @swamp/gcs-datastore extension moves a repository's data into a bucket. The two roles can share a project and even the same tooling, but they answer different questions. One is what your infrastructure looks like. The other is where the history of your automation is stored. Both fit the primitives described in How Swamp Works.

Resources as behavior types

The storage extension follows the shape of the Cloud Storage JSON API. Each API resource gets its own behavior type: buckets, objects, folders, managed folders, bucket and object access control entries, default object access controls, HMAC keys, Pub/Sub notifications, Anywhere Cache and Rapid Cache instances, and the project's Cloud Storage service account. A behavior made from one of these types holds the configuration of one resource as its global arguments. Its methods are the operations the API allows on that resource. Most types share create, get, update, delete, sync and list. Others add operations specific to that resource, such as compose and rewrite on objects or pause and resume on Anywhere Cache.

The types are not perfectly uniform, and the differences come from the API. Folders exist only in buckets with hierarchical namespace enabled. The service account type has only get and sync, because the Cloud Storage service agent is created by Google, not by you. Several types require a name global argument as an instance name, because they follow the factory pattern, where one definition stands for many resources. Behaviors, Types, and Methods covers this relationship in general.

Each method writes the resource's state as Swamp data. After a list on objects or a sync on a bucket, the repository holds a record of what the API returned at that moment. Workflow steps can query that record without calling Google Cloud again. The Data Layer explains how that data is versioned and kept.

Choosing how access is controlled

Cloud Storage has two ways to control who can read and write. The first is uniform bucket-level access, where IAM policies on the bucket decide everything. The second is the older system of access control lists on the bucket and on each object. The extension supports both. IAM is handled by methods on the buckets type. ACLs are handled by the bucketaccesscontrols, objectaccesscontrols and defaultobjectaccesscontrols types.

For new buckets, uniform access is the better default. One policy describes the whole bucket, so you can read the policy and know who has access, and with uniform access turned on, ACLs are not used at all. The bootstrap provisioner turns it on for this reason. The ACL types are useful when you work with existing buckets that still depend on per-object grants, or when objects in one bucket need different audiences and splitting the bucket is not an option.

Within IAM, there are two styles of change. set_iam_policy replaces the entire policy, which suits a definition that is meant to be the single source of truth for the bucket. add_iam_binding and remove_iam_binding change one binding. They read the current policy, modify it, and write it back with the etag they read, so an edit made by someone else between the read and the write causes a conflict and is not silently lost. Teams that share IAM ownership with other tools or other people usually prefer the binding methods. Teams that want Swamp to own the whole policy may prefer replacement.

Some bucket settings are fixed when the bucket is created, location being the main one. Changing location later is a relocate operation, not an update. For that reason, location and similar settings belong in the behavior definition from the start.

Credentials and secrets

By default, the storage behaviors use the credentials already in the environment: Application Default Credentials, or the GOOGLE_APPLICATION_CREDENTIALS_JSON and GCP_ACCESS_TOKEN variables. This works well on a developer's machine or on a runner with an attached service account. Where credentials must come from somewhere else, the accessToken and credentialsJson global arguments accept a vault.get() expression, and the value is then redacted from output like any other vault secret. The Vaults reference describes the expression, and Auto-Refresh Cloud Credentials covers keeping short-lived tokens current.

Two arguments deal with billing, not identity. quotaProject sets the project that quota and billing are charged to. userProject names the project billed for requests to Requester Pays buckets. apiEndpoint points a behavior at an emulator, which lets a definition be tested without touching a real project.

Composing storage tasks in workflows

A single storage change rarely stands alone. A new bucket usually needs IAM bindings, perhaps a notification to a Pub/Sub topic, and perhaps a lifecycle rule. Running these as ordered workflow steps means each step starts only after the one before it has succeeded. If a later step fails, the record shows exactly how far the work got. The Workflow Execution Model describes how steps are ordered and how failures stop a run.

The bootstrap workflow in @swamp/gcs-datastore-bootstrap is a small worked example. Its first step runs the provisioner, which creates or verifies a bucket and a custom role. Its second step is a command/shell behavior that switches the repository's datastore. Because the provisioner verifies what already exists before creating anything, the workflow can be run again safely after a partial failure. This property is worth designing for in your own storage workflows, as Make Workflows Idempotent discusses.

Keeping repository data in Cloud Storage

A local datastore is enough when one person works in one checkout. Once several machines, CI runners or swamp serve instances need the same history, the data has to live somewhere they can all reach. The GCS datastore keeps a local cache directory and syncs it with the bucket in both directions, so commands read local files and only exchange changes with the bucket.

Concurrency is the hard part, and the design is built around Cloud Storage's object generations. Locks are taken with generation-based preconditions and retried with exponential backoff when another writer holds them. The index is split into partition shards per behavior under _index/. Every shard write and every write to _meta.json is a compare-and-swap on the generation, so two writers cannot drop each other's index entries. A sequence counter lets a sync with nothing to do finish without transferring files. Sync runs in two phases, which keeps the global lock held for as short a time as possible. Namespace-scoped sync lets several repositories share one bucket, which is the basis of a Giga-Swamp. Datastore Architecture places this design alongside the other datastore providers.

The datastore needs five permissions: storage.buckets.get and storage.objects.get, create, delete and list. The predefined roles/storage.objectAdmin role includes all of them along with others. The bootstrap's custom role grants exactly those five and is scoped to the project. That is narrower, at the cost of one more role to manage. Teams that already hand out objectAdmin freely may not need the custom role. Teams that audit permissions closely usually will.

The tuning settings matter mostly for large repositories or slow networks. pullConcurrency defaults to 50 and pushConcurrency to 25, and requestTimeoutMs defaults to 30000. Raising the concurrency limits shortens syncs of many small files. Raising the timeout helps over slow links. The datastore also emits OpenTelemetry spans for its operations, locks and syncs, which makes contention visible in traces. See Set Up OpenTelemetry to enable tracing and Datastore Configuration for the full set of options.