Disclosure: this post was written by the Appaloft team about our own implementation and the operational gates we still keep open. It contains no production keys, account identifiers, secret names, or backup references.
This week we added versioned encryption envelopes and an explicit rotation workflow for Appaloft control-plane secrets. Secrets attached to Environments, Resources, Deployment snapshots, and dependency runtimes are encrypted at rest. Plaintext exists only at command ingress and during transient runtime materialization.
The encryption code was the short part. The design work sat around it: deciding how old ciphertext remains readable, how CI obtains the keyring, how a reviewed migration survives database drift, and how recovery works after a transaction has already committed.
We ended up treating custody, database migration, and recovery proof as three consecutive boundaries. The retained key stays available until all three have evidence.
One keyring serves writes and compatibility reads
Appaloft stores control-plane secrets in versioned, authenticated AES-256-GCM envelopes. Each envelope carries a key id. Runtime configuration supplies one active key and any retained decrypt-only keys:
{
"activeKeyId": "2026-q3",
"keys": {
"2026-q3": "<active 32-byte base64 key>",
"2026-q2": "<retained 32-byte base64 key>"
}
}New writes use the active key. A retained key can only read envelopes that have not been rewrapped yet. That lets an operator deploy both keys, prove that existing ciphertext still opens, and then move the write target.
A configuration overwrite would be dangerous here. If the new configuration contains only the new key, every envelope written with the old key immediately loses its read path. Appaloft therefore preflights all secret envelopes before planning, retrying, redeploying, rolling back, or materializing a Docker, Compose, SSH, or Swarm runtime. A missing key, wrong key, or damaged envelope blocks the mutation. An adapter cannot turn the failure into an empty string, omit the variable, and continue to success.
The provider-neutral contract is documented in public ADR-089 and the Appaloft key rotation workflow. The implementation and its tests are collected in appaloft/appaloft#687.
Custody lives outside the ciphertext failure domain
Storing ciphertext and its only decryption key behind the same database access boundary gives one credential the whole system. It also couples database recovery to keyring recovery. We wanted a separate access and recovery boundary, so Appaloft Cloud keeps the complete production keyring document in AWS Secrets Manager and loads it from GitHub Actions during deployment.
CI has no long-lived AWS access key. The workflow exchanges a GitHub OIDC token for short-lived AWS credentials. The IAM trust is scoped to the protected production environment, and the role can read only the target secret. GitHub’s AWS OIDC guide likewise recommends evaluating the sub claim in the trust policy so only the intended workflows can assume the role.
The custody document contains both the active key id and the complete active/retained key map. AWS Secrets Manager creates versions of a secret value and marks the default retrieval target with AWSCURRENT; it also tracks previous and pending stages. Those provider versions help with custody-document cutover. Appaloft’s retained keys keep database envelopes readable. The two mechanisms are adjacent but solve different compatibility problems. AWS describes the staging behavior in its secret version documentation.
Terraform creates the custody shell
Terraform creates the secret metadata, GitHub OIDC provider, IAM role, and narrow read policy. It does not create the secret version containing key material, and the stack accepts no key value as an input.
That distinction matters because Terraform’s sensitive flag primarily controls display. It does not automatically keep a value out of state. HashiCorp’s sensitive data documentation warns that secrets placed in configuration can reach plan and state files; the state still needs remote storage, encryption at rest, access controls, and auditing.
An operator generates the keyring on a controlled terminal, writes it to a new permission-restricted file, and passes that file to the secret-version API. After version metadata has been read back, the temporary file is removed. Output is limited to schema version, key ids, key count, and a provider version reference. Key material, ciphertext, and secret lengths stay out of logs.
Infrastructure as code still owns the durable resources and trust relationships. The most sensitive value never becomes part of its state.
Rotation begins with a read-only plan
Appaloft exposes separate plan and apply operations:
appaloft db secret-rotation plan
appaloft db secret-rotation apply \
--plan-digest sha256:<reviewed-digest> \
--backup-reference <external-backup-reference>The plan performs no writes. It reports record counts, variable-key counts, envelope states, safe key ids, readiness, and a deterministic digest. It returns neither plaintext nor ciphertext.
After reviewing the plan and obtaining a database backup reference, the operator passes the digest to apply. Apply computes the plan again. If a secret changed in the meantime, the digest is stale and the operation returns to planning. Migrating legacy plaintext also requires a separate explicit authorization.
Before opening a transaction, the adapter preflights every selected row. Only when every envelope is readable, the target key exists, and the backup reference and digest are valid does one transaction rewrap the full selection. Any row failure rolls back the migration. The transaction gives us database atomicity; the external backup reference covers failures discovered after commit.
Legacy adoption needs an ordered rollout
The first move from legacy plaintext to a keyring-aware runtime has an extra deployment-order problem.
An old runtime may continue writing plaintext while the new runtime rejects implicit deployment of legacy rows. Migrating the database first leaves the old version unable to understand the new state if the rollout fails. Deploying the new runtime without freezing secret-dependent writes lets the migration plan drift continuously.
Our sequence prepares the complete keyring and database backup first, then freezes writes that create or read control-plane secrets. We deploy the keyring-aware runtime, run the read-only plan, review the legacy and unreadable counts, and only then decide whether to apply. If we cannot establish a reliable write freeze, the migration stops.
That stop condition trades a maintenance window for a race we can neither explain nor recover with confidence.
The restore rehearsal retires the old key
A committed transaction proves that rewrapping finished in the database. It says nothing about whether a restored database still matches the independently held keyring, or whether a runtime can materialize the expected workload secrets.
After apply, the workflow runs the plan again and expects every record to be active. We then execute a harmless deployment fixture and read back only the workload’s variable-key set and count. The retained key remains available through that verification, an observation window, and a database restore rehearsal.
The rehearsal creates an independent database project from a production physical backup, then runs read-only connectivity, keyring-readiness, and cleanup checks. Supabase’s Restore to a new project copies the database schema, data, roles, and permissions, while Storage objects, Edge Functions, Auth settings, API keys, and several other project settings need separate handling. Extensions capable of external side effects also need to be disabled in the copy. A normal preview branch does not exercise the same backup and restore path.
We still track the live restore rehearsal, any legacy adoption, and eventual retained-key retirement as separate operational gates. A merged implementation and a green test suite do not close them automatically.
A rotation review checklist
Before the next control-plane key rotation, we want clear answers to six questions:
- Do ciphertext and the only usable key share an access or recovery failure domain?
- Can the runtime read active and retained keys, and does it fail closed when either required key is missing?
- Can key material enter Terraform state, CI logs, pull requests, shell history, or chat?
- Does apply bind to the reviewed current plan, or can it consume a stale dry-run?
- Beyond the database transaction, is there an immutable backup reference and a restore path that has actually run?
- Which evidence must pass before the retained key can be removed?
The useful outcome of rotation is a sequence in which every intermediate state has a defined read path, stop condition, and recovery proof.
For the wider deployment-recovery contract, see our notes on recovery paths after a failed deployment and single-host Compose rollback boundaries.