A rollback button is not a recovery plan

A single-host Docker Compose update can fit in one line:

docker compose pull && docker compose up -d

That line is useful. It is not a recovery plan.

A recovery plan has to answer harder questions. Was the new image available before the running service changed? Which container is the application target in a multi-service file? Did a real request pass, or did compose up merely exit with code zero? Is the previous verified version still identifiable? If the update fails, what can the platform restore automatically—and what still needs a database, volume, DNS, or provider-specific procedure?

We build Appaloft, an open-source deployment control plane, and this distinction exposed a gap in our own Compose runtime. Appaloft already retained successful deployment snapshots and could create rollback attempts, but its local and SSH Compose adapters treated a successful compose up as deployment success. Normal updates also did not explicitly pre-pull every image-backed service. We fixed those two boundaries before writing this article.

The result is not “automatic recovery for everything.” It is a smaller, testable contract for application updates on one Docker host.

The signals were ordinary, which is why they matter

A July 9 ALM System deployment request described the expected shape clearly: keep data outside the image, back up SQLite before an update, run startup and migrations, verify health, restore the previous image on failure, and retain the failure details. That is much more than a rollback button.

In a July 11 Pi-hole update incident, the service being updated was also part of the host’s DNS path. Stopping it before the replacement image was available made the pull depend on the service that had just been removed. The precise Compose behavior was debated, but the operational lesson is stable: acquire required artifacts before taking away a dependency you may need to acquire them.

On July 13, an rss.chat self-hosted deployment report described driving the application through its API rather than only through the browser. That path found a process-fatal first-post bug that normal browser use did not trigger. A container can be running while the request path operators care about is still broken.

Invariant 1: acquire the candidate before replacement starts

For image-backed Compose services, the pull is a preflight step. If it fails, the candidate project should not start and the running version should not be touched.

Appaloft now renders the Compose sequence in this order:

docker compose pull --ignore-buildable &&
docker compose build --pull &&
docker compose up -d --build

--ignore-buildable handles Compose files that mix registry images and locally buildable services. Docker documents the behavior of docker compose pull.

This sequence requires the supported docker compose v2/v5 CLI. Appaloft fails before candidate mutation when a target only provides legacy docker-compose v1. Docker lists Compose v1 among its retired products; quietly falling back would weaken the preflight contract.

Pre-pulling does not make the rollout atomic. A later build can still fail, and a fixed host port can prevent two versions from overlapping. The invariant is narrower: an unavailable image must not be discovered only after replacement has begun.

Invariant 2: image state and mutable state are different recovery domains

Useful state may live in databases, volumes, bind mounts, secret references, uploaded files, DNS, TLS, queues, object storage, and other external dependencies.

Rolling the application image back does not reverse a schema migration. It does not restore deleted rows, rewind a volume, remove a DNS record, or undo an external API call. If a migration is not backward-compatible, keeping the old image is not enough. The operator needs a tested backup and restore path, or a forward-compatible migration strategy.

Appaloft models application rollback and storage backup/restore as separate operations. Application rollback starts a new deployment attempt from a retained successful snapshot. It does not claim to roll back databases, volumes, DNS, TLS, or external dependencies automatically.

Invariant 3: compose up is apply evidence, not success evidence

docker compose up -d returning zero says that Compose accepted and applied the change. It does not prove that the intended service exists, stayed running, passed native health, answered on its configured port, or received traffic through the expected public route.

Appaloft now identifies candidate containers with deployment ownership labels and, for routed multi-service stacks, requires an explicit target service. Verification checks container state and Docker health, the target-service HTTP policy when configured, and the route through the edge proxy when public verification is required.

Guessing which of web, worker, db, and redis should receive proxy labels is an ambiguous mutation. Appaloft may infer the target only when the Compose model contains exactly one service.

The route check is distinct from certificate readiness. A local proxy-path check can prove that the Host rule reaches the candidate. It cannot prove that public DNS has propagated everywhere or that an ACME provider has issued a trusted certificate.

Invariant 4: keep the previous target and evidence until verification passes

A recoverable update needs more than an old tag. It needs an exact previous target and evidence such as the source revision, image digest, redacted configuration, target identity, health and route observations, failed phase, error kind, and timestamp.

Appaloft records the candidate as a new deployment attempt. If verification fails, cleanup selects containers by the failed deployment identity; it does not run a broad project or host prune. The previous successful runtime is released only after the candidate passes the required verification chain.

There are still limits. Candidate containers can be removed by identity, but user-defined Compose resources and external side effects may need their own cleanup rules. “Scoped cleanup” is not the same promise as “the host is byte-for-byte unchanged.”

Invariant 5: rollback must run the same checks as deploy

A rollback that only changes an image reference can reproduce the same bad state under an older version. The restored target still needs to start, pass native health, answer the configured request, and satisfy the route checks required for an ordinary deployment.

In Appaloft, rollback creates a new attempt from a retained successful deployment snapshot and sends it through the normal execution backend. Compose rollback therefore uses the same image preflight and candidate verification chain as a forward deployment. “Rollback requested” is not “service recovered.” Recovery is complete only when the restored application has fresh verification evidence.

What we changed in Appaloft

The public implementation is in appaloft/appaloft#682. It adds:

• preflight pulls before Compose build and up;
• fail-safe handling for legacy Compose v1;
• candidate and target-service container observation;
• native health, internal HTTP, and public-route verification;
• target-only proxy labels and edge-network attachment;
• deployment-identity cleanup for failed candidates;
• the same execution path for rollback attempts;
• lifecycle documentation and Test Matrix coverage.

We exercised the change with focused and full adapter suites and a real local Docker smoke covering Dockerfile, Compose, and prebuilt-image deployments. The smoke exposed three assumptions during implementation: a pull policy that tried to fetch a local-only fixture image, a Docker inspect template that assumed every container had a Health field, and an HTTP proxy that distorted a local route check. Verification code has its own failure modes.

Before the next Compose update

Write down five answers:

1. Are all pullable images available before the running version changes?
2. Which service is the request target, and what request proves it works?
3. Which previous image or artifact is retained by immutable identity?
4. Which database, volume, secret, DNS, TLS, and external side effects are outside application rollback?
5. What evidence distinguishes “rollback requested” from “recovery verified”?

If any answer is “the button handles it,” the recovery plan is not finished.

Appaloft’s deployment lifecycle documentation describes the plan, execute, verify, and recovery boundaries. For the broader failure model, see Deployment failures should leave a recovery path.