Skip to content

IronClaw Release Runbook

Operational guide for cutting, verifying, and yanking an IronClaw release.

Owner: Relay (Release Engineer). Scope: .github/workflows/release.yml, .github/workflows/image.yml, scripts/install.sh, scripts/install.ps1.

This runbook documents the pipeline as it ships on main. Where a section names an in-flight ticket (e.g. the post-release smoke gate), that capability is landing separately and the runbook is updated when it merges.

Trust model (non-negotiable). A release a user cannot verify is not a secured release. Every published set is checksummed, the checksum file is signed keylessly with cosign, and every archive carries a build-provenance attestation tied to the source commit and workflow. Never weaken or skip signing, checksums, or attestation to make a build go green — yank a bad release instead (see Yanking a release).

1. Pipeline at a glance

A release is cut automatically on every push to main (and can be run manually). Three GitHub Actions workflows are involved:

Workflow File Trigger Produces
CI ci.yml every push + PR build / vet / test with CGO_ENABLED=1 (gating check)
Release release.yml push to main, or workflow_dispatch tag, GitHub Release, archives, SHA256SUMS, SBOMs, cosign signature, attestations
Image image.yml workflow_run after Release succeeds, or workflow_dispatch GHCR control-plane image (ghcr.io/<owner>/ironclaw-controlplane) + image attestation

The Release workflow runs as a chain of jobs, each gated on the previous:

version ──> build (5-target matrix) ──> release ──> [smoke]
   │             │                          │           │
 derive tag   CGO build per OS/arch    checksums,    install via install.sh
 (skip if     archive + upload         GH Release,   on each target, assert
  already                              SBOM, cosign  "Checksum OK" + version
  tagged)                              sign, attest  (fail-closed gate)
  • build uses fail-fast: false but release needs: [version, build], so a release is published only if every matrix target built — no partial release sets.
  • All post-publish steps (SBOM, cosign signature, attestation) run after the binaries and checksums are uploaded, so a Sigstore/tooling hiccup cannot block the artifacts from shipping. See Partial-failure semantics — this is the one case that needs an operator decision.

2. How the version tag is derived

The version is v<BASE_VERSION>.<total commit count>, e.g. v0.1.123.

  • BASE_VERSION is hard-coded in the version job of release.yml (currently 0.1). Bump it there to roll the major/minor (e.g. set 0.2 to start the v0.2.x line).
  • The patch number is git rev-list --count HEAD — the total number of commits reachable from the released commit. It is monotonic, needs no manual bumps, and ties the tag to one exact commit.
  • The resolved tag is stamped into the binaries at build time via -ldflags "-X github.com/IronSecCo/ironclaw/internal/version.Version=<tag>", so ironctl version reports the exact release tag. (Unstamped/source builds report dev.)

Idempotency / re-run safety. The version job checks whether the tag already exists (git rev-parse --verify refs/tags/<tag>). If it does, exists=true and the build, release, and smoke jobs are all skipped (if: needs.version.outputs.exists == 'false'). This means re-running Release on a commit that is already released is a safe no-op — it will not overwrite or duplicate an existing release.

Consequence for yanking. Because the patch number is the commit count, the only way to get the same tag again is to release the same commit. If you delete a tag and push a new commit, the count increments and you get a new tag — you cannot accidentally collide with a yanked tag's number. If you delete a tag and re-run Release on the same commit, it rebuilds cleanly (exists is false again).

3. How to cut a release

3.1 The normal path (automatic)

Merge to main. The Release workflow fires on push: branches: [main], derives the next v0.1.<count> tag, builds the matrix, and publishes the Release + tag at the merged commit. No manual action is required. Releases are serialized (concurrency: group: release, cancel-in-progress: false) so two pushes never race or orphan a tag.

Watch the run:

gh run list --workflow=release.yml --limit 5
gh run watch <run-id>

A green Release run means: all five archives built, SHA256SUMS written and signed, SBOMs and the cosign signature/cert attached, and provenance attested. The Image workflow then chains off the success and publishes the GHCR image for that commit.

3.2 Manual dispatch (re-cut or pin a specific tag)

Use workflow_dispatch when you need to re-run a release or stamp a specific tag:

# Auto-derive the tag (same as a push to main):
gh workflow run release.yml

# Override the tag explicitly (e.g. to re-cut after a yank, or hotfix a specific number):
gh workflow run release.yml -f version=v0.1.99

If the supplied/derived tag already exists, the run is a safe no-op (see §2).

3.3 The build matrix (must stay in sync with the README)

OS / arch Runner C toolchain
darwin/amd64 macos-14 clang, cross via CGO_CFLAGS/LDFLAGS=-arch x86_64 on the universal SDK
darwin/arm64 macos-14 native clang
linux/amd64 ubuntu-latest native gcc
linux/arm64 ubuntu-latest cross aarch64-linux-gnu-gcc (apt: gcc-aarch64-linux-gnu)
windows/amd64 windows-latest native mingw-w64 gcc

Every target builds with CGO_ENABLED=1, Go 1.23 — CGO is mandatory because the encrypted-SQLite (SQLCipher) binding compiles a vendored C amalgamation. A pure-Go assumption will break the build. If you change the matrix, confirm each target still compiles with cgo, and update the README's Platform support / Installation tables to match — a platform that silently drops out of the matrix is a release defect.

3.4 What a successful release contains

Attached to the GitHub Release for tag <tag> (version <ver> = tag without the leading v):

  • ironclaw_<ver>_<os>_<arch>.tar.gz (and .zip for Windows) — one per matrix target. Each archive holds ironctl, ironclaw-controlplane, ironclaw-sandbox, LICENSE, README.md.
  • SHA256SUMS — checksums of every archive (the trust anchor).
  • SHA256SUMS.sig + SHA256SUMS.pem — the keyless cosign signature and its certificate.
  • ironclaw_<ver>.spdx.json + ironclaw_<ver>.cdx.json — SBOMs (syft, SPDX + CycloneDX).
  • Build-provenance attestations for each archive and for each raw binary (ironctl, ironclaw-controlplane, ironclaw-sandbox on every platform), so gh attestation verify works whether you point it at the downloaded .tar.gz/.zip or at a binary extracted from it.

3.5 Post-release verification gate (smoke — in flight, IRO-15)

A smoke job installs the freshly-cut release through the real, checksum-verifying scripts/install.sh (the normal user path, not --dev) on linux/amd64, linux/arm64, darwin/arm64, and darwin/amd64, asserts the installer printed Checksum OK, and asserts ironctl version reports the exact tag. It is fail-closed: a failure turns the whole Release run red, which also blocks the Image workflow (it chains on Release success). A red smoke run on an already-published release is the signal to yank (the assets are out by that point). This gate lands with IRO-15.

4. How to verify a release (user-facing)

This is the procedure a user — or you, post-release — runs to prove a release is trustworthy. It is also documented in the README's Verifying a release section. The trust chain is: cosign signature → SHA256SUMS → your archive, plus an independent provenance attestation.

Download the archive for your platform, plus SHA256SUMS, SHA256SUMS.sig, and SHA256SUMS.pem from the release.

Step 1 — verify the signature over SHA256SUMS (keyless cosign; no key to manage). The signing identity is the release workflow's OIDC identity, not a long-lived key:

cosign verify-blob SHA256SUMS \
  --signature SHA256SUMS.sig --certificate SHA256SUMS.pem \
  --certificate-identity-regexp '^https://github.com/IronSecCo/ironclaw/' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com

This proves SHA256SUMS was produced by the IronClaw release workflow and not tampered with.

Step 2 — verify your archive against the (now-trusted) checksum file:

sha256sum -c SHA256SUMS        # macOS: shasum -a 256 -c SHA256SUMS

Step 3 — verify build provenance (ties the artifact to the source commit + workflow):

# the downloaded archive:
gh attestation verify ironclaw_<ver>_<os>_<arch>.tar.gz --repo IronSecCo/ironclaw
# a binary extracted from it (e.g. after `tar xzf`):
gh attestation verify ./ironctl --repo IronSecCo/ironclaw
# and the container image:
gh attestation verify oci://ghcr.io/ironsecco/ironclaw-controlplane:<tag> --repo IronSecCo/ironclaw

The installers verify automatically. scripts/install.sh and scripts/install.ps1 download SHA256SUMS and refuse to install on a checksum mismatch (install.sh prints Checksum OK on success and dies on mismatch; install.ps1 throws on mismatch). The installers do not perform the cosign/attestation steps — run those manually (Steps 1 & 3) when you need full supply-chain assurance beyond the checksum.

5. Partial-failure semantics (operator decision)

The release job uploads the binaries + SHA256SUMS first, then attaches SBOMs, the cosign signature, and attestations. This ordering guarantees a tooling outage can't withhold the binaries — but it means a failure after publish can leave a release that is published but not fully signed/attested. That is not a verifiable release.

If a Release run goes red after the Publish release step:

  1. Check which post-publish step failed (gh run view <run-id> --log-failed).
  2. Re-run the failed job (gh run rerun <run-id> --failed). The signing/SBOM/attest steps --clobber their uploads, so re-running is safe and idempotent and will complete the set.
  3. If re-running cannot complete the signature/attestation (e.g. Sigstore is down for an extended window), yank the release rather than leave an unverifiable set published. Do not advertise or chain an image off a partially-signed release.

Never hand-sign or hand-upload a SHA256SUMS.sig from a local key — signing is keyless/OIDC by design; there is no long-lived signing secret. If you encounter one, stop and escalate.

6. Yanking a release

Yank when a published release is bad: a smoke failure, a broken/partially-signed artifact, a critical defect, or a wrong tag. Prefer a yanked release over a misleading green one.

TAG=v0.1.123          # the bad tag

# 1. Delete the GitHub Release AND its tag (so install.sh can no longer resolve it).
gh release delete "$TAG" --yes --cleanup-tag
#    (equivalently: gh release delete "$TAG" --yes && git push origin ":refs/tags/$TAG")

# 2. If the yanked release was marked --latest, repoint "latest" to the last good release
#    so `install.sh` (default: latest) stops serving the bad one.
gh release edit <previous-good-tag> --latest

# 3. Remove or repoint the GHCR image tags built from the bad commit.
#    Delete the version tag, and repoint :latest to the last good image if needed.
#    (GHCR package versions are managed under the repo owner's Packages settings / API.)
gh api -X DELETE "/orgs/IronSecCo/packages/container/ironclaw-controlplane/versions/<version-id>"

Notes:

  • Deleting the tag frees the v0.1.<count> number only for the same commit (see §2). To ship a fix, push the fix to main; the commit count increments and a fresh tag is cut. To re-cut the same commit after fixing tooling (not code), delete the tag and re-run Release — exists is false again and it rebuilds cleanly.
  • install.sh/install.ps1 default to the GitHub latest release, so Step 2 is what actually stops new installs of a yanked build. Users who pinned IRONCLAW_VERSION=<bad tag> will get a clean "no asset / release not found" error once the release is deleted — which is the intended fail-closed behavior.
  • Announce the yank (and the replacement tag) wherever releases are tracked, and record it on the triggering issue.

7. Pausing & resuming the pipeline

If prebuilt releases need to be paused (the README may carry a "paused" banner directing users to build from source), pause/resume cleanly without deleting the workflow:

gh workflow disable release.yml     # stop auto-cutting releases on push to main
gh workflow disable image.yml       # (optional) also stop image publishes
# ...resume:
gh workflow enable release.yml
gh workflow enable image.yml

When the pipeline is paused, the README directs users to build from source. When you resume, update the README's Installation / Verifying a release notes to drop the "paused" banners so the one-liner install path is advertised again.

8. Required status checks & branch protection

main is intended to be protected by enforced required checks. The spec lives at .github/rulesets/main.json (build + CodeQL required, linear history, signed commits, no force-push/deletion). Applying that file as an active GitHub ruleset is tracked separately (IRO-14); ratcheting protection up (more enforced checks) is the default direction. Never relax or disable a required check to unblock a merge — fix the check on its own ticket instead.

9. Quick command reference

# Watch the current release
gh run list --workflow=release.yml --limit 5
gh run watch <run-id>

# Manually cut / re-cut
gh workflow run release.yml                 # auto-derive tag
gh workflow run release.yml -f version=v0.1.99

# Re-run a failed (post-publish) release job — idempotent
gh run rerun <run-id> --failed

# Inspect a release
gh release view <tag>
gh release view <tag> --json assets -q '.assets[].name'

# Yank
gh release delete <tag> --yes --cleanup-tag
gh release edit <previous-good-tag> --latest

# Verify (user path)
cosign verify-blob SHA256SUMS --signature SHA256SUMS.sig --certificate SHA256SUMS.pem \
  --certificate-identity-regexp '^https://github.com/IronSecCo/ironclaw/' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com
sha256sum -c SHA256SUMS
gh attestation verify ironclaw_<ver>_<os>_<arch>.tar.gz --repo IronSecCo/ironclaw
gh attestation verify ./ironctl --repo IronSecCo/ironclaw   # extracted binary

Related tickets: pipeline handoff IRO-12; this runbook IRO-16; arm64 image IRO-13; ruleset enforcement IRO-14; release smoke test IRO-15.