Skip to content

How to harden a Prometheus container (and is prom/prometheus:v3.1.0 safe for your cluster?)

Prometheus is the monitoring brain of most clusters: it scrapes every target you run, holds a time-series record of how the whole system behaves, and often carries the alerting rules that page your team. That reach makes it a high-value target, and a stock docker run prom/prometheus:v3.1.0 does not fully earn the trust the role implies. Graded on IronClaw's seven-dimension containment scale, the default configuration scores 63 of 100, grade C (weak). Higher is safer. The image runs as the nobody uid out of the box, which is better than most, but it keeps the full capability set and a writable root filesystem. A few runtime flags take the same image to 89 of 100, grade B, one point off an A. The one dimension it cannot reach is the one a metrics server needs by definition: it has to scrape targets and serve its API. Here are the exact gaps and fixes from the scan data.

Graded from a read-only inspect of a running container started from prom/prometheus:v3.1.0 with plain docker run defaults, its entrypoint overridden with sleep purely to keep it alive. The scan itself executes nothing inside the container. It is the same data behind its isolation scorecard. How scoring works →

Where the default configuration leaks

ironctl scan grades seven independent containment boundaries. On a default docker run prom/prometheus:v3.1.0, two fail and one warns:

Dimension Verdict Score What the scan found
Non-root user (uid != 0) ✅ PASS 15/15 runs as nobody (uid != 0)
Dropped capabilities ❌ FAIL 4/20 default capability set retained (CAP_NET_RAW, CAP_MKNOD, and more)
Seccomp profile ✅ PASS 15/15 seccomp profile active
Network isolation / egress ⚠️ WARN 4/15 network=bridge: outbound egress is possible
Read-only root filesystem ❌ FAIL 0/10 root filesystem is writable
No docker.sock exposure ✅ PASS 15/15 no control socket mounted
No shared host namespaces ✅ PASS 10/10 no host PID/IPC/network sharing

The two that should worry you most are capabilities and egress. Prometheus already reaches out across your network by design, so an egress-capable Prometheus that gets code execution through a scrape-target parsing bug or an exporter CVE can pivot anywhere it can already route, and exfiltrate the full time-series history of your infrastructure. The default capability set gives such a foothold CAP_NET_RAW for raw-socket tricks, and the writable rootfs lets it persist.

Harden it: the exact --fix remediation

ironctl scan my-prometheus --fix prints one remediation per failed dimension, then one hardened run. For prom/prometheus:v3.1.0:

  • --cap-drop=ALL (Dropped capabilities, +16): drop every Linux capability. Prometheus scrapes over ordinary TCP and binds a high port; it needs none of the default set.
  • --read-only --tmpfs /tmp (Read-only rootfs, +10): make the root filesystem read-only and mount the TSDB path (/prometheus) as an explicit writable volume the nobody uid owns. Removes the persistence surface.
  • Scoped network (Network isolation): --network=none scores the full 15 but is wrong for a monitoring server, it has to reach its scrape targets and serve its query API. Any named or bridge network scores 4 of 15 (a WARN, not a fail): a connection path exists. Contain it anyway: attach a user-defined network scoped to just the exporters it scrapes and the Grafana or Alertmanager that read from it, with no default route out, so a compromised Prometheus cannot call arbitrary internet addresses.

The uid is already non-root, so that dimension is a PASS out of the box. Keep it that way: own the TSDB volume with the nobody uid rather than reverting to root to sidestep a permissions error.

Before and after

# Before: 63/100, grade C
docker run -d --name prometheus prom/prometheus:v3.1.0

# After: 89/100, grade B (scoped private network for scrape targets and readers)
docker run -d --name prometheus-hardened \
  --user nobody \
  --cap-drop=ALL \
  --security-opt=no-new-privileges \
  --read-only --tmpfs /tmp \
  -v prometheus-data:/prometheus \
  --network=observability-internal \
  prom/prometheus:v3.1.0

Rescan: ironctl scan prometheus-hardened reports 89/100 grade B. A 26-point swing with no custom image build, just the right flags. The only dimension still short of full marks is the network (4 of 15), because a monitoring server exists to reach out and be queried; network=none would score the last points but leave nothing to scrape and no API to serve. That is the honest ceiling for a metrics server, and it is a clear step up from the default C.

Verify it on your own Prometheus

# install (Homebrew)
brew install ironsecco/ironclaw/ironclaw

# grade your running container, then print the fixes
ironctl scan my-prometheus
ironctl scan my-prometheus --fix

ironctl scan also reads a docker-compose.yml service or a Kubernetes manifest, so you can grade the Prometheus in your stack, not just a bare docker run.

Keep going