Securing Prometheus with TLS: HTTPS Configuration for the Metrics Scrape Endpoint

Prometheus is almost always deployed without TLS because the documentation used to treat it as an internal-only tool. By the time someone asks “is this metrics data sensitive?”, the server has been running for a year on port 9090 without encryption. This article covers enabling HTTPS on Prometheus itself — the scrape endpoint and the web UI — and also securing the connections Prometheus makes when scraping TLS-enabled exporters.

Why Prometheus needs TLS

Prometheus metrics can expose internal hostnames, IP addresses, CPU/memory values, queue depths, and error rates. This is operational intelligence that attackers use for reconnaissance. More practically: if your Prometheus instance is reachable from a developer laptop and you do not use TLS, anyone who can intercept that traffic can observe your infrastructure topology.

Prometheus added native TLS support in version 2.24.0 (released January 2021). If you are running anything older, upgrade first.

Two separate TLS concerns

1. Prometheus serving HTTPS — the web UI on port 9090 and the remote write/read endpoints.
2. Prometheus scraping HTTPS exporters — when a node exporter or application is configured to require TLS, Prometheus must present a client certificate or trust the exporter’s CA.

This article covers both.

---

Part 1 — Enabling HTTPS on the Prometheus web server

Generate or obtain a certificate

For an internal Prometheus instance, a certificate issued by your internal CA is the right choice. For a public-facing instance, use Let’s Encrypt:

certbot certonly --standalone -d prometheus.example.com

For an internal CA:

openssl req -x509 -newkey rsa:4096 -keyout /etc/prometheus/prometheus.key \
  -out /etc/prometheus/prometheus.crt -days 365 -nodes \
  -subj "/CN=prometheus.example.com" \
  -addext "subjectAltName=DNS:prometheus.example.com"

Create the web configuration file

Prometheus reads TLS settings from a separate YAML file (not prometheus.yml). Create it:

sudo nano /etc/prometheus/web.yml

tls_server_config:
  cert_file: /etc/prometheus/prometheus.crt
  key_file:  /etc/prometheus/prometheus.key
  # Minimum TLS version — reject anything older
  min_version: TLS12
  # Optional: require client certificates (mutual TLS)
  # client_auth_type: RequireAndVerifyClientCert
  # client_ca_file: /etc/prometheus/ca.crt

Pass the web config file to Prometheus

Edit the systemd service or the startup script. On a standard Linux installation:

sudo systemctl edit prometheus

Add the --web.config.file flag:

[Service]
ExecStart=
ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus \
  --web.config.file=/etc/prometheus/web.yml \
  --web.listen-address=:9090

Fix permissions and restart

sudo chown prometheus:prometheus /etc/prometheus/prometheus.crt /etc/prometheus/prometheus.key
sudo chmod 640 /etc/prometheus/prometheus.key
sudo systemctl daemon-reload
sudo systemctl restart prometheus
sudo systemctl status prometheus

Verify HTTPS is active

curl -k https://localhost:9090/-/healthy
# Expected output: Prometheus Server is Healthy.

Remove -k once your certificate is trusted:

curl --cacert /etc/prometheus/prometheus.crt https://prometheus.example.com:9090/-/healthy

Or use openssl:

openssl s_client -connect prometheus.example.com:9090 -servername prometheus.example.com </dev/null 2>&1 \
  | openssl x509 -noout -dates

---

Part 2 — Configuring basic authentication alongside TLS

TLS encrypts the channel; it does not authenticate callers. Add basic auth to prevent unauthenticated access to metrics:

First, hash a password:

htpasswd -nBC 12 admin
# Output: admin:$2y$12$...

Add to web.yml:

basic_auth_users:
  admin: "$2y$12$..."

Grafana datasource configuration must then include the credentials:

• URL: https://prometheus.example.com:9090
• Basic auth: enabled, username admin, password your chosen password
• Skip TLS Verify: disabled (use the CA certificate field if Prometheus uses a private CA)

---

Part 3 — Scraping TLS-enabled exporters

Trusting an exporter’s self-signed or internal CA certificate

When an exporter (for example, node_exporter or a custom application) serves HTTPS with a certificate signed by an internal CA, add the CA to Prometheus’s scrape config:

scrape_configs:
  - job_name: node_exporter_tls
    scheme: https
    tls_config:
      ca_file: /etc/prometheus/internal-ca.crt
    static_configs:
      - targets: ['node1.example.com:9100']

Mutual TLS — presenting a client certificate when scraping

Some exporters require the scraper to present a certificate (mutual TLS). Generate a client cert signed by your internal CA:

# Generate client key and CSR
openssl genrsa -out /etc/prometheus/client.key 4096
openssl req -new -key /etc/prometheus/client.key \
  -out /etc/prometheus/client.csr \
  -subj "/CN=prometheus-scraper"

# Sign with your internal CA
openssl x509 -req -in /etc/prometheus/client.csr \
  -CA /etc/prometheus/ca.crt -CAkey /etc/prometheus/ca.key \
  -CAcreateserial -out /etc/prometheus/client.crt -days 365

Add to the scrape config:

scrape_configs:
  - job_name: secure_exporter
    scheme: https
    tls_config:
      ca_file:   /etc/prometheus/internal-ca.crt
      cert_file: /etc/prometheus/client.crt
      key_file:  /etc/prometheus/client.key
    static_configs:
      - targets: ['app-server.example.com:9100']

Per-target TLS configuration

You can override TLS settings per target using relabel_configs or by defining tls_config at the job level. If different targets use different CAs:

scrape_configs:
  - job_name: mixed_exporters
    scheme: https
    tls_config:
      ca_file: /etc/prometheus/default-ca.crt
    static_configs:
      - targets: ['server-a.example.com:9100']

  - job_name: special_exporter
    scheme: https
    tls_config:
      ca_file:            /etc/prometheus/special-ca.crt
      insecure_skip_verify: false
    static_configs:
      - targets: ['server-b.example.com:9100']

Never set insecure_skip_verify: true in production. It defeats the purpose of TLS.

---

Part 4 — Certificate renewal and automation

Let’s Encrypt certificates need renewal every 90 days. Add a post-renewal deploy hook:

sudo nano /etc/letsencrypt/renewal-hooks/deploy/prometheus.sh

#!/bin/bash
DOMAIN=prometheus.example.com
DEST=/etc/prometheus

cp /etc/letsencrypt/live/${DOMAIN}/fullchain.pem ${DEST}/prometheus.crt
cp /etc/letsencrypt/live/${DOMAIN}/privkey.pem   ${DEST}/prometheus.key
chown prometheus:prometheus ${DEST}/prometheus.crt ${DEST}/prometheus.key
chmod 640 ${DEST}/prometheus.key

# Prometheus reloads TLS config on SIGHUP without restarting
kill -HUP $(systemctl show prometheus --property=MainPID --value)

sudo chmod +x /etc/letsencrypt/renewal-hooks/deploy/prometheus.sh

Prometheus supports hot-reloading TLS certificates via SIGHUP — the web server picks up the new certificate file without restarting and without dropping active scrape connections.

---

Updating Alertmanager and Grafana after enabling Prometheus TLS

Once Prometheus requires HTTPS, all clients must be updated:

Alertmanager — if it reads Prometheus metrics, update its configuration to use HTTPS and supply any required CA bundle.

Grafana — update the datasource:

1. Go to Connections → Data Sources → Prometheus.
2. Change the URL to https://prometheus.example.com:9090.
3. In TLS settings, upload the CA certificate if you are using a private CA.
4. Save and test.

---

Troubleshooting

Symptom	Cause	Fix
x509: certificate signed by unknown authority	Prometheus scraping an exporter with a self-signed cert	Add the CA via ca_file in tls_config
remote error: tls: bad certificate	mTLS required but client cert not configured	Add cert_file and key_file to tls_config
Grafana shows “Bad Gateway” after TLS enabled	Grafana datasource still uses HTTP	Update datasource URL to https://
bind: permission denied on port 443	Non-root process cannot bind to privileged port	Use a reverse proxy or set cap_net_bind_service
Prometheus target shows “connection refused”	Old scrape URL uses http://	Update scheme: https in scrape config

---

Summary

Prometheus TLS works through a web configuration YAML file referenced at startup. Enable it, set cert_file and key_file, optionally add basic authentication, and restart the service. For scraping TLS-enabled exporters, configure tls_config per job with the appropriate CA bundle and, when required, client certificates. Certificate renewal is handled gracefully via SIGHUP without dropping connections, making Prometheus TLS one of the lower-maintenance components in your stack once it is set up correctly.