Loki TLS Configuration: Securing Log Ingestion and Querying with HTTPS

By /

Loki’s default configuration sends log data — including application errors, authentication events, and stack traces — over unencrypted HTTP. If Promtail, Grafana, or any other client is sending logs across a network boundary without TLS, those logs are readable in transit. This article shows how to enable HTTPS on Loki’s HTTP and gRPC endpoints, configure Promtail to use TLS when shipping logs, and set up mutual TLS when you want to restrict which clients can send data to Loki.

Loki’s network interfaces

Loki exposes two interfaces:

  • HTTP API (default port 3100) — used by Promtail for log push, by Grafana for queries, and for the /ready and /metrics endpoints.
  • gRPC (default port 9095) — used for internal cluster communication in multi-node deployments and optionally by gRPC-aware clients.

Both can be TLS-enabled independently. For most single-node deployments you only need to secure the HTTP interface.

Step 1: Obtain a certificate

For a public-facing Loki instance:

certbot certonly --standalone -d loki.example.com

For an internal instance use your internal CA or a self-signed certificate:

openssl req -x509 -newkey rsa:4096 \
  -keyout /etc/loki/loki.key \
  -out /etc/loki/loki.crt \
  -days 365 -nodes \
  -subj "/CN=loki.example.com" \
  -addext "subjectAltName=DNS:loki.example.com,IP:192.168.1.50"

Including the IP SAN is useful if Promtail agents connect via IP rather than hostname.

Step 2: Configure Loki to serve HTTPS

Loki’s configuration is a single YAML file, typically at /etc/loki/loki.yaml or /etc/loki/config.yaml. Open it and add or modify the server section:

server:
  http_listen_port: 3100
  grpc_listen_port: 9095

  # TLS for the HTTP interface
  http_tls_config:
    cert_file: /etc/loki/loki.crt
    key_file:  /etc/loki/loki.key

  # TLS for the gRPC interface (if used in a cluster)
  grpc_tls_config:
    cert_file: /etc/loki/loki.crt
    key_file:  /etc/loki/loki.key

If you need mutual TLS (Loki verifies client certificates), add:

  http_tls_config:
    cert_file:        /etc/loki/loki.crt
    key_file:         /etc/loki/loki.key
    client_ca_file:   /etc/loki/ca.crt
    client_auth_type: RequireAndVerifyClientCert

Step 3: Fix permissions

sudo chown loki:loki /etc/loki/loki.crt /etc/loki/loki.key
sudo chmod 640 /etc/loki/loki.key
sudo chmod 644 /etc/loki/loki.crt

If Loki runs as a different user (check with ps aux | grep loki), adjust the owner accordingly.

Step 4: Restart Loki and verify

sudo systemctl restart loki
sudo systemctl status loki
sudo journalctl -u loki -n 30 --no-pager

Verify the HTTPS endpoint responds:

curl --cacert /etc/loki/loki.crt https://loki.example.com:3100/ready
# Expected: ready

Use openssl to inspect the certificate being served:

openssl s_client -connect loki.example.com:3100 -servername loki.example.com </dev/null 2>&1 \
  | openssl x509 -noout -subject -dates

Step 5: Configure Promtail to use TLS

Promtail pushes logs to Loki via the HTTP API. After enabling HTTPS on Loki, every Promtail agent must be updated. Open each Promtail configuration file (often /etc/promtail/config.yaml) and update the clients section:

clients:
  - url: https://loki.example.com:3100/loki/api/v1/push
    tls_config:
      # Trust the CA that signed Loki's certificate
      ca_file: /etc/promtail/loki-ca.crt
      # If Loki requires mutual TLS, also provide a client cert:
      # cert_file: /etc/promtail/promtail.crt
      # key_file:  /etc/promtail/promtail.key
      # Set to true only during testing, never in production:
      insecure_skip_verify: false

Copy the Loki CA certificate (or the self-signed Loki certificate) to each Promtail host:

scp /etc/loki/loki.crt promtail-host:/etc/promtail/loki-ca.crt

Restart Promtail on each agent:

sudo systemctl restart promtail
sudo journalctl -u promtail -n 20 --no-pager

Check for a successful connection — you should see lines like:

level=info msg="Starting Promtail" version=...
level=info component=client host=loki.example.com:3100 msg="Flushing saved positions"

Step 6: Update the Grafana Loki datasource

In Grafana, navigate to Connections → Data Sources → Loki:

  1. Change the URL to https://loki.example.com:3100.
  2. Expand TLS settings.
  3. If Loki uses a private CA, toggle Skip TLS Verify off and paste the CA certificate content into CA cert.
  4. If mutual TLS is enabled, provide the client certificate and key.
  5. Click Save & test.

Issuing client certificates for mutual TLS

When you want strict control over which Promtail agents can send logs to Loki, mutual TLS is the right mechanism. Here is how to issue client certificates from your internal CA:

# On the CA host — generate a client key and CSR for a Promtail agent
openssl genrsa -out promtail-node1.key 4096
openssl req -new -key promtail-node1.key \
  -out promtail-node1.csr \
  -subj "/CN=promtail-node1.example.com/O=Monitoring"

# Sign the CSR with your internal CA
openssl x509 -req -in promtail-node1.csr \
  -CA /etc/loki/ca.crt -CAkey /etc/loki/ca.key \
  -CAcreateserial -out promtail-node1.crt -days 365

# Copy key and cert to the agent
scp promtail-node1.crt promtail-node1.key node1:/etc/promtail/

On the agent, reference the files in Promtail’s tls_config:

clients:
  - url: https://loki.example.com:3100/loki/api/v1/push
    tls_config:
      ca_file:   /etc/promtail/loki-ca.crt
      cert_file: /etc/promtail/promtail-node1.crt
      key_file:  /etc/promtail/promtail-node1.key

Automating certificate renewal

For Let’s Encrypt certificates, create a deploy hook:

sudo nano /etc/letsencrypt/renewal-hooks/deploy/loki.sh
#!/bin/bash
DOMAIN=loki.example.com
DEST=/etc/loki

cp /etc/letsencrypt/live/${DOMAIN}/fullchain.pem ${DEST}/loki.crt
cp /etc/letsencrypt/live/${DOMAIN}/privkey.pem   ${DEST}/loki.key
chown loki:loki ${DEST}/loki.crt ${DEST}/loki.key
chmod 640 ${DEST}/loki.key

systemctl reload-or-restart loki
sudo chmod +x /etc/letsencrypt/renewal-hooks/deploy/loki.sh

Unlike Prometheus, Loki does not support hot-reload of TLS certificates via signal in all versions — a full restart is safer. Test the renewal:

sudo certbot renew --dry-run

Troubleshooting Loki TLS

SymptomCauseFix
Promtail: x509: certificate signed by unknown authorityCA cert not trustedAdd ca_file pointing to the signing CA
Promtail: remote error: tls: certificate requiredLoki requires client cert, Promtail has noneAdd cert_file and key_file to Promtail tls_config
Loki: open loki.key: permission deniedKey not readable by loki userFix ownership with chown
Grafana shows Bad Gateway after TLS enabledDatasource URL still uses http://Update datasource URL
tls: no certificates configuredcert_file or key_file missing from Loki configAdd both fields to http_tls_config
Connection reset during high log volumeCertificate mismatch on reconnect after renewalEnsure renewal hook restarts Loki cleanly

Loki behind a reverse proxy

If you are using nginx or Traefik as a TLS terminator in front of Loki, set Loki to listen on plain HTTP (no http_tls_config) and proxy HTTPS traffic from the reverse proxy to Loki on localhost:

server {
    listen 443 ssl;
    server_name loki.example.com;
    ssl_certificate     /etc/letsencrypt/live/loki.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/loki.example.com/privkey.pem;
    ssl_protocols TLSv1.2 TLSv1.3;

    location / {
        proxy_pass http://127.0.0.1:3100;
        proxy_set_header X-Forwarded-Proto https;
    }
}

In this setup, TLS is handled entirely by nginx, and Promtail connects to https://loki.example.com but the traffic from nginx to Loki is unencrypted on localhost (acceptable if they are on the same host; not acceptable if they are on different hosts).

Summary

Loki TLS configuration lives entirely in the server.http_tls_config block of loki.yaml. Point it at a certificate and key, fix the file permissions, and restart the service. Every Promtail agent then needs its clients URL changed from http:// to https:// with a matching ca_file. The Grafana datasource needs the same URL update. For stronger access control, enable mutual TLS and issue per-agent client certificates. This setup keeps your log data encrypted both in transit and authenticated at the ingestion point.

Related Posts

Scroll to Top