Prometheus and Node Exporter for Linux Monitoring

By /

I like monitoring systems that make their assumptions obvious. Prometheus is one of them. It scrapes targets on a schedule, stores time series locally, evaluates rules, and fires alerts when conditions match. There is no mystery about where the numbers came from. If a graph looks wrong, I can usually inspect the raw metric in a few seconds and work backward.

That transparency is why I keep coming back to Prometheus for Linux infrastructure. It is not the only monitoring stack I use, and it is not automatically the right one for every environment. But for server metrics, service health, and quick operational visibility, Prometheus plus Node Exporter is still one of the most practical setups you can deploy.

This guide focuses on the part I actually use on Linux hosts: installing Prometheus, installing Node Exporter, writing a sane prometheus.yml, adding alert rules, thinking about retention, and not leaving the whole thing exposed to the internet with no authentication.

If you want a broader certificate-focused monitoring view as well, the SSL certificate alerting guide and the Zabbix certificate monitoring article are natural companions. For automated housekeeping around backups or reports, the systemd timers guide is also relevant because I often pair Prometheus with periodic maintenance jobs.

Understand the architecture before you install anything

Prometheus is usually easy to install. The harder part is understanding how it wants you to think.

Core pieces:

  • Prometheus server: scrapes targets, stores metrics, evaluates rules
  • exporters: expose metrics over HTTP, such as Node Exporter for Linux hosts
  • Alertmanager: receives alerts and handles routing, grouping, silencing, inhibition
  • Grafana: optional visualization layer
  • Pushgateway: used for short-lived batch jobs that cannot be scraped directly

Pull-based versus push-based monitoring

Prometheus is primarily pull-based. That means Prometheus calls the target.

I prefer this for infrastructure metrics because:

  • scrape timing is centralized
  • target configuration is easier to audit
  • failed scrapes are visible as failures, not silent data gaps
  • there is less agent-side state to manage

Push-based workflows still exist, but in Prometheus land they are the exception. Use Pushgateway for short-lived jobs, not as a generic replacement for scrapes.

A common mistake is trying to push everything because another monitoring system worked that way. That usually leads to awkward designs.

Install Prometheus on Linux

I usually install Prometheus from the official binary release when I want predictable upstream behavior. Distro packages can be fine, but they are sometimes older than I want.

Create a service account and directories:

sudo useradd --system --no-create-home --shell /usr/sbin/nologin prometheus
sudo mkdir -p /etc/prometheus /var/lib/prometheus
sudo chown prometheus:prometheus /var/lib/prometheus

Download and extract a release. Replace the version with the current one you intend to deploy:

cd /usr/local/src
sudo curl -LO https://github.com/prometheus/prometheus/releases/download/v2.54.1/prometheus-2.54.1.linux-amd64.tar.gz
sudo tar -xzf prometheus-2.54.1.linux-amd64.tar.gz

Install binaries and consoles:

sudo cp /usr/local/src/prometheus-2.54.1.linux-amd64/prometheus /usr/local/bin/
sudo cp /usr/local/src/prometheus-2.54.1.linux-amd64/promtool /usr/local/bin/
sudo cp -r /usr/local/src/prometheus-2.54.1.linux-amd64/consoles /etc/prometheus/
sudo cp -r /usr/local/src/prometheus-2.54.1.linux-amd64/console_libraries /etc/prometheus/
sudo chown -R prometheus:prometheus /etc/prometheus

Create prometheus.yml

A minimal starting config:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - /etc/prometheus/rules/*.yml

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets:
          - 127.0.0.1:9090

  - job_name: node
    static_configs:
      - targets:
          - 127.0.0.1:9100

I start small. Over-engineering discovery on day one is not a virtue.

Create the systemd service

[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus \
  --web.console.templates=/etc/prometheus/consoles \
  --web.console.libraries=/etc/prometheus/console_libraries \
  --storage.tsdb.retention.time=30d
Restart=on-failure

[Install]
WantedBy=multi-user.target

Save it as /etc/systemd/system/prometheus.service, then validate and start:

sudo promtool check config /etc/prometheus/prometheus.yml
sudo systemctl daemon-reload
sudo systemctl enable --now prometheus
sudo systemctl status prometheus --no-pager

If you want more detail on writing robust unit files, the systemd unit file guide is a good reference.

Install Node Exporter

Node Exporter is the easiest part of the stack. It exposes host-level Linux metrics over HTTP.

Create the service account:

sudo useradd --system --no-create-home --shell /usr/sbin/nologin node_exporter

Download and install it:

cd /usr/local/src
sudo curl -LO https://github.com/prometheus/node_exporter/releases/download/v1.8.2/node_exporter-1.8.2.linux-amd64.tar.gz
sudo tar -xzf node_exporter-1.8.2.linux-amd64.tar.gz
sudo cp /usr/local/src/node_exporter-1.8.2.linux-amd64/node_exporter /usr/local/bin/

Systemd unit:

[Unit]
Description=Prometheus Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter
Restart=on-failure

[Install]
WantedBy=multi-user.target

Save it as /etc/systemd/system/node_exporter.service, then start it:

sudo systemctl daemon-reload
sudo systemctl enable --now node_exporter
sudo systemctl status node_exporter --no-pager

Confirm the endpoint responds:

curl -fsS http://127.0.0.1:9100/metrics | head

If that command returns plain-text metrics, the exporter is alive.

What Node Exporter gives you

Node Exporter covers the standard Linux host metrics I care about most:

  • CPU usage and load
  • memory usage and pressure signals
  • disk space and inode availability
  • filesystem mount metrics
  • network counters and error rates
  • system uptime
  • entropy, file descriptor, and basic kernel stats

It does not replace application-specific exporters. That distinction matters.

If you want MySQL internals, use a MySQL exporter. If you want Nginx request metrics, use an Nginx exporter or app instrumentation. If you want TLS certificate expiry, Node Exporter alone is not enough.

TLS certificate expiry with Prometheus

This is where people often assume Node Exporter should do more than it does.

For certificate expiry monitoring, I usually use one of these:

  • Blackbox Exporter for probing HTTPS endpoints
  • x509 exporter for filesystem or endpoint certificate inspection
  • a custom script exporter for specific edge cases

A practical alert rule with Blackbox Exporter looks like this:

groups:
  - name: tls-alerts
    rules:
      - alert: TLSCertificateExpiringSoon
        expr: (probe_ssl_earliest_cert_expiry - time()) < 30 * 24 * 3600
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: TLS certificate expires within 30 days
          description: "{{ $labels.instance }} has a certificate expiring within 30 days"

I am calling that out explicitly because trying to invent certificate expiry from generic Node Exporter metrics is the wrong approach.

Add the scrape targets properly

Once Node Exporter runs on a monitored host, add it to Prometheus.

Single server example:

scrape_configs:
  - job_name: node
    static_configs:
      - targets:
          - 192.0.2.10:9100
          - 192.0.2.11:9100

If your Prometheus server cannot reach the exporter port, fix that before chasing config ghosts.

Quick network validation:

curl -fsS http://192.0.2.10:9100/metrics | head

Or test the TCP port:

nc -vz 192.0.2.10 9100

I usually bind exporters on internal addresses only and protect access with firewall rules.

PromQL basics I use constantly

PromQL is one of the reasons Prometheus is powerful. The basics are not hard, but they are easy to misuse.

rate()

Use rate() for steadily increasing counters across a time window.

Example: CPU busy percentage from idle time inversion:

100 * (1 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])))

I use rate() for most dashboard views because it smooths short spikes better than irate().

irate()

irate() uses the last two points and is more volatile.

irate(node_network_receive_bytes_total[5m])

This is useful when I want to see sharp short-term changes, but it is often too noisy for alerts.

histogram_quantile()

You will not use this much with Node Exporter itself, but it matters for application latency histograms.

Example:

histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))

That gives a 95th-percentile latency estimate based on histogram buckets. When a service exports histograms correctly, this is far more useful than simple averages.

Useful Linux queries I keep around

Memory available percentage:

100 * node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes

Filesystem free percentage ignoring ephemeral filesystems:

100 * node_filesystem_avail_bytes{fstype!~"tmpfs|overlay|squashfs"} /
node_filesystem_size_bytes{fstype!~"tmpfs|overlay|squashfs"}

Disk read throughput:

rate(node_disk_read_bytes_total[5m])

Network receive throughput:

rate(node_network_receive_bytes_total{device!="lo"}[5m])

Recording rules: do the expensive math once

If you reuse the same longer PromQL expressions in multiple dashboards or alerts, recording rules help.

Example:

groups:
  - name: node-recording-rules
    rules:
      - record: instance:node_cpu_utilisation:avg5m
        expr: 100 * (1 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])))

      - record: instance:node_memory_available:percent
        expr: 100 * node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes

That makes dashboards simpler and can reduce repeated query cost.

I do not create recording rules for everything. Only for queries I know I will reuse heavily.

Node Exporter collectors: do not enable everything blindly

Node Exporter exposes metrics through collectors, and most default collectors are sensible. Still, I pay attention to what is enabled, especially on very large or unusual hosts.

To see active collector behavior quickly:

curl -fsS http://127.0.0.1:9100/metrics | grep '^node_' | head

If I need to disable a noisy or irrelevant collector, I do it explicitly in the service unit. Example:

ExecStart=/usr/local/bin/node_exporter   --collector.systemd   --no-collector.wifi

The exact collector mix depends on the host role. I do not turn collectors on and off for sport, but I also do not assume defaults are perfect for every workload. Monitoring should be useful, not just verbose.

Service discovery grows up with the environment

Static targets are fine to start with. Once the estate grows, I usually move to some kind of service discovery or generated target file so Prometheus does not become a manual spreadsheet.

A simple file-based discovery example:

scrape_configs:
  - job_name: node
    file_sd_configs:
      - files:
          - /etc/prometheus/file_sd/node_targets.yml

And then a target file such as /etc/prometheus/file_sd/node_targets.yml:

- targets:
    - 192.0.2.10:9100
    - 192.0.2.11:9100
  labels:
    env: production

I like file-based discovery because it is easy to generate from CMDB data, Ansible, or a small inventory script without pulling in a full orchestration stack on day one.

Alertmanager: where alerts become usable

Prometheus can fire alerts, but Alertmanager is what makes them operationally tolerable.

A minimal Alertmanager config:

global:
  resolve_timeout: 5m

route:
  receiver: email-default
  group_by: ['alertname', 'instance']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h

receivers:
  - name: email-default
    email_configs:
      - to: ops@example.com
        from: prometheus@example.com
        smarthost: smtp.example.com:587
        auth_username: prometheus@example.com
        auth_password: ReplaceWithRealPassword
        require_tls: true

inhibit_rules:
  - source_matchers:
      - severity = critical
    target_matchers:
      - severity = warning
    equal:
      - alertname
      - instance

The inhibition rule above stops warning alerts from duplicating critical alerts for the same condition and instance. I use inhibition a lot because alert floods are how good monitoring gets ignored.

Example alert rules I actually find useful

Disk space warning:

groups:
  - name: node-alerts
    rules:
      - alert: HostDiskSpaceLow
        expr: (node_filesystem_avail_bytes{fstype!~"tmpfs|overlay|squashfs",mountpoint="/"} /
              node_filesystem_size_bytes{fstype!~"tmpfs|overlay|squashfs",mountpoint="/"}) < 0.15
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: Root filesystem low on space
          description: "{{ $labels.instance }} has less than 15% free space on /"

CPU usage high:

- alert: HostCPUHigh
  expr: 100 * (1 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m]))) > 90
  for: 10m
  labels:
    severity: warning
  annotations:
    summary: CPU usage is high
    description: "{{ $labels.instance }} CPU usage has been above 90% for 10 minutes"

Memory pressure:

- alert: HostMemoryLow
  expr: (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) < 0.10
  for: 10m
  labels:
    severity: warning
  annotations:
    summary: Available memory is low
    description: "{{ $labels.instance }} has less than 10% available memory"

TLS expiry again, with Blackbox Exporter metrics:

- alert: TLSCertificateCritical
  expr: (probe_ssl_earliest_cert_expiry - time()) < 7 * 24 * 3600
  for: 15m
  labels:
    severity: critical
  annotations:
    summary: TLS certificate expires within 7 days
    description: "{{ $labels.instance }} certificate expires within 7 days"

I prefer a 30-day warning and a 7-day critical threshold. That gives enough time for normal certificate renewal work without training everyone to ignore expiry warnings.

Retention and storage: decide on purpose, not default

Prometheus local storage is simple, but retention still needs a conscious decision.

Common startup flag:

--storage.tsdb.retention.time=30d

Or size-based retention:

--storage.tsdb.retention.size=50GB

If you do nothing, the default behavior may be acceptable for a lab, but I prefer to define it explicitly so disk usage is predictable.

What influences retention choice:

  • number of targets
  • scrape interval
  • metric cardinality
  • whether local Prometheus is the only long-term store
  • available disk space

For short-lived infrastructure, 15 to 30 days locally is often enough. For long-term trend analysis, I offload elsewhere.

remote_write for long-term storage

Prometheus local TSDB is not meant to be your forever archive in every environment.

A basic remote_write example:

remote_write:
  - url: https://mimir.example.com/api/v1/push
    basic_auth:
      username: prometheus
      password: ReplaceWithRealPassword

Common long-term destinations include systems like Thanos, Cortex, or Mimir. The right choice depends on scale and operational taste.

I do not add remote storage on day one unless I know I need it. But I plan for it early if the environment is growing fast.

Secure Prometheus: it has no auth by default

This is the warning I wish more guides put near the top.

Prometheus does not ship with built-in authentication in the simple sense many admins expect. If you bind it to a public interface and call it a day, you are exposing:

  • metric data
  • target inventory
  • labels that may reveal internal naming
  • alert configuration details

That is bad practice.

My defaults:

  • bind Prometheus on a private interface when possible
  • restrict access with a firewall or reverse proxy
  • add authentication in the fronting layer if users need access
  • never expose exporters blindly to the public internet

Quick firewall example with UFW:

sudo ufw allow from 192.168.1.0/24 to any port 9090 proto tcp
sudo ufw allow from 192.168.1.0/24 to any port 9100 proto tcp

If you need a stronger baseline host security posture overall, the Linux server hardening checklist is worth applying alongside monitoring.

Grafana integration

Prometheus works without Grafana, but Grafana is still the fastest way to hand usable dashboards to other people.

Basic workflow:

  1. install Grafana
  2. add Prometheus as a data source
  3. point it at http://prometheus-server:9090
  4. import a known-good dashboard

For Node Exporter, dashboard 1860 is the usual starting point.

That dashboard is not perfect, but it gets a team from zero to useful visibility very quickly. After that, I trim panels that do not matter and add environment-specific ones.

Troubleshooting checks I run first

When graphs are empty or alerts never fire, I check these before touching anything fancy:

  1. does the exporter respond with metrics?
  2. does Prometheus list the target as UP?
  3. does promtool check config pass?
  4. is firewall or routing blocking the scrape?
  5. is the query using the correct metric name and label set?

Useful commands:

curl -fsS http://127.0.0.1:9100/metrics | head
curl -fsS http://127.0.0.1:9090/-/healthy
sudo promtool check config /etc/prometheus/prometheus.yml
sudo journalctl -u prometheus -n 100 --no-pager
sudo journalctl -u node_exporter -n 100 --no-pager

That small set catches most setup mistakes.

Final thoughts

Prometheus plus Node Exporter is still one of the cleanest ways to monitor Linux infrastructure. The model is simple, the data is inspectable, and the tooling around it is mature enough that I can get from a fresh server to useful graphs and alerts quickly.

The practical lessons are not complicated. Keep the initial configuration small. Validate the exporter endpoint before editing PromQL. Use recording rules for repeated expensive queries. Let Alertmanager handle grouping and inhibition so people do not drown in duplicate noise. Monitor TLS expiry with the right exporter instead of forcing Node Exporter to do the wrong job. And do not expose Prometheus or exporters publicly without a protection layer.

Do that, and you end up with monitoring that is boring in the best possible way: dependable, understandable, and easy to troubleshoot when something drifts.

Related Posts

Scroll to Top