Nginx Upstream Load Balancing Configuration: What I Actually Tune in Production

By /

I like Nginx load balancing because it stays boring when the traffic pattern is boring, and that is a compliment. When a setup is healthy, most of the work is invisible. The trouble starts when the upstream layer is copied from an old blog post, nobody logs which backend answered the request, and an intermittent timeout turns into a two-hour argument about whether the app, Nginx, or the network is at fault.

This guide is how I set up and tune Nginx upstream load balancing in the real world. I am focusing on the upstream block itself, not on TLS termination or reverse proxy basics. If you need the proxy side first, read my Nginx reverse proxy configuration guide. If you are already seeing backend failures, my 502 troubleshooting guide and 504 timeout guide are the next places I would check.

What the upstream block does

The upstream block defines a pool of backend servers. A proxy_pass directive then sends requests to that named pool.

upstream app_backend {
    server 10.0.0.11:8080;
    server 10.0.0.12:8080;
}

server {
    listen 80;
    server_name example.com;

    location / {
        proxy_pass http://app_backend;
    }
}

That is the minimum working pattern. Nginx uses round-robin by default, so requests rotate between upstream servers in order. I still start simple like this in test environments because it is easy to verify and easy to debug. Only after I know traffic is flowing correctly do I add failure handling, keepalive tuning, and method-specific balancing.

A common mistake is treating the upstream block as just a list of IPs. It is more than that. It is where you decide how backends are selected, when a backend is considered failed, whether one server acts as backup, and whether state should be shared across workers.

Basic upstream syntax and server parameters

I prefer to write upstream blocks with explicit parameters once a service is heading to production:

upstream app_backend {
    zone app_backend 64k;

    server 10.0.0.11:8080 weight=3 max_fails=3 fail_timeout=30s;
    server 10.0.0.12:8080 weight=3 max_fails=3 fail_timeout=30s;
    server 10.0.0.13:8080 backup;
}

The useful parameters here are:

  • weight gives one server a larger share of traffic.
  • max_fails defines how many failed attempts are allowed.
  • fail_timeout defines the time window for those failures and the temporary disable interval.
  • backup keeps a server idle until the primary servers are considered unavailable.
  • down marks a server as unavailable without removing it from config.

down is handy when I want to keep the line in place during maintenance:

upstream app_backend {
    server 10.0.0.11:8080;
    server 10.0.0.12:8080 down;
    server 10.0.0.13:8080;
}

That is safer than deleting the server line in a hurry and trying to remember what changed later.

Round-robin, the default that often works fine

Default round-robin is good enough for a lot of application pools. If the app instances are nearly identical and each request costs about the same, I do not try to outsmart it.

upstream app_backend {
    server 10.0.0.11:8080;
    server 10.0.0.12:8080;
    server 10.0.0.13:8080;
}

Round-robin keeps things predictable. It also makes log analysis simpler because a sudden skew in traffic usually means one server is down or marked failed, not that the balancing algorithm is doing something subtle.

Where it falls down is long-lived requests, uneven backend performance, or mixed hardware. If one server has half the CPU of the others, I set weights. If one request can stay open for minutes, I consider least_conn instead.

Weighted balancing when servers are not identical

Weights are the first tuning knob I reach for.

upstream app_backend {
    server 10.0.0.11:8080 weight=4;
    server 10.0.0.12:8080 weight=2;
    server 10.0.0.13:8080 weight=1;
}

That ratio means server 10.0.0.11 should receive about four times as many requests as 10.0.0.13 over time. I use this when I am migrating between old and new hardware, or when one node is intentionally smaller.

What I do not do is use weight to hide a sick backend. If a server needs weight=1 because it starts falling over under normal traffic, that is an application or capacity problem, not a balancing strategy.

Least connections for uneven request duration

If request duration varies wildly, least_conn is usually the better choice.

upstream app_backend {
    least_conn;
    server 10.0.0.11:8080;
    server 10.0.0.12:8080;
    server 10.0.0.13:8080;
}

This method sends new requests to the server with the fewest active connections. I have had better results with it for APIs that mix fast reads with slow report generation, and for services that hold streaming connections open.

The trade-off is that connection count is only one signal. A backend with fewer open connections is not always the least busy backend. If your application is CPU-bound and a single request can pin a worker, least connections can still distribute poorly. It helps, but it is not a scheduler.

Sticky behavior: ip_hash and hash

Some apps still need a form of request affinity. That usually means sessions are stored in process memory, WebSocket connections need predictable routing, or a legacy app behaves badly if requests bounce between backends.

ip_hash

upstream app_backend {
    ip_hash;
    server 10.0.0.11:8080;
    server 10.0.0.12:8080;
    server 10.0.0.13:8080;
}

ip_hash tries to keep requests from the same client IP on the same backend. It is simple and sometimes good enough, especially for internal apps or small deployments.

The downside is obvious once you sit behind NAT, mobile carriers, or a corporate proxy. Hundreds of users can appear to come from a tiny number of public IPs. Then one backend gets hot while another does very little.

Generic hash

I prefer the generic hash method when I need affinity tied to something more meaningful than source IP.

upstream app_backend {
    hash $request_uri consistent;
    server 10.0.0.11:8080;
    server 10.0.0.12:8080;
    server 10.0.0.13:8080;
}

Or for application cookies:

upstream app_backend {
    hash $cookie_sessionid consistent;
    server 10.0.0.11:8080;
    server 10.0.0.12:8080;
    server 10.0.0.13:8080;
}

consistent reduces remapping when servers are added or removed. That matters for cache locality and sticky behavior. Without it, the distribution can shift more aggressively than you expect.

Sticky session trade-offs

I only keep sticky routing when I truly need it. Session affinity solves some operational pain, but it also creates new pain.

The main trade-offs are:

  • Hotspots are more likely.
  • Draining a node is harder because some users keep landing there.
  • Cache hit patterns become uneven.
  • Failover can feel worse because the app was relying on per-node state.

My preference is still to make the application stateless when possible. Shared sessions in Redis, a database, or signed client-side tokens are usually better long-term than depending on ip_hash. If you are proxying WebSockets, though, affinity is often practical. I cover that in more detail in my Nginx WebSocket proxy guide.

Shared state with zone

The zone directive is easy to overlook and worth using.

upstream app_backend {
    zone app_backend 64k;
    server 10.0.0.11:8080 max_fails=3 fail_timeout=30s;
    server 10.0.0.12:8080 max_fails=3 fail_timeout=30s;
}

Without a shared memory zone, each Nginx worker keeps its own view of upstream state. That can make failure handling less consistent. With zone, workers share runtime information about the upstream group. On busy systems with multiple workers, I consider that the sane default.

I do not wildly oversize the zone. For small upstream groups, tens of kilobytes is usually enough. The point is shared state, not throwing memory at the problem.

Failure handling: passive health checks

Open source Nginx gives you passive health checks. That means a server is considered failed based on real request failures.

upstream app_backend {
    zone app_backend 64k;
    server 10.0.0.11:8080 max_fails=3 fail_timeout=30s;
    server 10.0.0.12:8080 max_fails=3 fail_timeout=30s;
}

server {
    listen 80;
    server_name example.com;

    location / {
        proxy_next_upstream error timeout http_502 http_503 http_504;
        proxy_next_upstream_tries 2;
        proxy_pass http://app_backend;
    }
}

The upstream server is not proactively probed. It is judged by whether real traffic to it fails. That is often enough, but you need to understand the timing. If a backend is dead and no requests are hitting it, nothing changes until traffic reaches it. That is why accurate logging matters.

Also remember that proxy_next_upstream controls when Nginx should retry a different server. If you do not define sensible retry conditions, passive failure handling looks worse than it really is.

Active health checks: useful, but not in standard open source builds

Active health checks are the better operational model because Nginx can probe the backend before a user request fails. In NGINX Plus, that looks like this:

upstream app_backend {
    zone app_backend 64k;
    server 10.0.0.11:8080;
    server 10.0.0.12:8080;
}

server {
    listen 80;

    location / {
        proxy_pass http://app_backend;
        health_check;
    }
}

That health_check directive is not available in the standard open source package most people install from their distribution repository. I mention it because people see it in examples and paste it blindly, then wonder why nginx -t fails.

Current practice on open source Nginx is passive checks plus external monitoring. I usually pair that with application health endpoints and infrastructure-level alerts.

Keepalive connections to upstream servers

One of the easiest wins is upstream keepalive.

upstream app_backend {
    zone app_backend 64k;
    server 10.0.0.11:8080;
    server 10.0.0.12:8080;
    keepalive 32;
}

server {
    listen 80;

    location / {
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        proxy_set_header Host $host;
        proxy_pass http://app_backend;
    }
}

keepalive 32 lets each worker keep a pool of idle upstream connections open. That reduces TCP handshake churn and usually improves latency under load.

The two lines people forget are:

proxy_http_version 1.1;
proxy_set_header Connection "";

If you leave the upstream side on HTTP/1.0, you lose persistent connections. If you force Connection: close, you lose them too.

I also keep an eye on the application side. A backend with too-small connection pools or too-few workers can actually get worse when Nginx keeps many idle connections open. Load balancing is never just the proxy layer.

Connection limits, backlog, and practical capacity

Nginx can keep sending traffic happily even when the application is the real bottleneck. If backends have fixed worker counts, I tune upstream behavior with awareness of backend capacity.

At minimum, validate:

nginx -t
ss -tan '( sport = :8080 )'

On the app servers, I also check process counts, worker limits, and accept queues. If a backend can serve 200 concurrent requests and you point a large Nginx farm at it with aggressive keepalive, you can fill its sockets long before Nginx thinks anything is wrong.

This is one reason I do not rely on the proxy alone for resilience. Load balancing spreads requests. It does not create capacity.

Backup servers and maintenance patterns

A backup server is useful when I want overflow or emergency failover without giving that server normal traffic.

upstream app_backend {
    server 10.0.0.11:8080;
    server 10.0.0.12:8080;
    server 10.0.0.13:8080 backup;
}

I have used this for a lower-spec node, a temporary migration target, and an app instance running on a different availability zone with slightly worse latency.

What I do not do is treat backup as a substitute for proper redundancy planning. If the backup server is stale, underpowered, or missing a dependency, you just moved the outage one step later.

Slow start after recovery

Slow start is valuable when a recovered backend should not receive a full traffic share instantly. Historically this has been associated with NGINX Plus and version-specific builds, so I always validate support before using it.

Example syntax, only if your build supports it:

upstream app_backend {
    server 10.0.0.11:8080;
    server 10.0.0.12:8080 slow_start=30s;
}

That gradually increases traffic to the recovered node over thirty seconds. It is useful for JVM apps, cold caches, or anything that starts slowly after restart.

If your package does not support it, do not fake the behavior with odd weight edits during an incident. I would rather leave the config simple and manage reintroduction carefully than build fragile operational rituals.

Upstreams over Unix sockets

If Nginx and the application run on the same host, Unix sockets are often cleaner than loopback TCP.

upstream app_backend {
    server unix:/run/app/app.sock;
}

server {
    listen 80;

    location / {
        proxy_set_header Host $host;
        proxy_pass http://app_backend;
    }
}

I use Unix sockets for local PHP-FPM and some local app services because they avoid TCP overhead and make firewall rules simpler. The trade-off is that they are only local, so this is not a multi-host balancing method.

If you mix socket and TCP upstreams, document it clearly. During troubleshooting, people often forget the socket path even exists. For PHP-FPM specifics, my PHP-FPM and Nginx tuning guide is worth keeping nearby.

Logging which upstream answered the request

If you are not logging the selected upstream, you are making incident response harder than it needs to be.

I usually add a log format like this:

log_format upstreamlog '$remote_addr - $host [$time_local] '
                       '"$request" $status $body_bytes_sent '
                       'rt=$request_time uct=$upstream_connect_time '
                       'uht=$upstream_header_time urt=$upstream_response_time '
                       'ua="$upstream_addr" us="$upstream_status"';

access_log /var/log/nginx/access.log upstreamlog;

Now I can see which backend was hit and whether retries occurred. That is the difference between guessing and knowing.

A quick validation request loop:

for i in $(seq 1 10); do curl -s -o /dev/null http://example.com/; done
sudo tail -n 10 /var/log/nginx/access.log

If the app adds its own hostname to the response, that helps too, but I still want the proxy log to tell the story independently.

Debugging distribution and failures

When I want to see how traffic is spreading, I use the logs instead of intuition.

awk -F'ua="|" us=' '/ua="/ {print $2}' /var/log/nginx/access.log | sort | uniq -c | sort -nr

That shows how many requests each upstream address handled. If one server is missing, I immediately check whether it is marked down, failing health checks, or simply absent from the upstream block.

To catch timeouts and retries:

grep ' us="502\|504' /var/log/nginx/access.log | tail -n 20
sudo grep -E 'upstream timed out|connect\(\) failed|no live upstreams' /var/log/nginx/error.log | tail -n 20

Those patterns show up constantly in real incidents. If you have never seen no live upstreams while connecting to upstream, you have not been on call long enough.

Common mistakes I keep seeing

Forgetting proxy_next_upstream

If retries are not allowed for the failure mode you are seeing, a single bad backend can look far worse than necessary.

Using ip_hash behind large shared IP ranges

This creates uneven distribution very quickly.

Leaving out upstream keepalive tuning

The setup still works, but latency and backend connection churn are worse than they need to be.

No upstream visibility in logs

Then every backend problem turns into speculation.

Assuming passive health checks are proactive

They are not. A user request usually discovers the failure first.

Caching the wrong layer of statefulness

If a backend is fragile because of sticky sessions, hiding that problem with load-balancer tricks usually delays the real fix. Sometimes pairing balancing work with a proxy cache configuration review helps reduce backend pressure enough to make the real issue obvious.

Security and operational considerations

Load balancers are not just traffic splitters. They shape failure domains.

A few things I always consider:

  • Restrict direct backend access with firewalls so clients cannot bypass Nginx.
  • Preserve client information correctly with X-Forwarded-For and related headers.
  • Keep TLS policy at the proxy consistent with the rest of the estate. I review headers, TLS policy, and backend exposure together because proxy changes often happen together.
  • Avoid exposing internal IPs in public error pages.
  • Test reloads with nginx -t before systemctl reload nginx.

On busy sites, I also watch file descriptor limits and worker connections. Upstream tuning does not help much if the proxy itself is constrained.

Final thoughts

Most Nginx upstream load balancing problems are not caused by the balancing algorithm. They come from missing visibility, unrealistic assumptions about backend health, or stateful application behavior that the proxy is expected to paper over.

My default approach is simple: start with round-robin, add zone, add upstream keepalive, log the upstream address and timings, and only then decide whether least_conn, ip_hash, or hash is justified. That keeps the configuration understandable and the failure modes obvious, which is exactly what I want when something breaks at 02:00.

Related Posts

Scroll to Top