HTTP 504 Gateway Timeout: Nginx and Apache Debugging That Finds the Real Bottleneck

By /

A 504 Gateway Timeout means the proxy or gateway did not get a response from the upstream fast enough. In practical terms, a frontend server such as Nginx waited for Apache, PHP-FPM, Gunicorn, Node.js, or some internal API, and the answer took longer than the configured limit.

I treat 504 as a timing problem with a dependency chain behind it. The temptation is to increase proxy_read_timeout and move on. I have done that as a temporary mitigation when production traffic needed breathing room. But I have also seen that shortcut turn a clear failure into a slower failure while the real cause stayed hidden: a bad SQL query, a blocked PHP worker pool, slow object storage, synchronous API calls, or an application server hanging on external DNS.

That is why my first goal with a 504 is not to make it disappear. My first goal is to identify which layer is slow and why.

If you are not sure whether you are dealing with a timeout or a broken upstream, compare this with HTTP 502 Bad Gateway in Nginx. If the backend is outright failing rather than merely slow, the investigation path changes quickly.

504 vs 502: same proxy layer, different failure mode

The two errors get mixed together constantly.

  • 502 Bad Gateway usually means the upstream connection failed or the response was invalid.
  • 504 Gateway Timeout means the upstream did not answer within the timeout window.

That sounds simple, but it matters operationally.

With 502, I look first for dead services, wrong ports, invalid headers, missing sockets, or crashes. With 504, I look for slow paths: long-running requests, blocked workers, database latency, network stalls, overloaded upstreams, or unrealistic timeout values.

Sometimes an unhealthy app causes both. Under light load it may answer incorrectly and give 502. Under heavy load it may hang until Nginx gives 504. I still start with the timeout evidence before editing configuration.

How the timeout chain actually works

In Nginx, several timeout directives can affect upstream behavior. The most relevant ones are:

proxy_connect_timeout 5s;
proxy_send_timeout 60s;
proxy_read_timeout 60s;
keepalive_timeout 65s;

What they mean:

  • proxy_connect_timeout: how long Nginx waits to establish the connection to upstream
  • proxy_send_timeout: how long Nginx waits while sending the request to upstream
  • proxy_read_timeout: how long Nginx waits to receive data from upstream
  • keepalive_timeout: idle keepalive behavior on client connections, often relevant for connection management but not a cure for slow upstream work

For FastCGI and PHP-FPM, the equivalent read timeout is often:

fastcgi_read_timeout 60s;

A 504 most often maps to proxy_read_timeout or fastcgi_read_timeout, but I do not assume. The error log usually says more.

Start with logs and timings, not timeout tuning

The Nginx error log is the first place I look:

sudo tail -n 100 /var/log/nginx/error.log

A common line looks like this:

upstream timed out (110: Connection timed out) while reading response header from upstream

That tells me a lot. Nginx connected, sent the request, then waited too long for the upstream to return headers. That is usually application work, not a TCP reachability failure.

I also inspect the access log because response times matter:

sudo tail -n 50 /var/log/nginx/access.log

Better yet, if your log format includes upstream timing fields, use them. A useful custom format might include:

log_format timed '$remote_addr - $host "$request" $status '
                 'rt=$request_time urt=$upstream_response_time '
                 'uht=$upstream_header_time uct=$upstream_connect_time';

Then you can read lines like:

203.0.113.10 - example.com "GET /reports HTTP/1.1" 504 rt=60.001 urt=60.000 uht=60.000 uct=0.001

That immediately tells me the connection was fast, but the upstream response was slow.

Confirm whether the upstream is reachable but slow

My next step is to bypass Nginx and talk to the app directly.

For an HTTP upstream:

time curl -sS -o /dev/null -w 'status=%{http_code} total=%{time_total}\n' http://127.0.0.1:5000/reports

For a health endpoint:

time curl -sS http://127.0.0.1:5000/health

For PHP-backed pages, I usually reproduce through the normal URL and correlate with PHP-FPM slow logs or application logs rather than trying to bypass FastCGI manually.

If the direct request is also slow, the proxy is doing its job. The bottleneck is behind it.

If the direct request is fast but the proxied request is slow, I look for Nginx-level issues, proxy buffering behavior, network hops, TLS between proxies, or inconsistent path handling.

Identify where the time is going

This is the heart of useful 504 troubleshooting.

Application logs

I check application logs around the request timestamp. For systemd services:

sudo journalctl -u myapp.service --since '10 minutes ago' --no-pager

For Apache:

sudo tail -n 100 /var/log/apache2/error.log

For PHP applications, the app logs may be in the framework log directory.

Database queries

Slow queries are one of the most common root causes behind 504s. I have lost count of how many “Nginx timeout” incidents turned out to be a missing database index.

Signs I look for:

  • one endpoint times out consistently
  • CPU is not maxed, but request latency spikes
  • DB process shows sustained load or lock waits
  • app logs show query durations clustered near the timeout value

If the app exposes query logs, enable or inspect them. Do that carefully in production and keep the exposure minimal.

External API dependencies

Applications that synchronously call third-party APIs can easily trigger 504s. I have seen this with payment providers, geolocation APIs, object storage, and SMTP checks done in request paths that should have been asynchronous.

Measure it directly if possible:

time curl -sS https://api.vendor.example/health

Or inspect app-level timing around outbound calls.

Disk and filesystem pressure

Slow disks can make uploads, report generation, cache fills, and PHP session writes painfully slow.

Useful checks:

iostat -xz 1 3
vmstat 1 5
df -h

A nearly full filesystem can also turn routine work into a timeout incident.

Nginx timeout directives that matter in practice

I prefer to document why I change each one.

proxy_connect_timeout

Relevant when the upstream takes too long to accept a connection.

proxy_connect_timeout 5s;

If the upstream is local and healthy, I keep this fairly low. Raising it rarely fixes the real issue unless the network path is genuinely slow or overloaded.

proxy_read_timeout

This is the most common 504 lever for proxied HTTP applications.

proxy_read_timeout 60s;

If users are hitting 60-second timeouts during report generation and you know the operation legitimately takes 90 seconds, raising it may be an acceptable short-term mitigation. But I still ask why that work is synchronous.

proxy_send_timeout

Less common, but relevant for large uploads or slow upstream reads.

proxy_send_timeout 60s;

fastcgi_read_timeout

Critical for PHP-FPM-backed apps.

location ~ \.php$ {
    include snippets/fastcgi-php.conf;
    fastcgi_pass unix:/run/php/php8.2-fpm.sock;
    fastcgi_read_timeout 120s;
}

If a PHP request takes too long to produce output, this setting often controls when the user sees 504.

PHP-FPM and slow scripts

PHP timeouts are often a stack of multiple limits.

Nginx may wait 60 seconds, but PHP itself may have:

  • max_execution_time
  • PHP-FPM pool pressure
  • slow database access
  • session locking
  • filesystem issues

Check PHP-FPM status and logs:

sudo systemctl status php8.2-fpm --no-pager
sudo journalctl -u php8.2-fpm -n 100 --no-pager

I also like enabling a slow log in PHP-FPM when I need evidence:

request_slowlog_timeout = 10s
slowlog = /var/log/php8.2-fpm.slow.log

Then restart safely:

sudo php-fpm8.2 -t
sudo systemctl restart php8.2-fpm

That gives real stack traces for slow requests, which is far better than only increasing timeouts.

Related deployment details are covered in PHP-FPM with Nginx configuration and tuning.

Python, Node.js, and application worker bottlenecks

For Python and Node.js, 504 usually means the app is up but tied up.

Python / Gunicorn

Things I inspect:

  • worker count too low for load
  • synchronous workers blocked on I/O
  • long DB queries
  • app startup or import issues under load
  • timeout mismatch between Gunicorn and Nginx

Check service logs:

sudo journalctl -u gunicorn -n 100 --no-pager

If Gunicorn has a shorter timeout than Nginx, workers may die first. If it has a much longer timeout, Nginx may give 504 while the app keeps working on a request nobody is waiting for.

Node.js

Node.js services get into 504 territory when the event loop is blocked, the app does heavy CPU work on the main thread, or downstream calls are too slow.

Check process health and logs:

sudo systemctl status mynodeapp --no-pager
sudo journalctl -u mynodeapp -n 100 --no-pager

If one endpoint is slow because it generates PDFs or compresses large files inline, the real fix is often architecture, not proxy tuning.

For broader proxy context around app servers, Nginx reverse proxy with Node.js configuration is useful alongside this guide.

Apache behind Nginx

When Nginx fronts Apache, I still use the same method: determine whether Apache is slow or whether the application behind Apache is slow.

Useful checks:

sudo systemctl status apache2 --no-pager
sudo tail -n 100 /var/log/apache2/error.log
curl -I http://127.0.0.1:8080/

If Apache serves static requests quickly but dynamic PHP pages slowly, the real bottleneck is often PHP or the database. If Apache itself is saturated, look at MPM settings, keepalive behavior, and worker pressure.

Short-term fixes vs root-cause fixes

I do both, but in the right order.

Short-term fix

If a production endpoint is timing out at 60 seconds and the work normally completes at 70 seconds after a recent data growth event, I may increase the relevant timeout to reduce user-facing failures immediately.

Example:

proxy_read_timeout 120s;

Then validate:

sudo nginx -t
sudo systemctl reload nginx
curl -I https://example.com/reports

Root-cause fix

Then I still investigate:

  • optimize the query
  • add indexes
  • move the job to background processing
  • cache expensive responses
  • increase worker concurrency where appropriate
  • fix external dependency latency

A current practice is using timeout increases as a controlled mitigation. A deprecated practice is treating bigger timeouts as the permanent solution to slow architecture.

Detecting redirect loops and client-side confusion

Not every long request is true upstream slowness. I have seen redirect chains waste so much time that they look like timeout bugs.

Check with:

curl -I -L -o /dev/null -sS -w 'redirects=%{num_redirects} total=%{time_total}\n' https://example.com/

If redirects are involved, HTTP 301 vs 302 redirects in Nginx and Apache and Apache mod_rewrite redirects and HTTPS enforcement help clean up the logic.

Reading access-log timing like a performance map

If I have one piece of advice for persistent 504 cases, it is this: add or use timing fields in the access log before you start changing timeouts. Timing data turns a vague complaint into a map.

When uct is tiny, the upstream accepted the connection quickly. When uht or urt grows until it matches the configured timeout almost exactly, the upstream is doing work too slowly or not producing output soon enough. If request_time is much larger than upstream time, I start looking at client upload behavior, buffering, or chained redirects.

Once you log those fields consistently, you can grep for the slowest requests and compare endpoints:

sudo grep ' 504 ' /var/log/nginx/access.log | tail -n 20

I also compare healthy and failing requests for the same route. If a report page normally takes 2 seconds and suddenly climbs to 58, I know I am looking for a regression or dependency issue, not a timeout value that was always too low.

Database locks, queue buildup, and the kind of latency that hides in plain sight

Not all slow requests are CPU-heavy. Some of the worst 504 incidents I have handled happened on machines that looked almost idle from a casual top output. The reason was blocking, not raw compute.

Examples I have run into:

  • a transaction holding row locks too long
  • workers waiting for a depleted DB connection pool
  • a background job filling the same table the web request wants to read
  • synchronous queue publishing or email sending in the request path

That is why I correlate web timing with whatever the app uses internally for dependency metrics. A server can have spare CPU and still time out users if each request spends 55 seconds waiting on a lock or remote service. If raising the timeout suddenly makes the 504 disappear but users still wait far too long, I treat that as evidence that I made the symptom quieter, not that I fixed the system.

Common mistakes I see with 504 incidents

Raising every timeout at once

That makes the system slower to fail and harder to diagnose.

Ignoring access log timings

If you log upstream timings, they usually tell you more than generic CPU graphs.

Blaming Nginx for slow SQL

Nginx is often only the timer that exposed the real problem.

Forgetting PHP-FPM slow logs

Those logs have saved me a lot of time.

Not checking filesystem capacity

A disk at 100% can trigger all kinds of “mysterious” latency.

Security considerations while debugging

During a timeout incident, I avoid quick fixes that weaken the host:

  • do not expose internal upstream ports publicly for convenience
  • do not enable verbose debug pages to the internet
  • do not disable TLS or security headers just to “simplify testing” unless you isolate the test path
  • if you add status endpoints, restrict them to localhost or an admin network

Operational fixes should not create an easier attack surface. HTTP security headers in Nginx and UFW firewall setup on Ubuntu help keep that discipline during firefighting.

My repeatable 504 checklist

1. Reproduce and measure

time curl -I https://example.com/reports

2. Read the Nginx error log

sudo tail -n 100 /var/log/nginx/error.log

3. Check access-log timings

sudo tail -n 50 /var/log/nginx/access.log

4. Test the upstream directly

time curl -sS -o /dev/null -w 'status=%{http_code} total=%{time_total}\n' http://127.0.0.1:5000/reports

5. Inspect app logs and worker state

sudo journalctl -u myapp.service --since '10 minutes ago' --no-pager

6. Check DB, disk, and memory pressure

vmstat 1 5
iostat -xz 1 3
free -m

7. Only then adjust timeouts if justified

sudo nginx -t
sudo systemctl reload nginx

Conclusion

A 504 Gateway Timeout is rarely fixed well by guesswork. It means a proxy waited and the upstream took too long, so the real job is to map the request path and identify where the time disappears. Nginx timing directives matter, especially proxy_read_timeout and fastcgi_read_timeout, but they should be changed with evidence. In my experience, the lasting fixes are usually deeper: query optimization, worker tuning, background jobs, dependency cleanup, or removing synchronous work from request paths. Once you can prove which layer is slow, 504 stops being a vague gateway problem and becomes a concrete performance issue you can actually solve.

Related Posts

Scroll to Top