OpenSSL HollowByte: 11-Byte TLS Payload Causes Server Memory Exhaustion

By /

A denial-of-service vulnerability in OpenSSL, named HollowByte by Okta’s Red Team, enables a remote unauthenticated attacker to permanently exhaust server memory using a crafted 11-byte TLS handshake payload. The flaw was quietly fixed in a multi-branch OpenSSL release on June 9, 2026. No official CVE advisory was published by the OpenSSL project at the time of the fix; public details became widely known only after Okta’s disclosure in July 2026.

Any internet-facing service that terminates TLS using an unpatched version of OpenSSL is potentially vulnerable — including NGINX, Apache, Node.js, Python, Ruby, PHP, MySQL, and PostgreSQL deployments.

Overview

OpenSSL, the ubiquitous open-source cryptography library used across virtually every Linux server environment, contains a flaw in how it handles incoming TLS handshake messages. Before the June 9, 2026 patch, OpenSSL allocated memory for a handshake payload based solely on the size declared by the connecting client — without waiting to verify that the declared payload would actually arrive.

An attacker exploiting this behavior can send a minimal 11-byte TLS record (just the handshake header, with a large bogus length field), forcing the server to allocate up to 131 KB of memory per connection. Because Linux glibc does not immediately return freed heap memory to the operating system, those allocations persist even after the connection closes. Repeating this attack across many sequential connections produces cumulative, persistent memory growth — a condition that can lead to Out-of-Memory (OOM) kills, severe performance degradation, or a complete service outage, all without triggering conventional rate-limiting or connection-count defenses.

Understanding the mechanics of TLS handshake negotiation is important context for grasping why this vulnerability is effective and difficult to block at the application layer.

Vulnerability Details

FieldDetails
Vulnerability NameHollowByte
CVE IDNo official CVE
Vulnerability TypeUncontrolled Memory Allocation / Denial of Service
CWECWE-789: Memory Allocation with Excessive Size Value
Affected ComponentOpenSSL TLS handshake message processing
Attack VectorNetwork
Attack ComplexityLow
Privileges RequiredNone
User InteractionNone
ScopeUnchanged
CVSS ScoreNo official CVSS score published (no formal advisory from OpenSSL)
SeverityHigh (based on impact and exploitability as assessed by public researchers)

Note: The OpenSSL project treated this as a hardening fix rather than a formal security vulnerability at the time of the June 9, 2026 release. No CVE was assigned by the OpenSSL team, and no dedicated security advisory was published. The CVE-2026-1234 identifier appears in third-party security reporting but has not been officially confirmed or scored by the NVD as of July 2026.

Timeline

DateEvent
UnknownVulnerability present in OpenSSL TLS handshake path
June 9, 2026OpenSSL releases 4.0.1, 3.6.3, 3.5.7, 3.4.6, and 3.0.21 — fix included silently with no advisory
July 2026Okta’s Red Team publishes public research naming the bug “HollowByte” and detailing exploitation
July 2026Security news outlets report widely; administrators become aware of the silent fix

Affected Versions

OpenSSL BranchVulnerable VersionsFixed Version
4.0.xAll versions before 4.0.14.0.1
3.6.xAll versions before 3.6.33.6.3
3.5.xAll versions before 3.5.73.5.7
3.4.xAll versions before 3.4.63.4.6
3.0.x (LTS)All versions before 3.0.213.0.21
1.1.1.xEnd-of-Life — no fix expectedNo fix planned
1.0.2.xEnd-of-Life — no fix expectedNo fix planned

OpenSSL 1.1.1 reached end-of-life on September 11, 2023. OpenSSL 1.0.2 reached end-of-life on December 31, 2019. Neither branch will receive security fixes. Systems running these versions should be treated as permanently vulnerable.

Technical Details

Root Cause

During the TLS handshake, OpenSSL receives handshake records that include a 4-byte header containing the message type and the declared length of the payload that follows. In versions prior to the June 9, 2026 fix, OpenSSL allocated the full buffer described by this length field immediately upon reading the header — before any portion of the payload had arrived and without enforcing any validation that the payload would actually follow.

An attacker can send a well-formed 11-byte TLS record (4-byte TLS record header + 4-byte handshake header with a large length value + 3 additional bytes), causing OpenSSL to pre-allocate a buffer of up to 131,072 bytes (128 KB) per connection. The attacker then simply closes the connection without sending payload data.

The resulting heap allocation is freed by OpenSSL when the connection closes. However, the glibc allocator on Linux retains small-to-medium freed heap chunks in the process’s resident memory rather than immediately returning them to the OS. When an attacker repeats this attack across many sequential connections with randomized declared lengths, glibc heap fragmentation ensures that an ever-growing portion of the process’s RSS (Resident Set Size) remains consumed.

Attack Prerequisites

  • No authentication required
  • No prior connection state or session required
  • Works across TLS 1.2 and TLS 1.3 handshake paths
  • Works against any port where OpenSSL is terminating TLS

Exploitation

Okta’s Red Team demonstrated that against a 1 GB RAM server, the attack locked 547 MB of memory before the process was OOM-killed. Against a 16 GB server, 25% of total RAM was consumed. Critically, the attack stayed beneath typical connection-count rate limits because only a small number of sequential connections were required — not a high-volume concurrent flood.

The fixed versions change the buffer allocation strategy so that OpenSSL grows the handshake buffer incrementally as actual bytes arrive from the client, rather than pre-allocating the full declared size upfront.

For administrators who use OpenSSL’s s_client for TLS debugging, note that s_client itself is not a server-side component and is not directly exploitable by this attack.

Real-World Impact

  • Denial of Service: Primary confirmed impact. Server processes can be OOM-killed or rendered severely degraded.
  • Service Disruption: NGINX, Apache, and other TLS-terminating services built on OpenSSL can be taken offline or made unstable.
  • Cascading Failures: In containerized environments with limited memory, OOM kills on TLS workers can trigger cascading service failures across a cluster.
  • Difficult to Detect in Real Time: Memory growth from fragmentation is gradual and may be misattributed to application memory leaks.
  • Remote Code Execution: Not demonstrated or confirmed. The publicly available research does not indicate that RCE is achievable via this vulnerability.
  • Active Exploitation: As of July 2026, active exploitation in the wild has not been confirmed. No exploit code was publicly released by Okta’s Red Team. However, the simplicity of the required payload makes independent exploitation likely achievable by a technically capable attacker.
  • CISA KEV: This vulnerability has not been added to the CISA Known Exploited Vulnerabilities (KEV) Catalog as of July 2026.

Vendor Response

The OpenSSL project released patched versions across all supported branches on June 9, 2026:

  • OpenSSL 4.0.1
  • OpenSSL 3.6.3
  • OpenSSL 3.5.7
  • OpenSSL 3.4.6
  • OpenSSL 3.0.21

The fix changes handshake buffer allocation to grow incrementally as payload bytes are received, eliminating the pre-allocation behavior based on client-declared length.

No dedicated security advisory, no CVE assignment, and no changelog entry specifically calling out the fix were published at the time of release. The OpenSSL project categorized the change internally as a hardening improvement rather than a security vulnerability. Full disclosure only followed Okta’s Red Team research publication in July 2026.

Official release information:

How to Check if You Are Affected

Linux — Check installed OpenSSL version

openssl version -a

Output example for a vulnerable system:

OpenSSL 3.0.14 4 Jun 2024

Output example for a patched system:

OpenSSL 3.0.21 9 Jun 2026

Linux — Check OpenSSL package version via package manager

Debian / Ubuntu:

dpkg -l | grep -i openssl
apt-cache policy openssl libssl3

RHEL / AlmaLinux / Rocky / CentOS:

rpm -q openssl openssl-libs

Linux — Check which OpenSSL library a running process uses

# Find processes linked to a specific OpenSSL shared library
lsof | grep libssl | awk '{print $1, $2}' | sort -u

# Or check against a specific binary, e.g., nginx
ldd /usr/sbin/nginx | grep ssl

Linux — Check for statically linked OpenSSL (e.g., in Go or Rust binaries)

# Go binaries often bundle their own TLS stack (not OpenSSL), but verify
strings /usr/local/bin/yourbinary | grep -E "OpenSSL [0-9]"

Docker

# Check OpenSSL version inside a running container
docker exec -it <container_name> openssl version

# Check base image in Dockerfile and verify it uses updated packages
docker run --rm <image_name> openssl version

Kubernetes

# Check OpenSSL version in a pod
kubectl exec -it <pod-name> -- openssl version

# Check across all pods in a namespace
kubectl get pods -n <namespace> -o name | xargs -I{} kubectl exec {} -- openssl version 2>/dev/null

Mitigation

Official Fix (Permanent Solution)

Upgrade OpenSSL to a fixed version immediately:

BranchUpgrade Target
4.0.x4.0.1 or later
3.6.x3.6.3 or later
3.5.x3.5.7 or later
3.4.x3.4.6 or later
3.0.x (LTS)3.0.21 or later

On Debian/Ubuntu:

apt update && apt upgrade openssl libssl3

On RHEL/AlmaLinux/Rocky:

dnf update openssl openssl-libs

After upgrading system packages, restart all services that link against OpenSSL so they load the patched library:

# Restart nginx
systemctl restart nginx

# Restart apache
systemctl restart apache2   # Debian/Ubuntu
systemctl restart httpd     # RHEL/CentOS

# For applications not managed by systemd, restart them manually

Temporary Workaround

No fully effective workaround exists without patching. However, the following measures can reduce exposure:

  • Rate-limit new TLS connections per source IP at the firewall or load balancer level to make sustained attacks slower. This does not prevent the attack but increases the time required.
  • Restrict TLS listener exposure to trusted IP ranges where possible (applies to internal-only services).
  • Configure kernel OOM behavior to ensure OOM-killed services are restarted automatically (e.g., via systemd Restart=always + RestartSec=5).

Monitoring Recommendation

Monitor server-side TLS worker process memory (RSS) for unexpected growth patterns. A sudden, sustained increase in resident memory on NGINX workers, Apache prefork/worker processes, or Node.js processes — without a corresponding increase in legitimate request volume — may indicate exploitation attempts.

Proper TLS cipher suite hardening and server hardening practices remain important complements to patching.

End-of-Life Versions

For systems running OpenSSL 1.1.1 or 1.0.2, no fix will be provided. These systems should be migrated to a supported OpenSSL branch as a priority.

Detection

No official Indicators of Compromise (IOCs), YARA rules, or Sigma rules have been published for HollowByte as of July 2026.

Log Signals

HollowByte attack traffic does not generate distinctive application-layer errors by default. The clearest signal is memory growth in TLS-serving processes without a corresponding increase in legitimate traffic.

NGINX — monitor worker process memory:

# Watch nginx worker memory growth in real time
watch -n 5 'ps -o pid,rss,vsz,comm -p $(pgrep -d, nginx)'

General — watch for OOM events in kernel logs:

dmesg | grep -i "oom\|out of memory\|killed process"
journalctl -k | grep -i "oom\|killed process"

Connection rate monitoring:

Look for source IPs establishing many short-lived TLS connections. In NGINX access logs, TLS-layer rejections may appear with status code 400 or abrupt connection resets before request completion:

# Count 400 errors per source IP from nginx access logs
awk '$9 == 400 {print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -20

IDS/IPS

As of July 2026, no public Snort or Suricata rules targeting HollowByte have been released. The attack uses standard TLS record formatting, making signature-based detection at the network layer difficult without deep packet inspection and stateful connection tracking.

Practical Recommendations

  1. Identify all OpenSSL versions in use. Survey all servers, containers, and virtual machines. Do not assume OS-level package updates are sufficient — applications that statically link OpenSSL (some Go/Rust binaries do not, but some compiled C/C++ applications do) require independent updates.
  2. Prioritize internet-facing TLS endpoints. Services that accept inbound TLS from untrusted sources (public web servers, API gateways, load balancers) carry the highest risk.
  3. Upgrade OpenSSL packages on all supported branches to the patched versions listed above. On RHEL-family systems, backported security patches may be available under the original version number — verify using rpm -q --changelog openssl.
  4. Restart services after upgrading to ensure patched libraries are loaded. Verify with lsof -p <pid> | grep ssl.
  5. Audit statically linked applications. Container images and pre-compiled binaries may bundle their own OpenSSL copy. Rebuild container images from updated base images.
  6. Migrate off End-of-Life branches. Systems running OpenSSL 1.1.1 or 1.0.2 have no patch path for this vulnerability. Plan migration to a supported branch as a priority.
  7. Configure service restart policies. Ensure that OOM-killed TLS services restart automatically via systemd to limit availability impact if an attack occurs before patching.
  8. Monitor memory usage trends on TLS-serving processes as an indicator of ongoing exploitation attempts.
  9. Review OpenSSL command-line tools for version verification and certificate chain validation after upgrades.

Sources

Related Posts

Scroll to Top