p95·p99 Latency Difference Analysis

p95 means that 95% of all requests are processed within this time, and the bottom 5% of users receive slower responses than this. It represents the typical worst-case experience for the majority of users.

As you might have guessed, p99 means that 99% are processed within this time, and only the bottom 1% receive slower responses than this. It represents long-tail latency, extremely slow responses, and stability.

p95. p99 Difference Analysis, Meaning of the Gap #

One might simply wonder what the difference is between 95 and 99. Since they are just numerically displayed values. Let's explore the detailed differences and what they signify.

Of course, it might vary depending on the domain type of the server system, but I will explain it based on universal standards.

The latency gap between p95 and p99 indicates the system's predictability and stability.

When the gap is small (stable)

If p95 = 50ms and p99 = 60ms, there is almost no difference.
Since it's about 10ms, this means the system operates very consistently even under load or exceptional circumstances.

When the gap is large

p95 = 50ms, p99 = 800ms
While 95% experience comfort, the remaining tiny fraction of users are experiencing very severe delays. In a large-scale service with high traffic, 1% is not a small number. It means 10,000 out of 1 million requests are slow responses.

Main Causes of Spiking p99 Latency #

While most requests are processed normally, the significant delay for only a tiny fraction of requests has the following backend/infrastructure causes:

Garbage Collection STW: This is the latency for requests that arrive at the moment an application's execution is paused due to STW (Stop-The-World) for memory cleanup in languages like Java, Go, and Node.js.
Cache Miss: While 95% of requests are cache hits and are processed by fast caches like Redis or Memcached, a long latency difference occurs when the remaining requests are not in the cache and have to go all the way to the slower main database.
DB Lock: This occurs when transactions on a specific table or row pile up, causing a lock, and some queries enter a waiting state.
Network Packet Loss and Retransmission: When TCP packets are lost due to temporary network congestion and retransmission occurs, the latency spikes significantly.
Cold Start: This is the initial delay incurred when a newly created instance processes its first request in a serverless environment or during container scale-out.

Management Methods #

Average latency is not suitable as a monitoring metric because it hides extreme delay outliers.

SLO (Service Level Objectives) should be set based on p99. If aiming for a high-availability system, targets should be set based on 99 or 99.9 metrics, not p95.

Distributed tracing is also beneficial; through tools like Jaeger and Zipkin, the 1% of slow requests corresponding to p99 should be traced to pinpoint exactly which part of the microservice architecture (DB, external API, specific service) they are being delayed in.