The Danger of Retry Thundering Herd
Thundering Herd, also known as a retry storm, refers to a phenomenon where, when a system experiences a temporary failure, the retry logic of clients attempting to recover from it simultaneously overlaps, leading to the complete shutdown of the system.
- Thundering Herd: When a cache expires or a server restarts, numerous clients simultaneously flood the database or server with requests. (Stampede)
- Retry Storm: When Service A calls Service B and a timeout occurs, leading to a retry, the problem arises when 10,000 users simultaneously experience failure, and 10,000 clients attempt retries at the exact same time, with no more than a 0.1-second difference.
Problem #
The problem is that it doesn't just increase traffic; it escalates into a self-DDoS attack where the system attacks itself.
Traffic amplifies exponentially. Let's assume a server that originally processed 1,000 TPS (transactions per second) paused for 1 second.
If clients are configured to retry 3 times, then after the next second, the server will instantly receive over 4,000 requests: 1,000 original requests, plus 1,000 delayed requests * 3 retries.
This can lead to a cascading failure. Even if Service B barely recovers, the retry requests from Service A can exhaust Service A's threads, eventually bringing down the user gateway that was calling Service A.
It can also lead to an unrecoverable state, similar to a deadlock. Even if you scale out or restart the server because it can't handle the traffic, the moment the server comes online, the waiting retry traffic can flood it and kill it again.
Example #
In a normal situation, let's say 500 users per second are issuing coupons on an event page.
The server can comfortably handle 1,000 requests per second, so it's peaceful.
Due to a retry storm, let's say the DB server's network switch briefly faltered, causing the DB connection to be lost for about 3 seconds.
During these 3 seconds, 1,500 users experience failures in their apps, and let's assume the app's built-in library is naively configured to retry 3 times upon failure.
What if, the moment the DB network normalizes after 3 seconds, 4,500 connection requests (3 retries each from 1,500 apps) hit the DB simultaneously within 0.001 seconds?
The DB would exceed its maximum connections and crash. Even if a server developer quickly restarts the DB, the system could completely crash due to users refreshing or similar situations.
Solution #
The core of this problem is to break concurrency and scatter retry intervals.
To achieve this, we need to combine Exponential Backoff and Jitter (random delay jitter) algorithms.
First, let's check for concurrency spikes in access logs through terminal analysis.
# Aggregate requests per second in Nginx access.log for a specific time range to demonstrate a spike phenomenon
$ awk '{print $4}' /var/log/nginx/access.log | cut -d: -f2,3,4 | sort | uniq -c | sort -n | tail -n 5
120 22:45:01
125 22:45:02
0 22:45:03 <-- Network Blip occurred (0 requests processed)
0 22:45:04
8540 22:45:05 <-- Retry storm hit immediately after recovery, traffic surged 70-fold
We can see that traffic surged from 120 requests to 8540 at 22:45:05.
This is how it can be detected.
And let's add jitter defense code at the application level.
It is essential to mix in random time jitter to prevent traffic from hitting simultaneously.
import io.github.resilience4j.retry.Retry
import io.github.resilience4j.retry.RetryConfig
import io.github.resilience4j.core.IntervalFunction
import org.slf4j.LoggerFactory
import org.springframework.context.annotation.Bean
import org.springframework.context.annotation.Configuration
import java.time.Duration
import java.util.concurrent.TimeoutException
@Configuration
class ResilientRetryConfig {
@Bean
fun customRetryConfig(): RetryConfig {
// Configure Exponential Backoff + Jitter
// Initial wait time 1 second, increases by 2x on retry, mixed with a random error (Jitter) of 0.5 (50%)
// Example: After 1st failure, wait (1 second ± 0.5 seconds); after 2nd failure, wait (2 seconds ± 1 second)
val intervalFunction = IntervalFunction.ofExponentialRandomBackoff(
Duration.ofSeconds(1), // Initial interval
2.0, // Multiplier
0.5 // Jitter randomization factor
)
return RetryConfig.custom<Any>()
.maxAttempts(3) // Retry a maximum of 3 times
.intervalFunction(intervalFunction) // Apply the distributed wait queue created above
.retryExceptions(TimeoutException::class.java, CustomNetworkException::class.java)
.build()
}
@Bean
fun paymentRetry(retryConfig: RetryConfig): Retry {
return Retry.of("paymentServiceRetry", retryConfig)
}
}
// Example of usage in business logic
class PaymentService(private val paymentRetry: Retry) {
private val log = LoggerFactory.getLogger(this::class.java)
fun processPaymentWithRetry(orderId: String): String {
// Execute with a Retry decorator applied.
// Now, even if 10,000 people fail simultaneously, retry timings will be randomly scattered between 0.5 and 1.5 seconds.
val decoratedSupplier = Retry.decorateSupplier(paymentRetry) {
callExternalPaymentApi(orderId)
}
return try {
decoratedSupplier.get()
} catch (e: Exception) {
log.error("[Payment Failed] Final failure due to exceeding maximum retry attempts - orderId: $orderId", e)
"Payment system delayed. Please try again later."
}
}
private fun callExternalPaymentApi(orderId: String): String {
// Actual external communication logic (timeout may occur)
return "SUCCESS"
}
}
By mixing in jitter like this, the 4,500 retries won't hit within 0.001 seconds; instead, the traffic will be spread out over several seconds, allowing the recovering server to handle the distributed load.
Beyond this, it's also good to decide whether retries are absolutely necessary, and to establish a retry policy by determining whether to retry immediately or use exponential backoff (with longer intervals).
While preventing retry storms is important, there should actually be a defense mechanism that integrates the Circuit Breaker pattern to quickly return errors and prevent even sending retries to an already dead server.
Because that seems even more crucial.