Thread Pool Size Calculation Principles

Does a higher number of threads always lead to higher throughput? In reality, no. Too many threads can actually cause performance to plummet due to context switching overhead and memory exhaustion.

To determine the optimal thread pool size, let's explore the core principles that must be understood, step by step.

Analyzing Workload Characteristics: CPU-Bound vs. I/O-Bound #

The absolute criterion for determining thread pool size is to understand what kind of tasks the application primarily performs.

CPU-bound (computation-intensive tasks): Tasks that continuously use the CPU, such as complex mathematical operations, encryption/decryption, and image processing.
I/O-bound (I/O-intensive tasks): Tasks that involve long waiting times for responses from network disks, such as database queries, API calls, and file I/O. Most web applications fall into this category.

For CPU-bound tasks, having more threads than CPU cores won't allow the CPU to process them simultaneously, so increasing threads only wastes context switching costs.

For I/O-bound tasks, when a thread enters a blocked or waiting state while waiting for a database response, the CPU becomes idle.

In such cases, the thread pool size should be generous enough to allow other threads to use the CPU, maximizing CPU utilization.

Brian Goetz's Formula #

Brian Goetz, a master of Java concurrent programming, established a famous formula for calculating the optimal number of threads in his book 'Java Concurrency in Practice'.

This formula serves as a fundamental criterion for system design, regardless of the language.

N_{threads} = N_{cpu} \times U_{cpu} \times \left(1 + \frac{W}{C}\right)

$N_{threads}$ : Optimal number of threads
$N_{cpu}$ : Number of available CPU cores
$U_{cpu}$ : Target CPU utilization ( $0 \le U_{cpu} \le 1$ )
$W$ : Wait time (time spent waiting for I/O operations, etc.)
$C$ : Compute time (time spent actually using the CPU)

Just by looking at it, you can see that the higher the compute time or the lower the wait time, the smaller the optimal number of threads.

For CPU-bound tasks, the wait time W approaches 0, so $\frac{W}{C} \approx 0$ .

Therefore, the optimal number of threads becomes approximately $N_{cpu} + 1$ . The reason for adding +1 is to allow a spare thread to occupy the CPU during page faults or temporary interruptions.

For I/O-bound tasks, the wait time W is much longer than the compute time C. If the time spent waiting for the database is 10 times the actual compute time ( $\frac{W}{C} = 10$ ), then with 4 CPU cores, the calculation shows that a minimum of 4 x 1 x (1 + 10) = 44 threads are needed.

Realistic Constraints #

In addition to theoretical formulas, the following factors determine the limits of a thread pool in a real environment:

Memory Limits
- Each thread is allocated its own independent call stack memory upon creation. While it varies by OS and language, it's typically around 1MB per thread. So, if you set a thread pool of 1000 threads, 1GB of memory is consumed just for thread maintenance, meaning available memory capacity must be considered.
DB Connection Pool
- Even if you increase the web server thread pool indefinitely, if the DB connection pool size is 50, a bottleneck will occur when requests exceed 50. This is because only 50 threads can communicate with the DB. Not only for DBs, but also for external API integrations, there can be limitations on the number of connections provided. If 200 requests come in, only 50 are processed, and the remaining 150 are waiting. Generally, the thread pool size should be set to be similar to or slightly larger than the connection pool size for optimal efficiency.
Critical Sections and Locks
- No matter how many threads there are, if there is severe lock contention for shared resources, only one thread will end up executing sequentially, leading to performance degradation.

Derivation Process #

First, set initial values. Analyze the application's characteristics to establish a baseline. For example, an I/O-intensive API server might start with 10-20 times the number of cores.

After that, utilize profiling tools like APM tools (Datadog, New Relic, Scouter, etc.) to measure the CPU processing time (C) and I/O wait time (W) ratios for individual requests under actual traffic.

Use load testing tools such as k6, Locust, nGrinder, etc., to gradually increase the load and monitor metrics like CPU utilization, wait time, memory usage, and response time.

If the CPU is idle but TPS isn't increasing -> Increase the thread pool (connection pool) size.
If the CPU hits 100% and response times skyrocket -> Either there are too many threads causing context switching bottlenecks, or the logic itself needs improvement.