Connection Pool Exhaustion Symptoms

A Connection Pool is a caching technique that pre-creates a certain number of physical TCP connections between an application and a database, lending them out and reclaiming them as client requests come in.

In Spring Boot, HikariCP is primarily used as the default implementation.

The process of a database establishing a connection, such as the TCP 3-way handshake, consumes significant computing resources and time. Pooling techniques eliminate this overhead, thereby increasing system throughput.

Connection pool exhaustion occurs when a new database access request arrives while all connections up to the maximum configured pool size are in use (active).

New requests will wait in a blocking state in the queue for the configured connection timeout period to be allocated a connection.

Problem Definition #

Connection pool exhaustion doesn't merely result in database query failures; it has the characteristic of escalating into a complete application server outage.

WAS Thread Pool Exhaustion: Worker threads of a Web Application Server (WAS) like Tomcat enter a waiting state to acquire a DB connection.

If connection release is delayed, even available WAS threads become exhausted, leading to 503 Service Unavailable errors even for simple health checks or static resource requests that don't require DB access.

Latency also skyrockets because requests entering the queue have connection acquisition wait times added to their query execution times, resulting in overall API response delays.

There's also a deadlock risk. If there's logic within a single transaction that requires multiple connections, and the pool size is small, threads might enter a permanent deadlock state, holding onto some connections while waiting for others.

Example #

Let's say the application's max pool size is set to 10.

The average hold time for each query is 10ms. This means one connection can process 100 requests per second, allowing the entire pool to handle 1000 requests per second (100 x 10).

A newly deployed feature introduced a SELECT query missing an index. Because it's a table full scan, the query execution time increases to 5 seconds. If 10 requests come in simultaneously, they will occupy all 10 connections.

These connections are then not released for 5 seconds, causing hundreds of additional normal requests to be unable to acquire connections and pile up in the queue.

Once the default connection timeout (e.g., 30s) elapses, SQLTransientConnectionExecptions occur concurrently, leading to a system outage.

Solution #

When an outage occurs, the connection waiting state should be identified through application thread dumps or logs, and connection hold times should be minimized at the code level.

Check if threads are waiting in the connection pool using jstack based on the WAS's PID.

# 1. Java 프로세스의 스레드 덤프를 추출하여 HikariCP 관련 대기(WAITING) 상태의 스레드 수를 집계
$ jstack $(pgrep -f "spring-boot-app.jar") | grep "HikariPool" | grep "WAITING" | wc -l

# 2. 애플리케이션 로그에서 커넥션 획득 실패(Timeout) 에러 발생 확인
$ grep -A 2 "Connection is not available" /var/log/app/application.log | tail -n 5

# jstack 결과 (대기 중인 스레드 수)
154

# 로그 grep 결과
2026-03-09 22:45:10.123 ERROR [http-nio-8080-exec-15] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Connection is not available, request timed out after 30000ms.
2026-03-09 22:45:10.123 ERROR [http-nio-8080-exec-15] org.hibernate.exception.JDBCConnectionException : Unable to acquire JDBC Connection

Result Analysis: We confirmed with data that 154 web request threads are blocked waiting for connections, and a 30-second TIMEOUT occurred, indicating complete connection pool exhaustion.

Let's implement defensive Kotlin code by separating transaction boundaries.

The biggest cause of connection pool exhaustion is anti-patterns where external API calls occur within transaction blocks, unnecessarily holding DB connections for too long. This should be separated using a Facade pattern.

import org.springframework.stereotype.Service
import org.springframework.transaction.annotation.Transactional
import org.slf4j.LoggerFactory

// [안티 패턴]: 외부 API 호출 시간 동안 DB 커넥션을 불필요하게 점유함
@Service
class BadOrderService(
    private val orderRepository: OrderRepository,
    private val paymentClient: PaymentClient
) {
    @Transactional
    fun processOrderBad(orderId: Long) {
        // 1. 커넥션 획득 및 로직 수행
        val order = orderRepository.findById(orderId) 
        
        // 2. 문제 발생 구간: 외부 네트워크 통신 (예: 3초 소요)
        // 이 3초 동안 DB 커넥션은 아무 작업도 하지 않으면서 반납되지 않고 유지됨
        paymentClient.pay(order.amount) 
        
        // 3. 상태 변경 및 트랜잭션 커밋, 커넥션 반납
        order.complete() 
    }
}


// [해결 방안]: 트랜잭션 경계를 최소화하여 DB 커넥션 점유 시간을 단축
@Service
class GoodOrderFacade(
    private val orderQueryService: OrderQueryService,
    private val paymentClient: PaymentClient,
    private val orderCommandService: OrderCommandService
) {
    private val log = LoggerFactory.getLogger(this::class.java)

    // @Transactional 어노테이션을 상위 메서드에서 제거함
    fun processOrderGood(orderId: Long) {
        // 1. 필요한 데이터만 짧은 트랜잭션으로 조회 (커넥션 즉시 반납)
        val orderAmount = orderQueryService.getOrderAmount(orderId)
        
        // 2. 외부 API 호출 (DB 커넥션을 점유하지 않은 상태에서 실행됨)
        // 외부 API가 타임아웃(3초)이 발생하더라도 DB 커넥션 풀에는 영향을 주지 않음
        try {
            paymentClient.pay(orderAmount)
        } catch (e: Exception) {
            log.error("Payment failed", e)
            throw RuntimeException("결제 처리 중 오류 발생")
        }
        
        // 3. 상태 변경을 위한 짧은 트랜잭션 실행 (커넥션 잠깐 점유 후 즉시 반납)
        orderCommandService.completeOrder(orderId)
    }
}

@Service
class OrderQueryService(private val orderRepository: OrderRepository) {
    @Transactional(readOnly = true)
    fun getOrderAmount(orderId: Long): Long {
        return orderRepository.findById(orderId).amount
    }
}

@Service
class OrderCommandService(private val orderRepository: OrderRepository) {
    @Transactional
    fun completeOrder(orderId: Long) {
        orderRepository.findById(orderId).complete()
    }
}

By applying architectural refactoring to move network I/O operations and external API calls outside of @Transactional blocks, as shown in the solution above, we can structurally prevent external system failures from propagating into internal DB connection pool exhaustion.

It would be beneficial to consider whether the API absolutely needs to be bound within the same transaction, if it must operate atomically, and how tightly coupled the two functionalities should be, before applying this approach.