estudy 한국어/English

Handling Resource Constraints of External Integration Systems and Surging Traffic

4,286 단어·9 분·원문(.md)

We designed a system with the following structure:

External Platform - In-house Loan Pre-inquiry Server - Queue - In-house Loan Pre-inquiry Consumer - (External Agency Communication)

In this structure, when an external platform sends a request to our in-house loan pre-inquiry service, it immediately receives a response. Then, the consumer server calculates interest rates and loan limits, including customer information keys, and sends a response back. The external platform then displays the limits calculated by our system.

This also involves communication with external agencies, such as NICE/KCB (credit score evaluation agencies), and our internal logic for the AI model for interest rate calculation + our core banking system (for loan application data creation).

Now, if traffic becomes heavy, there could be bottlenecks in various services:

  1. The in-house loan pre-inquiry API server. This was manageable due to its asynchronous response processing, allowing for quick responses.

Specifically, we observed about 500,000 requests over 2 hours, averaging 60-70 TPS, with peaks jumping to 200 TPS. While it might not seem like much, each call is heavy, making even this level of traffic quite burdensome.

Since the in-house loan pre-inquiry server processes requests asynchronously and provides responses to the platform, two Spring Boot server pods were sufficient to handle this traffic.

Secondly, the queue could be a bottleneck. If a large volume of traffic is received and put into the queue, many messages will accumulate in it.

The in-house loan pre-inquiry consumer processes this data within 1-2 seconds on average. A typical Spring Boot service with 200 threads would handle this smoothly, but here's where a problem arises.

Constraints #

This is due to the NICE system, which only supported up to 40 dedicated sessions.

Since the NICE system responds in about 0.5ms, let's assume it can somehow handle up to 80 TPS (40 sessions x 2). However, what if traffic spikes during peak times? Response times would gradually increase to 1s, 2s, reducing the throughput of the pre-inquiry platform. More requests would also burden the NICE system, consuming bandwidth, and since it was 1 Mbps, the limited bandwidth caused delays.

In any case, this system becomes a problem.

So, how can we solve this? The first method that comes to mind is applying backpressure or a circuit breaker to the NICE system.

What if we just receive an empty response for now and process it later? That would temporarily remove the 80-session constraint, and we could retry from the DLT later? Structurally, it seems correct. But it won't work.

This is because of a domain constraint: NICE provides the customer's credit score.

Therefore, since interest rates and loan limits are calculated based on this credit score, NICE calls must always be performed atomically together.

In this case, there's no solution. It's physically impossible. It's physically impossible to handle a large volume of traffic.

Because it's coupled with the NICE system. Furthermore, NICE is not only used for credit scores for limit calculation but also for functions that require even stronger consistency and are more critical than loan pre-inquiry, such as loan registration to synchronize loan statuses generated by our system, and change calls related to repayment/delinquency/debt status changes. Therefore, a failure in NICE is even more critical.

Choice #

How can we deal with this? Increasing the number of sessions and bandwidth would be the easiest. Didn't they say money solves everything?

As we launched a new service, we naturally needed more dedicated line bandwidth and sessions for NICE, which were previously underutilized, so they had to be increased.

However, we could only increase it to 80. We also raised the bandwidth to 4 Mbps, but pre-inquiry traffic didn't consume that much bandwidth, except for batches running in the early morning. At most, about 2 Mbps?

It would be good to increase it to 120, but let's assume we have to handle the traffic with just 80 sessions.

First, let's look at the pre-loan processing consumer server. If this server processes the queue but directly sends requests to NICE, it will place a heavy load on the NICE server, which only has 80 sessions. Since other services besides the pre-inquiry service also hit NICE, failures could propagate.

Therefore, flow control is needed, so we deployed multiple nice-proxy servers to handle traffic for the NICE system.

Communication with NICE is based on TCP socket specialized communication. When our service sends a credit score call to the NICE proxy, the proxy first receives it, then sends it to NICE. After receiving the data, it writes it to the proxy buffer. Then, it sends a response back to the service that initially sent the request. The receiving service server will then do something with that data.

So, reason 1 for having a proxy: traffic control. Anyway, NICE has 80 sessions. It's natural for more than 80 requests to come in.

Internal services can send thousands of such requests. However, without sessions, these requests would fail.

Therefore, we decided to use Netty for the nice-proxy. The reason is that a Netty-based proxy server can maintain thousands of incoming connections with a small number of threads, even if there are long delays, while efficiently managing only the outgoing pipeline to NICE.

This allows for responses even if delayed, rather than immediate failures. Also, by directly handling ByteBuf through the ChannelHandler pipeline, frame decoding, message parsing, and timeout handling can be performed with great precision.

Furthermore, in Netty, if a surge in requests is anticipated, connections can be pre-established to reduce 3-way handshake costs and efficiently cycle through the 80 channels.

class NiceChannelInitializer : ChannelInitializer<SocketChannel>() {
    override fun initChannel(ch: SocketChannel) {
        val pipeline = ch.pipeline()

        // 1. 타임아웃 설정: 10초 동안 응답(read)이 없으면 ReadTimeoutException 발생
        pipeline.addLast("timeoutHandler", ReadTimeoutHandler(10))

        // 2. 전문 디코딩: 금융 전문이 고정 길이(예: 200byte)라면 FixedLengthFrameDecoder 사용
        // 만약 가변 길이라면 LengthFieldBasedFrameDecoder를 사용합니다.
        pipeline.addLast("decoder", FixedLengthFrameDecoder(200))

        // 3. 비즈니스 로직 핸들러: 응답 전문을 파싱하여 결과를 큐나 콜백으로 전달
        pipeline.addLast("clientHandler", NiceClientHandler())
    }
}

Pre-establishing sessions

class NiceConnectionPool(
    private val host: String,
    private val port: Int,
    private val poolSize: Int = 80
) {
    // 가용한 채널을 담아두는 큐 (Backpressure 역할 병행)
    private val pool = ArrayBlockingQueue<Channel>(poolSize)
    private val bootstrap = Bootstrap()

    init {
        val group = NioEventLoopGroup(4) // 적은 수의 쓰레드로 80개 관리
        bootstrap.group(group)
            .channel(NioSocketChannel::class.java)
            .option(ChannelOption.SO_KEEPALIVE, true)
            .option(ChannelOption.CONNECT_TIMEOUT_MILLIS, 3000)
            .handler(NiceChannelInitializer())
    }

    // 서버 시작 시 호출하여 80개 세션을 미리 맺음
    fun prewarm() {
        val futures = (1..poolSize).map {
            bootstrap.connect(host, port).addListener { future: ChannelFuture ->
                if (future.isSuccess) {
                    pool.offer(future.channel())
                    println("Session connected: ${future.channel().id()}")
                } else {
                    // SRE 관점: 연결 실패 시 재시도 로직이나 알림 필수
                    println("Failed to connect: ${future.cause().message}")
                }
            }
        }
        // 모든 연결이 완료될 때까지 대기하거나 로깅
    }

    // 세션 빌려오기
    fun acquire(): Channel? = pool.poll(5, TimeUnit.SECONDS)

    // 세션 반납하기
    fun release(channel: Channel) {
        if (channel.isActive) {
            pool.offer(channel)
        } else {
            // 끊어진 채널이면 새로 연결해서 채워넣는 로직 필요
            reconnect(channel)
        }
    }
}

Pre-establishing sessions can save approximately 220ms to 400ms in total, including 3-way handshakes (20ms ~ 100ms) and TLS/SSL handshakes and encryption authentication (200ms ~ 300ms).

When maintaining such connections, NICE's firewall or L4 switch might detect idle states where sessions are connected but no actual data is flowing, and forcibly disconnect them, even if TCP keepalive is enabled.

To prevent such ghost sessions, we implemented a heartbeat logic in the Netty pipeline that periodically sends dummy messages to maintain sessions. Of course, we also increased the TCP keepalive settings themselves.

Regarding duplicate users, there's no issue as the user itself is used as a key in our system and NICE. It operates idempotently. Furthermore, credit score data itself doesn't change much, and since it's queried multiple times in many subsequent processes like loan review and CB inquiry before final approval, the accuracy of the credit score during pre-inquiry is not critically important.

We have established a foundation to handle NICE requests even if they slow down across multiple services.

However, if many requests still come in, it will place a greater load on NICE, and both the consumer server and NICE will be burdened.

This is where a Circuit Breaker pattern is needed.

Circuit Breaker #

Considering the 80-session and SQS consumer structure, we can open the circuit based on the following criteria:

Latency-based, Error Rate-based, Saturation-based

Latency-based #

This seems like the first thing to react to. In an 80-session environment, if responses slow down, all sessions become occupied, effectively halting the system.

  • Condition: Set to when more than 50% of the last 100 requests take longer than 5 seconds.
  • Reason: If requests taking longer than 5 seconds exceed half, the throughput arithmetically drops to 16 TPS (because 80/5). If the incoming traffic is 200 TPS, the queue will explode instantly, so it's better to block it in advance and respond to the user to try again later, or give up.

Error Rate-based #

Detects system failures or network errors on the NICE side.

  • Condition: When the error rate (5xx or specialized communication errors) among the last 100 requests is 20-30% or higher.
  • Reason: In financial specialized communication, an error rate of over 20% should be considered a severe failure of the counterparty's system, not just a temporary network outage. Continuously sending requests in this situation is a meaningless waste of resources and could hinder the recovery of the counterparty's system.

Personally, I think 20% is a bit high.

Saturation-based #

This isn't applied to Netty, but to the pre-inquiry service.

While not present in typical circuit breaker libraries like Resilience4j, in this SQS structure, SQS Lag and session acquisition wait time should be used as conditions.

When the SQS ApproximateAgeOfOldestMessage exceeds 60 seconds, it means that messages that entered the queue 60 seconds ago are now being processed. At this point, the user or upstream platform might have already timed out. In this situation, the circuit should be opened to immediately reject new incoming requests and give the consumer time to quickly process the invalid messages accumulated in the queue.

Decision Window #

To distinguish between temporary delays and failures, a sliding time window approach is used.

As a configuration example, the window size is set to about 30 seconds. In a peak traffic situation of 200 TPS, approximately 6,000 requests would come in over 30 seconds.

This is enough time to meet the minimum request count condition of 100. If it's too short, it reacts sensitively to temporary network spikes, and if it's too long, failure response is delayed.

Monitoring #

Monitoring should be performed, not just limited to the circuit opening.

Metrics such as state (Closed, Open, Half-open) and failure_rate provided by libraries like Resilience4j should be collected by Prometheus and visualized in Grafana.

We configured Slack alerts to notify on-call engineers immediately when the circuit state changes from closed to open. At this time, the response speed metrics of the NICE system and our consumer SQS Lag metrics should be compared together to determine if the cause is internal or external.

Manual Control #

Since automated systems cannot be perfect, means for operational intervention must be prepared.

If, even after a NICE-side failure is resolved, the circuit breaker or error rate calculation method delays its return to the closed state, we can provide capabilities to forcibly change the circuit state to closed or temporarily disable the circuit function via Spring Actuator or management API endpoints.

This is because, based on business judgment, even 1% of applicants in the financial domain are valuable. Therefore, upon confirming failure recovery, it might be necessary for engineers to directly verify the status and normalize it, rather than waiting for the system's automatic half-open recovery.

Since it's a pure Kotlin system based on Netty, it can be controlled as follows:

import io.github.resilience4j.circuitbreaker.CircuitBreaker
import io.github.resilience4j.circuitbreaker.CircuitBreakerConfig
import io.github.resilience4j.circuitbreaker.CircuitBreakerRegistry
import java.time.Duration

object NiceCircuitManager {
    // 1. 서킷 브레이커 설정
    private val config = CircuitBreakerConfig.custom()
        .failureRateThreshold(20.0) // 에러율 20% 이상 시 오픈
        .waitDurationInOpenState(Duration.ofSeconds(30)) // 오픈 후 30초 대기
        .slowCallDurationThreshold(Duration.ofSeconds(5)) // 5초 이상이면 느린 호출
        .slowCallRateThreshold(20.0) // 느린 호출 20% 이상 시 오픈
        .minimumNumberOfCalls(100) // 최소 100개 요청부터 판단 시작
        .slidingWindowSize(100) // 윈도우 사이즈 100개
        .build()

    // 2. 레지스트리 생성 및 인스턴스 등록
    val registry: CircuitBreakerRegistry = CircuitBreakerRegistry.of(config)
    val niceCircuitBreaker: CircuitBreaker = registry.circuitBreaker("niceProxy")
}

And since Micrometer is added and there's no Actuator, it's implemented directly.

Directly bind Resilience4j's metrics.

import io.github.resilience4j.micrometer.tagged.TaggedCircuitBreakerMetrics
import io.micrometer.prometheus.PrometheusConfig
import io.micrometer.prometheus.PrometheusMeterRegistry

object MetricsManager {
    // 1. Prometheus 레지스트리 생성
    val prometheusRegistry = PrometheusMeterRegistry(PrometheusConfig.DEFAULT)

    init {
        // 2. Resilience4j 메트릭을 Prometheus에 바인딩
        TaggedCircuitBreakerMetrics
            .ofCircuitBreakerRegistry(NiceCircuitManager.registry)
            .bindTo(prometheusRegistry)
            
        // 3. JVM 기본 메트릭(CPU, Memory)도 함께 바인딩 (SRE 필수)
        io.micrometer.core.instrument.binder.jvm.JvmMemoryMetrics().bindTo(prometheusRegistry)
        io.micrometer.core.instrument.binder.system.ProcessorMetrics().bindTo(prometheusRegistry)
    }

    // 4. Prometheus가 Scrape해갈 텍스트 결과물 반환
    fun getMetrics(): String = prometheusRegistry.scrape()
}

Manual control implementation

class NiceAdminService(private val circuitBreaker: CircuitBreaker) {

    // 강제로 서킷을 정상 상태로 복구 (Manual Close)
    fun forceClose() {
        circuitBreaker.transitionToClosedState() 
        println("Circuit Breaker [${circuitBreaker.name}] has been manually CLOSED.")
    }

    // 강제로 서킷을 차단 (Manual Open)
    fun forceOpen() {
        circuitBreaker.transitionToForcedOpenState()
        println("Circuit Breaker [${circuitBreaker.name}] has been manually FORCED_OPEN.")
    }

    // 서킷 상태 확인
    fun getStatus(): String = circuitBreaker.state.name
}

After creating this service, we can set up separate ports like 8081 for it, instead of the existing port 80.

// 관리용 전용 Netty Handler 예시
class AdminHttpHandler(private val adminService: NiceAdminService) : SimpleChannelInboundHandler<FullHttpRequest>() {
    override fun channelRead0(ctx: ChannelHandlerContext, req: FullHttpRequest) {
        val uri = req.uri()
        
        val responseText = when {
            uri == "/admin/circuit/close" -> {
                adminService.forceClose()
                "Circuit manually CLOSED"
            }
            uri == "/admin/circuit/open" -> {
                adminService.forceOpen()
                "Circuit manually FORCED_OPEN"
            }
            else -> "Unknown command"
        }
        
        sendResponse(ctx, responseText)
    }
}

class NiceNettyHandler(private val circuitBreaker: CircuitBreaker) : ChannelInboundHandlerAdapter() {
    
    override fun channelRead(ctx: ChannelHandlerContext, msg: Any) {
        // 1. 서킷 권한 획득 확인 (Non-blocking)
        if (!circuitBreaker.tryAcquirePermission()) {
            // 서킷이 오픈된 상태라면 즉시 거절
            ReferenceCountUtil.release(msg)
            ctx.writeAndFlush(ErrorResponse("CIRCUIT_OPEN"))
            return
        }

        val start = System.nanoTime()
        // 2. 실제 NICE 전문 전송 로직 수행 (예시)
        ctx.fireChannelRead(msg) 
        
        // 3. 성공/실패 기록은 해당 작업이 완료되는 Future Listener에서 수행
        // circuitBreaker.onSuccess(duration, TimeUnit.NANOSECONDS)
        // circuitBreaker.onError(duration, TimeUnit.NANOSECONDS, throwable)
    }
}

Besides HTTP, it's also possible to register it as a JMX MBean and use forceClose() in the UI by connecting to the server with tools like JConsole or VisualVM.

Alternatively, one could use a config storage with dynamicConfig.

Thus, a foundation has been laid for flow control on the NICE proxy server, allowing it to handle numerous requests despite having only 80 sessions.

With the current request volume, a single instance is sufficient, so there are 16 worker threads. There shouldn't be any particular issues.

We deploy two NICE proxies because if one pod fails, the remaining one must be able to handle the average traffic.

The most important thing is NICE itself, so if using coroutines, use a Semaphore, or if pure Java, maintain exactly 80 sessions with a ConnectionPool(80).

Let's maintain a buffer of about 2000-3000 messages (30 seconds' worth) in SQS, and for traffic exceeding this, instead of immediately dropping it with a circuit breaker, let's keep the system alive.

Pre-inquiry Consumer Performance Improvement #

Now, let's check the pre-inquiry system.

The pre-inquiry system needs to perform tasks such as customer registration in our core banking system and internal CSS (Credit Scoring Service), in addition to NICE, and handle responses.

Here, NICE and KCB calls can be processed concurrently, so we can reduce latency by parallelizing them using coroutines.

What if NICE takes 2 seconds, KCB 1 second, internal CSS 1 second, and core banking registration 2 seconds? Processing all of them sequentially would take 6 seconds. Since core banking registration requires NICE, KCB, and internal CSS information, it should be done last. If NICE, KCB, and CSS are processed concurrently, it would only take 4 seconds.

The existing synchronous code is as follows:

fun registerLoanSequential(user: User): RegistrationResult {
    val start = System.currentTimeMillis()

    // 1. NICE 조회 (2s)
    val niceData = externalService.fetchNiceData(user)
    
    // 2. KCB 조회 (1s) - NICE가 끝날 때까지 대기함
    val kcbData = externalService.fetchKcbData(user)
    
    // 3. 내부 CSS 산출 (1s) - KCB가 끝날 때까지 대기함
    val cssData = internalService.calculateCss(user, niceData, kcbData)

    // 4. 코어뱅킹 등록 (2s) - 위 모든 정보가 필요함
    val result = coreBankingService.register(user, niceData, kcbData, cssData)

    println("Total Time: ${System.currentTimeMillis() - start}ms") // 약 6000ms
    return result
}

suspend fun registerLoanParallel(user: User) = coroutineScope {
    val start = System.currentTimeMillis()

    // 1~3번 작업을 동시에 시작 (Non-blocking)
    val niceDeferred = async { externalService.fetchNiceData(user) }
    val kcbDeferred = async { externalService.fetchKcbData(user) }
    
    // 내부 CSS 산출 (NICE/KCB 데이터가 CSS 산출에 필요하다면 CSS는 await 후 실행해야 함)
    // 여기서는 CSS가 독립적이라고 가정하셨으므로 병렬 실행
    val cssDeferred = async { internalService.calculateCss(user) }

    // 모든 작업이 끝날 때까지 대기 (가장 오래 걸리는 NICE의 2초에 수렴)
    val niceData = niceDeferred.await()
    val kcbData = kcbDeferred.await()
    val cssData = cssDeferred.await()

    // 4. 코어뱅킹 등록 (2s) - 결과가 다 모인 시점(2s)에 실행
    val result = coreBankingService.register(user, niceData, kcbData, cssData)

    println("Total Time: ${System.currentTimeMillis() - start}ms") // 약 4000ms
    result
}

You can register it with parallel processing as shown above. Setting individual timeouts for each task is essential, and while thread occupancy is short, be mindful of a surge in concurrent requests.

But what if an error occurs in one system? It should fail. If an exception occurs in one async block, the failure propagates up to the parent coroutine. This means everything is canceled.

Usually, if there's no dependency, one might build a fault-tolerant system using supervisorScope, but due to data consistency, a failure means a complete failure here.

We set it up for manual intervention in case of an exception.

Failed requests leave events in a DB based on a DLQ or transaction outbox pattern, and developers are set up to handle them manually with a manual response and notification system.

This part could also be automated, but for consistency, we'll leave it as is for now.

When the pre-inquiry process starts, a pending record is first created in the DB.

If any of the coroutine parallel tasks fail and an exception occurs, the catch block updates the status of that record to FAILED and records the detailed reason for the failure.

At this point, a message indicating that manual verification is required is published to the queue simultaneously with the DB update, using the Transaction Outbox Pattern.

The ErrorQueue contains failed tasks that developers need to check in production, and the DLQ is the last resort where messages ultimately arrive if even the consumer processing the Error Queue fails due to logic errors or infrastructure failures.

suspend fun handleInquiryFailure(userId: String, error: Throwable) {
    db.transaction {
        // 1. DB 상태 업데이트 (NICE_KEY 등이 null인 상태로 FAILED 마킹)
        inquiryRepository.updateToFailed(userId, error.message)
        
        // 2. Outbox 테이블에 수기 대응 이벤트 삽입
        // 이 이벤트는 별도 Relay를 통해 SQS(Error Queue)로 전송됨
        outboxRepository.save(ManualInquiryEvent(userId = userId, reason = error.message))
    }
}

Here, core banking user registration system data is stored in the DB, holding keys for inquiry results like NICE and KCB. Since null keys indicate failed agencies,

if developers create a console to retry for users with null keys at a specific time, it can be handled.

Then, only data with null keys would be retried. If all keys become non-null (inquiry successful), the subsequent process can proceed, and a response can be sent. While using an RDB might struggle with load, it's sufficient for the current traffic.

However, if traffic increases to a point where an RDB can no longer be used, data should be stored using a CDC-based mechanism that reads the DB's WAL log directly instead of having a separate outbox table and adding inserts. The Debezium Kafka Connect combination is the current standard. While NoSQL or local log appends are options,

the reason why reading the WAL is faster is that an outbox involves two writes (insert + update + outbox insert), whereas CDC only performs the application's business logic. The WAL is data that is kept anyway for DB recovery or replication, and CDC asynchronously reads already recorded files, thus not adding additional load to the application's transactions.

Since it stores the change history itself, it reads only business logic inserts, not an outbox table. Because it's looking at the change history, it can detect if a business change history failed. In other words, it takes on the role of the outbox table itself.


If traffic increases from 200 TPS -> 2000 TPS -> 20000 TPS, #

At this point, it goes beyond simply increasing the number of servers; it requires a shift in architectural paradigm.

2000 TPS Stage: Resource Optimization and Infrastructure Advancement #

At this stage, the existing RDB and basic auto-scaling reach their limits.

HPA & Tuning: Increasing the number of pods to dozens and kernel parameter tuning (somaxconn, file-max) are essential.

Along with Netty's epoll-based optimization, direct buffer usage must be precisely monitored to reduce unnecessary GC overhead.

And the CDC mentioned earlier must be introduced. RDBs will struggle to handle 2000 outbox writes per second, so a complete transition to WAL-based CDC (Debezium Kafka) is needed to reduce DB write load.

External integration sessions – there's no other answer, they must be increased. The physical number of sessions must be increased from 80 to 800. At this point, to efficiently manage sessions, consider a structure where multiple pods share a session pool, or deploy a dedicated session gateway.

20000 TPS Stage #

This stage involves world-class traffic levels, which are difficult to handle with a single cluster or DB.

Cell-based architecture. Instead of processing all traffic in one massive cluster, isolate it into units called cells.

Assign independent Netty proxy groups, databases, and queues per specific user group (e.g., sharding by user ID).

Even if a failure occurs in a specific cell, only a portion of the 20000 TPS is affected, completely preventing failure propagation.

Adaptive Concurrency Limiting

Simple fixed-threshold circuit breakers might react too slowly or be unnecessarily sensitive in a 20000 TPS environment.

Let's introduce an adaptive limiter that dynamically allows traffic based on real-time latency, using Little's Law, similar to Netflix's concurrency-limits library.

Assuming it's the NICE system, even a 10ms delay in system response would immediately adjust incoming traffic to prevent system collapse.

Edge Logic Pre-filtering: All 20,000 requests should not be allowed to reach the business logic. Here, the Netty proxy is used.

At the AWS WAF or API Gateway level, Redis-based global rate limiters or similar mechanisms can be implemented to immediately drop invalid or duplicate requests at the edge.

External System Capacity #

Assuming it's NICE, the question is whether the system itself can withstand 20,000 TPS.

First, prioritize business needs. If NICE only supports up to 5000 TPS, the remaining 15000 requests (or about 17000 considering other calls) must be discarded. In this case, priority queuing should be implemented to retrieve requests based on priority, such as VIP customers or imminent loan approvals.

Kernel-level optimization is also needed. To maintain thousands of TCP connections for high TPS, issues like per-pod FD (File Descriptor) limits and TCP port exhaustion must be resolved.

This requires outbound traffic distribution using VIPs (virtual IPs) instead of a single IP.

Back-End/nice.md