estudy 한국어/English

Symptoms of MTU Mismatch

1,019 단어·5 분·원문(.md)

MTU, or Maximum Transmission Unit, refers to the maximum packet size in bytes that a network interface can transmit at once without fragmentation.

The default specification typically uses 1500 bytes in standard Ethernet environments, but modern cloud internal networks like AWS and GCP often use 9000 bytes by default for performance improvement.

MTU Mismatch is a phenomenon that occurs when the MTU sizes configured between the source and destination that send and receive data, or between network devices such as routers, VPNs, and firewalls in between, are different.

DF Bit (Don't Fragment): Modern TCP/IP communication transmits packets with a df=1 flag in the packet header to prevent performance degradation, indicating that the packet should not be fragmented. If a packet size is, say, 9000, and it's larger than an intermediate router's MTU of 1500, but the DF bit is set, the router cannot fragment the packet and simply drops it.

Problem #

The scary thing about MTU Mismatch is that it doesn't completely kill a service, but rather puts it in a half-dead state.

It's difficult to debug because it tricks you into thinking the network is connected.

  • The problem is that small packets succeed: small data packets of tens or hundreds of bytes, such as pings, 3-way handshakes, SSH connections, and health checks, pass through without issues. From the monitoring system's perspective, everything is normal.
  • Large packets vanish: connection hangs, users upload large images, retrieve thousands of rows from a DB select query, or download large JSON responses, causing traffic to momentarily stop. Intermediate devices drop packets without properly returning even an icmp fragmentation needed error message, leaving clients and servers waiting for data that never arrives.
  • There's also thread pool exhaustion: from the application's perspective, the connection isn't broken, so it enters an indefinite waiting state. If these requests accumulate indefinitely, the WAS worker threads eventually become exhausted, paralyzing the entire service.

Example #

In a normal situation, communication between EC2 instances within an AWS VPC, both supporting MTU 9000, allows for ultra-fast processing of large file transfers or massive DB queries without bottlenecks.

A failure scenario (which is also difficult to debug) might be, for example, when a company builds a hybrid cloud, connecting an AWS VPC (MTU 9000) and an on-premise data center (MTU 1500) via an IPSec VPN. Let's say that during the VPN tunneling process, encryption headers are added, shrinking the actual available MTU to 1436 bytes.

  • A developer connects to an EC2 server via SSH (under 1436 bytes) and confirms the connection works.
  • Let's say a db select 1 from EC2 to on-premise also works.
  • In a production environment, a user logs in and executes select * from payment_history. The result is 5MB, and from this point, the browser loading bar stops, and server logs halt. The network team says firewalls are open and pings work, and the DBA says the query has finished executing. In other words, only the backend developer is going crazy in the middle.

Solution #

Finding the blocked path MTU at the infrastructure level and unifying device settings is indeed the fundamental solution.

As an SRE, you should also be able to prove the problem using the terminal and defend against service failure at the code level.

Proving Bottleneck Sections by Checking Terminal Ping DF Bit #

Instead of a simple ping, you force a larger packet size and set the DF bit to prevent fragmentation, then trace where packets are being dropped.

# -M do: Set Don't Fragment bit
# -s 1472: Creates a packet of exactly 1500 bytes by adding ICMP header (8) + IP header (20)
$ ping -c 4 -M do -s 1472 10.100.1.50

PING 10.100.1.50 (10.100.1.50) 1472(1500) bytes of data.
From 10.0.5.1 icmp_seq=1 Frag needed and DF set (mtu = 1436)
From 10.0.5.1 icmp_seq=2 Frag needed and DF set (mtu = 1436)

--- 10.100.1.50 ping statistics ---
4 packets transmitted, 0 received, +2 errors, 100% packet loss, time 3045ms

Looking at the terminal results, you can provide clear evidence to the network team in one go: the intermediate router 10.0.5.1 has an MTU limited to 1436, and large packets are being dropped. You need to ask them to enable MSS clamping on the VPN device or adjust the MTU.

Application-Level Defense Logic #

If the server gets stuck in an indefinite hang before network devices are fixed, the entire server could crash.

When packets don't arrive due to MTU issues, thorough read timeouts and defensive HTTP client settings must be applied.

import okhttp3.OkHttpClient
import okhttp3.Request
import org.slf4j.LoggerFactory
import org.springframework.context.annotation.Bean
import org.springframework.context.annotation.Configuration
import java.net.SocketTimeoutException
import java.util.concurrent.TimeUnit

@Configuration
class ResilientHttpClientconfig {
    private val log = LoggerFactory.getLogger(this::class.java)

    @Bean
    fun customOkClient(): OkHttpClient {
        
        return OkHttpClient.Builder()
            // 1. Connection Timeout: TCP handshake (small packets) usually succeeds, so keep it short (e.g., 3 seconds)
            .connectionTimeout(3, TimeUnit.SECONDS)
            // 2. Read Timeout: If large response packets vanish mid-transmission due to an MTU black hole, prevent threads from waiting indefinitely by throwing an exception immediately after 5 seconds and returning the thread.
            .readTimeout(5, TimeUnit.SECONDS)
            // 3. Write Timeout: Also prevents hangs when uploading large data to the server.
            .writeTimeout(5, TimeUnit.SECONDS)
            // 4. Connection Pool: Manage and clean up hung/dead connections.
            .connectionPool(okhttp3.ConnectionPool(50, 5, TimeUnit.MINUTES))
            // 5. Disable immediate retries on network issues. MTU issues will fail again on retry, so it's better to turn this off.
            .retryOnConnectionFailure(false)
            .build()
    }
}

class PaymentClient(private val httpClient: OkHttpClient) {
    private val log = LoggerFactory.getLogger(this::class.java)

    fun fetchLargePaymentHistory(userId: String): String {
        val request = Request.Builder(
            .url("http://on-premise-db-api.internal/payments/$userId")
            .build()
        )

        return try {
            httpClient.newCall(request).execute().use { response ->
                if (!response.isSuccessful) throw RuntimeException("API Error")
                response.body?.string() ?: ""
            }
        } catch (e: SocketTimeoutException) {
            // Falls here if a black hole occurs due to MTU Mismatch
            // Prevents entire service paralysis and allows for quick responses to temporary outages or fallback for customers.
            log.error("MTU suspected/network delay, timeout occurred while receiving large response")
            throw CustomNetworkException()
        }
    }
}

Network infrastructure issues like MTU mismatch can occur at any time.

Therefore, it is essential for application code to always assume the worst network conditions and implement timeout safeguards.

This is a fundamental quality of a true SRE and server engineer.

SRE/question/q_34.md