estudy 한국어/English

Causes of DNS Lookup Timeout

797 단어·4 분·원문(.md)

A DNS Lookup Timeout occurs when an application attempts to connect to a specific domain (e.g., api.service.com),

and the system goes through a process (DNS Resolution) to convert this name into an IP address. If a response is not received within the configured timeout period,

a failure occurs, which is known as a DNS Lookup Timeout.

  • Intermittent Errors: The system usually works fine, but suddenly UnknownHostException or Temporary failure iname resolution errors appear.
  • Response Latency: If the timeout is set to 5 seconds, an API call that should ideally complete in 0.1 seconds will instead take the full 5 seconds due to DNS resolution before failing.
  • Cascading Failure: As DNS requests pile up, all application worker threads become occupied, leading to the entire service becoming unresponsive.

These phenomena indicate an error type that requires careful attention.

So, why do they occur? Here are some of the main causes:

  1. UDP Packet Loss: DNS primarily uses UDP. UDP operates on a "fire and forget" basis, meaning if the network is congested, packets may be dropped without immediate retransmission.
  2. ndots Configuration Issue (k8s environment): Due to the ndots:5 setting in /etc/resolv.conf, when searching for a domain like google.com, the system might make up to 5 unsuccessful attempts, starting with google.com.default.svc.cluster.local. (This needs to be resolved separately, either by reducing the value in k8s or by developers explicitly appending a dot to the end of the FQDN.)
  3. Conntrack Table Full: If the Linux server's connection tracking table (conntrack) becomes full, DNS response packets might be ignored.
  4. Rate Limiting: In cloud environments like AWS and GCP, there's a limit on DNS queries per second per instance (e.g., 1024 PPS). Exceeding this limit will result in immediate drops.

Resolving these issues is crucial as it ensures system stability, reduces API failure rates, saves costs (by eliminating retry expenses), and improves user experience.


Reproducing and Verifying DNS Timeout #

Let's verify this directly in the terminal. First, let's look at the server's DNS configuration.

cat /etc/resolv.conf

nameserver 8.8.8.8
options timeout:2 attempts:3 ndots:5

timeout 2 means waiting for 2 seconds, and attempts 3 means trying 3 times, so it will take a total of 6 seconds (2 x 3).

You can check the response speed using the dig command.

# Specify a particular DNS server (8.8.8.8) to look up google.com
dig @8.8.8.8 google.com

; <<>> DiG 9.16.1-Ubuntu <<>> google.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 45678  # <--- status: NOERROR is important!
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; QUESTION SECTION:
;google.com.			IN	A

;; ANSWER SECTION:
google.com.		237	IN	A	142.250.207.46  # <--- The IP address we wanted

;; Query time: 15 msec  # <--- 1. Response speed (very fast)
;; SERVER: 8.8.8.8#53(8.8.8.8)  # <--- 2. Who was queried
;; WHEN: Thu Feb 19 18:50:00 KST 2026
;; MSG SIZE  rcvd: 55

I will force a timeout by specifying a fake IP DNS server.

dig @1.2.3.4 google.com +timeout=2

;; [Network connection delayed...]
;; connection timed out; no servers could be reached  # <--- This is what will appear in your error logs.

Let's also trace the packet flow. Using tcpdump, you can see in real-time whether packets were sent and if no response was received.

# -n: don't convert hostnames, -vv: verbose output, -i any: all interfaces
sudo tcpdump -i any port 53 -n -vv

# Normal
18:55:01.123 IP 192.168.1.10.54321 > 8.8.8.8.53: [1234] A? google.com. (28)
18:55:01.138 IP 8.8.8.8.53 > 192.168.1.10.54321: [1234] 1/0/0 A 142.250.207.46 (44)

# Timeout occurred
18:56:00.000 IP 192.168.1.10.54321 > 8.8.8.8.53: [5678] A? google.com. (28)
18:56:05.000 IP 192.168.1.10.54321 > 8.8.8.8.53: [5678] A? google.com. (28) # <--- Retrying!
18:56:10.000 IP 192.168.1.10.54321 > 8.8.8.8.53: [5678] A? google.com. (28) # <--- Retrying again...

If you manage k8s, you should be wary of the ndots trap. This is an example of DNS slowing down due to the /etc/resolv.conf setting on a Linux server.

cat /etc/resolv.conf

nameserver 10.96.0.10
search my-namespace.svc.cluster.local svc.cluster.local cluster.local
options ndots:5  # <--- This is the culprit!

Because of this setting, when the system calls google.com, it mistakenly treats all domains with fewer than 5 dots as internal domains and queries them in the following order:

  1. google.com.my-namespace.svc.cluster.local (No response/Error)
  2. google.com.svc.cluster.local (No response/Error)
  3. google.com.cluster.local (No response/Error)
  4. google.com (Finally successful!)

Conclusion #

Consequently, you can try optimizing ndots and adjusting the Timeout value (e.g., allowing requests to fail faster if they are piling up).

You can also check if nscd or NodeLocal DNSCache are installed for Local DNS Caching to help resolve the issue.

Alternatively, you can modify your code to use FQDNs by appending a dot at the end when calling external domains (e.g., google.com.) to skip ndots lookups.

Adding options single-request-reopen can also prevent kernel race conditions that occur during simultaneous IPv4/IPv6 requests.

SRE/question/q_22.md