Symptoms of File Descriptor Exhaustion

FD exhaustion is a phenomenon encountered when dealing with high-performance servers or large-scale data.

File Descriptor #

A file descriptor is an integer handle for accessing abstract resources. Following the Unix/Linux philosophy of "everything is a file," all I/O resources such as sockets, pipes, and devices, not just regular files, are managed through FDs.

Kernel's Internal 3-Stage Mapping Structure #

When a process opens a file, the kernel doesn't just assign an ID; it creates a complex table structure.

process descriptor (task struct): Each process has its own FD table. Here, FD values 0, 1, and 2 act as array indices.
open file table (system-wide): This table is shared by all processes and stores the state of how a file is opened, such as its offset, read/write permissions, etc.
Inode table(Vnode Table): This contains information about actual file data on disk or whatever hardware devices are connected.

Symptoms of File Descriptor Exhaustion #

When a system or process reaches the upper limit of FDs it can be allocated, it becomes unable to acquire new resources, leading to service unavailability.

Software-level Errors: EMFILE, ENFILE #

EMFILE (Too many open files): Occurs when a single process has exhausted its allocated per-process limit.
ENFILE (File table overflow): Occurs when the system-wide limit for the maximum number of FDs available to the entire system is reached. This is a dangerous signal that can paralyze the entire system.

These can be distinguished at the process level and the system-wide level.

Network Service Interruption: Accept Failures #

This is a critical phenomenon in server applications.

When a new client attempts to connect, the kernel must create a new socket FD via accept().
If FDs are exhausted, accept() will fail, causing clients to experience "connection refused" or indefinite waiting. Existing connections are maintained, but new incoming connections are blocked.

Alternatively, if an error occurs and there are no FDs available to open the log file itself, it can lead to log omissions and a "silent failure" state where debugging becomes impossible, making root cause analysis difficult.

Pipe and IPC communication failures can also occur, as the creation of Pipes or Eventfds for inter-process communication becomes impossible. In multi-process/thread environments, this can halt synchronization or data transfer, making the application appear like a zombie or in a deadlock state.

Managing FD Limits #

In Linux systems, there are ways to check and control these limits.

ulimit -Sn # Recommended limit currently available to a process

ulimit -Hn # Maximum limit that can only be modified with root privileges (S Soft, H Hard)

cat /proc/sys/fs/file-max # Maximum number of FDs that can be opened system-wide by the kernel

lsof -p <pid> | wc -l # Check the number of FDs used by a specific process

Let's look at an example.

For instance, if your application logs show:

java.io.IOException: Too many open files
# 혹은
Accept error: errno = 24 (EMFILE)

Let's say you see logs like these. We'll analyze them.

First, you need to check the capacity of your system.

$ ulimit -Sn  # Soft limit: currently applied limit
1024
$ ulimit -Hn  # Hard limit: maximum expandable limit
4096

This process is currently in a situation where it can only open up to 1024 FDs. For a server-grade system, 1024 is a relatively small number, so it should be set to tens of thousands.

Next, identify the culprit process.

Find which process is consuming all the FDs.

$ lsof | awk '{print $2}' | sort | uniq -c | sort -nr | head -n 5
   1020 12345  # PID 12345 is using 1,020 (almost at its limit!)
    150 6789
     45 1

Detailed analysis of a specific process is also possible, as shown below.

$ ls /proc/12345/fd | wc -l
1020

Here, with a ulimit of 1024, process 12345 is using 1020 FDs, indicating that this process will soon crash or be unable to accept new requests.

Finally, determine if this is simply due to a high number of users or if there's a code error causing an FD leak.

$ lsof -p 12345
COMMAND  PID  USER   FD   TYPE  DEVICE SIZE/OFF    NODE NAME
java    12345 user   10u  IPv4  123456      0t0     TCP *:8080 (LISTEN)
java    12345 user   11u  REG    8,1     4096  789012 /tmp/tempfile.tmp (deleted)
java    12345 user   12u  REG    8,1     4096  789013 /tmp/tempfile.tmp (deleted)
... (수백 개 반복) ...

The "deleted" tag next to the filename indicates a typical resource leak: the program deleted the file but failed to call close(), so the kernel is still holding onto the FD.

The immediate temporary solution is, of course, to increase the process limit, using prlimit for this.

# Change both Soft/Hard limits for PID 12345 to 65535
$ sudo prlimit --pid 12345 --nofile=65535:65535

This change won't persist after a server reboot. To make it permanent, you need to modify /etc/security/limits.conf.

$ sudo vi /etc/security/limits.conf

# Add to the bottom
* soft    nofile    65535
* hard    nofile    65535

Fundamentally, you should find the leak point identified by lsof and analyze whether resource-closing code, like try-with-resources (Java) or defer file.Close() (Go), is missing, or if there's a similar issue in any open-source libraries being used, and then resolve it.