In a log collection system, filtering refers to the preprocessing step of refining raw log data pouring in from application servers, network infrastructure, etc., before transmitting it to the final central storage (e.g., Elasticsearch, Loki, Splunk).

It's not just about discarding logs; it also involves the following operations:

Drop: Completely discard meaningless logs with no storage value from the collection pipeline.
Masking/Redaction: Obscure strings containing personal information or security-sensitive data with ***.
Enrichment: Add log IP-based location information or container labels to make the data suitable for analysis.

Problem Definition #

Let's assume a "log everything" system was designed to store all generated logs. This approach inevitably leads to problems as the system scales.

Exploding Storage Costs and Ingest Bottlenecks: The volume of log data is directly proportional to service traffic. If all logs, including meaningless normal logs like "200 OK," are stored during a data surge, storage costs will increase exponentially, and the collection server itself will become overloaded, leading to the loss of critical error logs.
Security and Compliance Violations: Due to options enabled for debugging in development environments or unexpected errors, user passwords, session tokens, or credit card numbers might be stored in plain text in the log repository, leading to severe security breaches.
Reduced Visibility: If hundreds of millions of logs accumulate daily, finding the few lines of error logs indicating the cause of a failure requires sifting through an enormous amount of garbage data. Log search queries themselves might time out, causing you to miss the golden hour.

Example #

To illustrate a normal situation first, in the early stages of development with low traffic, collecting all web server logs (e.g., Nginx access logs) directly from one or two servers works fine.

Let's consider a situation that triggers an incident cost bomb. Without filtering, when a service grows, migrates to k8s, and needs to scale out to 100 pods, k8s liveness/readiness probes call the /health endpoint every 2-3 seconds for each pod.

Imagine the log collector's CPU usage suddenly hits 100%, and 500GB of log storage fills up in just half a day. You try to query the storage to analyze the cause, but the engine crashes due to the massive amount of data.

Upon finally inspecting the logs, you find that over 90% of them are "GET /health 200"... a situation dominated by valueless health check success logs for system operations.

Solution #

It is necessary to analyze the log distribution, identify targets, and apply strong filtering rules at the collector-side agent or shipper.

Identifying Garbage Log Culprits Through Terminal Analysis #

When unsure which logs to filter, directly analyze the original log files on the production server using terminal tools to quantify the proportion of unnecessary logs.

# Extract HTTP Request URI call frequency in descending order from Nginx access.log
$ awk -F'"' '{print $2}' /var/log/nginx/access.log | awk '{print $2}' | sort | uniq -c | sort -nr | head -n 5

850412 /health
  42105 /api/v1/users/profile
  12040 /api/v1/payments/status
   5021 /login
   1024 /favicon.ico

It was confirmed that the /health path accounts for an overwhelming proportion of the total log volume, so this is confirmed as a filtering target! Something like this.

Configuring Unnecessary Log Dropping and Sensitive Information Masking in Log Collectors (Fluent Bit) #

Taking Fluent Bit, one of the most popular collectors, as an example, you can perform preprocessing by adding a Filter pipeline to its configuration file. The key is to discard logs at the client node before they travel across the network to the central storage.

``lni

fluent-bit.conf

1. Completely discard useless health check logs (Drop)

[FILTER] Name grep Match nginx.* # Exclude logs containing the /health path from the collection pipeline Exclude log ^.GET /health.$

2. Masking passwords/tokens included in logs (Masking)

[FILTER] Name modify Match app.logs.* # Replace "password=blahblah" or "token=blahblah" parts with *** Condition Key_Value_Matches log .(password|token)=. Set log_masked true # (In a real environment, use a Lua script filter or a regex replacement plugin to "REDACT" the corresponding value)


By going through such a preprocessing pipeline, the amount of data flowing into the central storage channel can be dramatically reduced, while retaining only the core signals necessary for actual debugging and analysis. This simultaneously ensures the performance and visibility of the monitoring system.