You’ve got a mixed-used Kubernetes cluster. You have a very nifty security logging system that wants to audit everything from certain namespaces, and of course you want general logs from all. How would you go about configuring Fluent-bit to route all logs to one output, and only a single namespace to another, simultaneously? Read on to learn how to perform Fluent-Bit rlog outing by namespace.
First, let’s understand the flow of information. Fluent-bit operates with a set of concepts (Input, Output, Filter, Parser). Inputs consume data from an external source, Parsers modify or enrich the log-message, Filter’s modify or enrich the overall container of the message, and Outputs write the data somewhere.
The typical flow in a Kubernetes Fluent-bit environment is to have an Input of type
Tail, which conceptually does a
tail -f on all your log files. It relies on the fact the files have a magic name (incorporating the pod/namespace/container information).
From the Input we then go through a
Filter of type
Kubernetes. This takes the input parsed from the filename (Pod/Namespace/Container), does a lookup to the Kubernetes API (e.g. to look at annotations). From here we go to the output stage.
In the example use case here, we want to send *all* records to the default Output, and *some* (e.g. all from a namespace) to a secondary Output. Let’s dig in.
[SERVICE] Flush 1 Daemon Off Log_Level info Parsers_File parsers.conf [INPUT] Name tail Path /var/log/containers/*log Parser docker_no_time Tag kube... Tag_Regex (?[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?[^_]+)_(?.+)- Refresh_Interval 5 Mem_Buf_Limit 50KB Skip_Long_Lines On [FILTER] Name kubernetes Match kube.* Merge_Log Off Regex_Parser kube-tag K8S-Logging.Parser On K8S-Logging.Exclude On [OUTPUT] Match * Name stdout Format json_lines JSON_Date_Format iso8601 [OUTPUT] Name file Match kube.ns2.* Path ns2-logs-only.txt
OK, we added two different outputs. One matches all, one only namespace ns2. This Match parameter is how we do Fluent-Bit routing by namespace.
Decode_Field_As escaped log
Decode_Field_As escaped stream
OK what have we achieved? Well, for namespace
ns2 we route that output to a file. For all logs (including
ns2) we route that to stdout. We have done Fluent-Bit log routing by namespace in Kubernetes.
file with your outputs. And profit!
You may use this tag match routing for other users. I use it with 2 different clusters, each running fluent-bit. 1 of them is configured to forward to the other, and the logs are routed by cluster source.
I also use this technique to route the Node logs to one Elasticsearch Index, and the Pod logs to another.