Fluent-Bit log routing by namespace in Kubernetes

You’ve got a mixed-used Kubernetes cluster. You have a very nifty security logging system that wants to audit everything from certain namespaces, and of course you want general logs from all. How would you go about configuring Fluent-bit to route all logs to one output, and only a single namespace to another, simultaneously? Read on to learn how to perform Fluent-Bit rlog outing by namespace.

First, let’s understand the flow of information. Fluent-bit operates with a set of concepts (Input, Output, Filter, Parser). Inputs consume data from an external source, Parsers modify or enrich the log-message, Filter’s modify or enrich the overall container of the message, and Outputs write the data somewhere.

The typical flow in a Kubernetes Fluent-bit environment is to have an Input of type Tail, which conceptually does a tail -f on all your log files. It relies on the fact the files have a magic name (incorporating the pod/namespace/container information).

From the Input we then go through a Filter of type Kubernetes. This takes the input parsed from the filename (Pod/Namespace/Container), does a lookup to the Kubernetes API (e.g. to look at annotations). From here we go to the output stage.

In the example use case here, we want to send *all* records to the default Output, and *some* (e.g. all from a namespace) to a secondary Output. Let’s dig in.

[SERVICE]
    Flush        1
    Daemon       Off
    Log_Level    info
    Parsers_File parsers.conf

[INPUT]
    Name             tail
    Path             /var/log/containers/*log
    Parser           docker_no_time
    Tag              kube...
    Tag_Regex        (?[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?[^_]+)_(?.+)-
    Refresh_Interval 5
    Mem_Buf_Limit    50KB
    Skip_Long_Lines  On

[FILTER]
    Name                kubernetes
    Match               kube.*
    Merge_Log           Off
    Regex_Parser        kube-tag
    K8S-Logging.Parser  On
    K8S-Logging.Exclude On

[OUTPUT]
    Match *
    Name  stdout
    Format json_lines
    JSON_Date_Format iso8601

[OUTPUT]
    Name file
    Match kube.ns2.*
    Path ns2-logs-only.txt

OK, we added two different outputs. One matches all, one only namespace ns2. This Match parameter is how we do Fluent-Bit routing by namespace.

[PARSER]
     Name   json
     Format json
     Time_Key time
     Time_Format %Y-%m-%dT%H:%M:%S.%L
 [PARSER]
     Name    kube-tag
     Format  regex
     Regex    (?[^.]+).(?[^.]+).(?[^.]+).(?[^.]+)
 [PARSER]
     Name         docker_no_time
     Format       json
     Time_Key     time
     Time_Format  %Y-%m-%dT%H:%M:%S.%L
     Time_Keep    Off
     Decode_Field_As   escaped    log
     Decode_Field_As   escaped    stream

OK what have we achieved? Well, for namespace ns2 we route that output to a file. For all logs (including ns2) we route that to stdout. We have done Fluent-Bit log routing by namespace in Kubernetes.

Now, replace stdout and file with your outputs. And profit!

You may use this tag match routing for other users. I use it with 2 different clusters, each running fluent-bit. 1 of them is configured to forward to the other, and the logs are routed by cluster source.

I also use this technique to route the Node logs to one Elasticsearch Index, and the Pod logs to another.