# Increasing the usefulness of your Kubernetes Ingress logging

- Link: https://www.agilicus.com/increasing-the-usefulness-of-your-kubernetes-ingress-logging/
- Published: 2019-04-18T12:45:00+00:00
- Author: Don Bowman

Like most cloud folks you are probably using Kibana + Elasticsearch as part of your
log management solution. But did you know with a little regex-fu you can make that
logging more interesting? See the kibana expansion in the image, the URI, host, 
service, etc are all expanded for your reporting pleasure.

First, lets install our ingress with some annotations. I’ve made the interesting
bits red.

    ```bash
    helm install stable/nginx-ingress --name ingress \
      --set controller.service.externalTrafficPolicy=Local \
      --set rbac.create=true \
      --set controller.podAnnotations.fluentbit\\.io/parser=k8s-nginx-ingress
    ```

If your ingress is already running you can use this instead:

    ```has-small-font-size
    kubectl annotate pods --overwrite ingress-nginx-#### \
     fluentbit.io/parser=k8s-nginx-ingress
    ```

Now, lets install fluent-bit (to feed the Elasticsearch). We will add a custom-regex
for the [nginx-ingress log format](https://github.com/kubernetes/ingress-nginx/blob/master/docs/user-guide/nginx-configuration/log-format.md).
Its not the same as the nginx default so we can’t use the built-in.

    ```language-yaml
    image:
      fluent_bit:
        repository: fluent/fluent-bit
        tag: 0.14.1
      pullPolicy: IfNotPresent
    metrics:
      enabled: true
      service:
        port: 2020
        type: ClusterIP
    trackOffsets: false
    backend:
      type: es
      forward:
        host: fluentd
        port: 24284
      es:
        host: elasticsearch
        port: 9200
        index: kubernetes_cluster
        type: flb_type
        logstash_prefix: logstash
        time_key: "@timestamp"
        http_user:
        http_passwd:
        tls: "off"
        tls_verify: "on"
        tls_ca: ""
        tls_debug: 1

    parsers:
      enabled: true
      regex:
        - name: k8s-nginx-ingress
          regex:  '^(?<host>[^ ]*) - \[(?<real_ip>)[^ ]*\] - (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*) "(?<referer>[^\"]*)" "(?<agent>[^\"]*)" (?<request_length>[^ ]*) (?<request_time>[^ ]*) \[(?<proxy_upstream_name>[^ ]*)\] (?<upstream_addr>[^ ]*) (?<upstream_response_length>[^ ]*) (?<upstream_response_time>[^ ]*) (?<upstream_status>[^ ]*) (?<last>[^$]*)'
          timeKey: time
          timeFormat: "%d/%b/%Y:%H:%M:%S %z"
      json: []

    env: []

    podAnnotations: {}

    resources:
      limits:
        memory: 100Mi
      requests:
        cpu: 100m
        memory: 100Mi

    tolerations: []
    nodeSelector: {}

    filter:
      kubeURL: https://kubernetes.default.svc:443
      kubeCAFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      kubeTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
      kubeTag: kube
      mergeJSONLog: true

      enableParser: true

      enableExclude: true

    rbac:
      create: true

    serviceAccount:
      create: true
      name:
    ```

Once this is done you’ll have something like below in your logs. See how all the
fields are expanded to their own rather than being stuck in log: ?

    ```has-small-font-size
    {
      "_index": "logstash-2018.09.13",
      "_type": "flb_type",
      "_id": "s_0x1GUB6XzNVUp1wNV6",
      "_version": 1,
      "_score": null,
      "_source": {
        "@timestamp": "2018-09-13T18:29:14.897Z",
        "log": "10.40.0.0 - [10.40.0.0] - - [13/Sep/2018:18:29:14 +0000] \"GET / HTTP/1.1\" 200 9056 \"-\" \"curl/7.58.0\" 75 0.000 [default-front-end-80] 10.40.0.9:8079 9056 0.000 200 a134ebded3504000d63646b647e54585\n",
        "stream": "stdout",
        "time": "2018-09-13T18:29:14.897196588Z",
        "host": "10.40.0.0",
        "real_ip": "",
        "user": "-",
        "method": "GET",
        "path": "/",
        "code": "200",
        "size": "9056",
        "referer": "-",
        "agent": "curl/7.58.0",
        "request_length": "75",
        "request_time": "0.000",
       "proxy_upstream_name": "default-front-end-80",
        "upstream_addr": "10.40.0.9:8079",
        "upstream_response_length": "9056",
        "upstream_response_time": "0.000",
        "upstream_status": "200",
        "last": "a134ebded3504000d63646b647e54585",
        "kubernetes": {
          "pod_name": "ingress-nginx-ingress-controller-6577665f8c-wqg76",
          "namespace_name": "default",
          "pod_id": "0ea2b2c8-b5e8-11e8-bc8c-d237edbf1eb2",
          "labels": {
            "app": "nginx-ingress",
            "component": "controller",
            "pod-template-hash": "2133221947",
            "release": "ingress"
          },
          "annotations": {
            "fluentbit.io/parser": "k8s-nginx-ingress"
          },
          "host": "kube-spawn-flannel-worker-913bw7",
          "container_name": "nginx-ingress-controller",
          "docker_id": "40daa91b8c89a52e44ac1458c90967dab6d8a0e43c46605b0acbf8432f2d9f13"
        }
      },
      "fields": {
        "@timestamp": [
          "2018-09-13T18:29:14.897Z"
        ],
        "time": [
          "2018-09-13T18:29:14.897Z"
        ]
      },
      "highlight": {
        "kubernetes.labels.release.keyword": [
          "@kibana-highlighted-field@ingress@/kibana-highlighted-field@"
        ],
        "kubernetes.labels.app": [
          "nginx-@kibana-highlighted-field@ingress@/kibana-highlighted-field@"
        ],
        "kubernetes.annotations.fluentbit.io/parser": [
          "k8s-nginx-@kibana-highlighted-field@ingress@/kibana-highlighted-field@"
        ],
        "kubernetes.container_name": [
          "nginx-@kibana-highlighted-field@ingress@/kibana-highlighted-field@-controller"
        ],
        "kubernetes.pod_name": [
          "@kibana-highlighted-field@ingress@/kibana-highlighted-field@-nginx-@kibana-highlighted-field@ingress@/kibana-highlighted-field@-controller-6577665f8c-wqg76"
        ],
        "kubernetes.labels.release": [
          "@kibana-highlighted-field@ingress@/kibana-highlighted-field@"
        ]
      },
      "sort": [
        1536863354897
      ]
    }
    ```