# They got in via the logging! remote exploits and DDoS using the security logs

- Link: https://www.agilicus.com/they-got-in-via-the-logging-remote-exploits-and-ddos-using-the-security-logs/
- Published: 2019-04-19T12:45:00+00:00
- Author: Don Bowman

So the other day I posted my pride and joy [regex](https://www.agilicus.com/increasing-the-usefulness-of-your-kubernetes-ingress-logging/).
You know, this one?

    ```python
    '^(?<host>[^ ]*) - \[(?<real_ip>)[^ ]*\] - 
    (?<user>[^ ]*) \[(?<time>[^\]]*)\] 
    "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" 
    (?<code>[^ ]*) (?<size>[^ ]*) 
    "(?<referer>[^\"]*)" "(?<agent>[^\"]*)" 
    (?<request_length>[^ ]*) 
    (?<request_time>[^ ]*) 
    \[(?<proxy_upstream_name>[^ ]*)\] 
    (?<upstream_addr>[^ ]*) 
    (?<upstream_response_length>[^ ]*) 
    (?<upstream_response_time>[^ ]*) 
    (?<upstream_status>[^ ]*) (?<last>[^$]*)'
    ```

Seems simple, right? But, it leads to a set of questions:

 1. If we can get a ” in the path, we can do a quoting-style-escape to avoid getting
    logged
 2. The regex engine used in fluent-bit is onigmo. And it has some [CVE](https://www.cvedetails.com/cve/CVE-2017-6181/).
    This means its conceivable that a pattern that a user can put on the wire can escape
    into our trusted most privileged logging container (running privileged, node filesystem
    mounted, etc)
 3. DDoS. We log a lot. But the logs are often bigger than the thing they are logging.

For #3, consider this. Its a connection log from istio. Yes you read that right,
a TCP SYN( ~64 bytes) creates this in JSON of 816 bytes:

_{“level”:”info”,”time”:”2018-09-17T20:12:59.912982Z”,”instance”:”tcpaccesslog.logentry.
istio-system”,”connectionDuration”:”12.740646ms”,”connectionEvent”:”close”,”connection\
_security\_policy”:”none”,”destinationApp”:””,”destinationIp”:”10.244.1.57″,”destinationName”:”
payment-6cdc5b656-fkhxh”,”destinationNamespace”:”socks”,”destinationOwner”:”kubernetes://
apis/extensions/v1beta1/namespaces/socks/deployments/payment”,”destinationPrincipal”:””,”
destinationServiceHost”:””,”destinationWorkload”:”payment”,”protocol”:”tcp”,”receivedBytes”:
117,”reporter”:”destination”,”requestedServerName”:””,”sentBytes”:240,”sourceApp”:””,”
sourceIp”:”10.244.1.1″,”sourceName”:”unknown”,”sourceNamespace”:”default”,”sourceOwner”:”
unknown”,”sourcePrincipal”:””,”sourceWorkload”:”unknown”,”totalReceivedBytes”:117,”
totalSentBytes”:240}_

Hmm, so you are seeing where I am going. You remember a few years ago where we found
that NTP could be asked its upstream list? So a small packet would create a large
response? And, being UDP, could be spoofed, so the response could go to someone 
else? Making it a [great DDoS](https://www.csoonline.com/article/3118234/networking/ntp-reflection-attacks-hit-record-high.html)
source.

Well, my log. Your SYN costs me a lot more to receive than it costs you to send.
Think of all the mechanisms below that (elasticsearch, fluent-bit, kibana, storage,
network, cpu, ram, …).

Hmm.

Now about #2. That is a bit of a trouble point. Who wants to find that the regex
that is parsing the field that any user can send you via netcat is itself prone 
to a crash, or remote escape? Not me.