# DoS’ing the cloud with logs

- Link: https://www.agilicus.com/dosing-the-cloud-with-logs/
- Published: 2019-04-01T04:16:40+00:00
- Author: Don Bowman

A few years ago an NTP issue came to light that caused a lot of damage. Cloudflare
did a good [writeup](https://blog.cloudflare.com/technical-details-behind-a-400gbps-ntp-amplification-ddos-attack/)
on this if you want to see the details. But in a nutshell, if there is a request
which can be sent which causes a larger response, you have amplification. In the
NTP case, a small packet requesting the peer list would get a much larger response.
Coupled with being UDP and a lot of providers not implementing [BCP38](http://www.bcp38.info/index.php/Main_Page),
meaning I can generate a request from your IP, and the large response goes to you,
and you have a problem.

OK what does this have to do with cloud you ask? Well, lets look at logging. A lot
of people use ‘logging as a service’ (stackdriver, elasticsearch, …). It can be 
managed by your cloud provider (Google, Microsoft, …), or by a 3rd party. But, well,
you log a lot of stuff in (moderatly) abnormal circumstances.

Now lets look at a particular tool. ‘[Tranquility](https://github.com/druid-io/tranquility)‘(
this observation seems true for nearly any Java program as far as I have observed).
When something happens, it logs a stack trace. As an end-user? useless. Lets look
at one below. When someone connects to my service without knowing my credentials,
they eventually time out and leave this bomb. Its 9747 bytes. From a single connect.
This is in fact worse than the above NTP issue.

But it gets worse. Each of these lines is bundled up into JSON, with a timestamp,
some fields around originating host, etc. It turns out by the time this lands on
the wire on the way out to my log provider its more like 20KB. That’s right, a hundred
bytes or so of inbound SYN and ACK cause this. Now I know what you are thinking,
what endpoint of Don’s can I try this on? Well its ‘[http://127.0.0.1](http://kittenwar.com/)‘.
Go ahead.

How long before someone finds and attacks this vector? Hmm.

    ```wp-block-preformatted
    2019-02-03 22:08:00,959 [qtp1247938090-34] WARN  c.m.t.server.http.TranquilityServlet - Server error serving request to http://tranquility.druid.agilicus.ca:443/v1/post/logs
    java.lang.InterruptedException: null
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1039) ~[na:1.8.0_181]
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328) ~[na:1.8.0_181]
        at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277) ~[na:1.8.0_181]
        at com.twitter.util.Promise.ready(Promise.scala:667) ~[com.twitter.util-core_2.11-6.42.0.jar:6.42.0]
        at com.twitter.util.Promise.result(Promise.scala:673) ~[com.twitter.util-core_2.11-6.42.0.jar:6.42.0]
        at com.twitter.util.Await$$anonfun$result$1.apply(Awaitable.scala:151) ~[com.twitter.util-core_2.11-6.42.0.jar:6.42.0]
        at com.twitter.concurrent.LocalScheduler$Activation.blocking(Scheduler.scala:220) ~[com.twitter.util-core_2.11-6.42.0.jar:6.42.0]
        at com.twitter.concurrent.LocalScheduler.blocking(Scheduler.scala:285) ~[com.twitter.util-core_2.11-6.42.0.jar:6.42.0]
        at com.twitter.concurrent.Scheduler$.blocking(Scheduler.scala:115) ~[com.twitter.util-core_2.11-6.42.0.jar:6.42.0]
        at com.twitter.util.Await$.result(Awaitable.scala:151) ~[com.twitter.util-core_2.11-6.42.0.jar:6.42.0]
        at com.twitter.util.Await$.result(Awaitable.scala:140) ~[com.twitter.util-core_2.11-6.42.0.jar:6.42.0]
        at com.metamx.tranquility.tranquilizer.Tranquilizer.flush(Tranquilizer.scala:243) ~[io.druid.tranquility-core-0.8.3.jar:0.8.3]
        at com.metamx.tranquility.server.http.TranquilityServlet$$anonfun$doSend$3.apply(TranquilityServlet.scala:204) ~[io.druid.tranquility-server-0.8.3.jar:0.8.3]
        at com.metamx.tranquility.server.http.TranquilityServlet$$anonfun$doSend$3.apply(TranquilityServlet.scala:204) ~[io.druid.tranquility-server-0.8.3.jar:0.8.3]
        at scala.collection.mutable.HashMap$$anon$2$$anonfun$foreach$3.apply(HashMap.scala:108) ~[org.scala-lang.scala-library-2.11.8.jar:na]
        at scala.collection.mutable.HashMap$$anon$2$$anonfun$foreach$3.apply(HashMap.scala:108) ~[org.scala-lang.scala-library-2.11.8.jar:na]
        at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230) ~[org.scala-lang.scala-library-2.11.8.jar:na]
        at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40) ~[org.scala-lang.scala-library-2.11.8.jar:na]
        at scala.collection.mutable.HashMap$$anon$2.foreach(HashMap.scala:108) ~[org.scala-lang.scala-library-2.11.8.jar:na]
        at com.metamx.tranquility.server.http.TranquilityServlet.doSend(TranquilityServlet.scala:204) ~[io.druid.tranquility-server-0.8.3.jar:0.8.3]
        at com.metamx.tranquility.server.http.TranquilityServlet.com$metamx$tranquility$server$http$TranquilityServlet$$doV1Post(TranquilityServlet.scala:141) ~[io.druid.tranquility-server-0.8.3.jar:0.8.3]
        at com.metamx.tranquility.server.http.TranquilityServlet$$anonfun$4.apply(TranquilityServlet.scala:87) ~[io.druid.tranquility-server-0.8.3.jar:0.8.3]
        at com.metamx.tranquility.server.http.TranquilityServlet$$anonfun$4.apply(TranquilityServlet.scala:85) ~[io.druid.tranquility-server-0.8.3.jar:0.8.3]
        at org.scalatra.ScalatraBase$class.org$scalatra$ScalatraBase$$liftAction(ScalatraBase.scala:270) [org.scalatra.scalatra_2.11-2.3.1.jar:2.3.1]
        at org.scalatra.ScalatraBase$$anonfun$invoke$1.apply(ScalatraBase.scala:265) ~[org.scalatra.scalatra_2.11-2.3.1.jar:2.3.1]
        at org.scalatra.ScalatraBase$$anonfun$invoke$1.apply(ScalatraBase.scala:265) ~[org.scalatra.scalatra_2.11-2.3.1.jar:2.3.1]
        at org.scalatra.ScalatraBase$class.withRouteMultiParams(ScalatraBase.scala:341) [org.scalatra.scalatra_2.11-2.3.1.jar:2.3.1]
        at org.scalatra.ScalatraServlet.withRouteMultiParams(ScalatraServlet.scala:49) [org.scalatra.scalatra_2.11-2.3.1.jar:2.3.1]
        at org.scalatra.ScalatraBase$class.invoke(ScalatraBase.scala:264) [org.scalatra.scalatra_2.11-2.3.1.jar:2.3.1]
        at org.scalatra.ScalatraServlet.invoke(ScalatraServlet.scala:49) [org.scalatra.scalatra_2.11-2.3.1.jar:2.3.1]
        at org.scalatra.ScalatraBase$$anonfun$runRoutes$1$$anonfun$apply$8.apply(ScalatraBase.scala:240) ~[org.scalatra.scalatra_2.11-2.3.1.jar:2.3.1]
        at org.scalatra.ScalatraBase$$anonfun$runRoutes$1$$anonfun$apply$8.apply(ScalatraBase.scala:238) ~[org.scalatra.scalatra_2.11-2.3.1.jar:2.3.1]
        at scala.Option.flatMap(Option.scala:171) ~[org.scala-lang.scala-library-2.11.8.jar:na]
        at org.scalatra.ScalatraBase$$anonfun$runRoutes$1.apply(ScalatraBase.scala:238) ~[org.scalatra.scalatra_2.11-2.3.1.jar:2.3.1]
        at org.scalatra.ScalatraBase$$anonfun$runRoutes$1.apply(ScalatraBase.scala:237) ~[org.scalatra.scalatra_2.11-2.3.1.jar:2.3.1]
        at scala.collection.immutable.Stream.flatMap(Stream.scala:489) ~[org.scala-lang.scala-library-2.11.8.jar:na]
        at org.scalatra.ScalatraBase$class.runRoutes(ScalatraBase.scala:237) [org.scalatra.scalatra_2.11-2.3.1.jar:2.3.1]
        at org.scalatra.ScalatraServlet.runRoutes(ScalatraServlet.scala:49) [org.scalatra.scalatra_2.11-2.3.1.jar:2.3.1]
        at org.scalatra.ScalatraBase$class.runActions$1(ScalatraBase.scala:163) [org.scalatra.scalatra_2.11-2.3.1.jar:2.3.1]
        at org.scalatra.ScalatraBase$$anonfun$executeRoutes$1.apply$mcV$sp(ScalatraBase.scala:175) ~[org.scalatra.scalatra_2.11-2.3.1.jar:2.3.1]
        at org.scalatra.ScalatraBase$$anonfun$executeRoutes$1.apply(ScalatraBase.scala:175) ~[org.scalatra.scalatra_2.11-2.3.1.jar:2.3.1]
        at org.scalatra.ScalatraBase$$anonfun$executeRoutes$1.apply(ScalatraBase.scala:175) ~[org.scalatra.scalatra_2.11-2.3.1.jar:2.3.1]
        at org.scalatra.ScalatraBase$class.org$scalatra$ScalatraBase$$cradleHalt(ScalatraBase.scala:193) [org.scalatra.scalatra_2.11-2.3.1.jar:2.3.1]
        at org.scalatra.ScalatraBase$class.executeRoutes(ScalatraBase.scala:175) [org.scalatra.scalatra_2.11-2.3.1.jar:2.3.1]
        at org.scalatra.ScalatraServlet.executeRoutes(ScalatraServlet.scala:49) [org.scalatra.scalatra_2.11-2.3.1.jar:2.3.1]
        at org.scalatra.ScalatraBase$$anonfun$handle$1.apply$mcV$sp(ScalatraBase.scala:113) [org.scalatra.scalatra_2.11-2.3.1.jar:2.3.1]
        at org.scalatra.ScalatraBase$$anonfun$handle$1.apply(ScalatraBase.scala:113) [org.scalatra.scalatra_2.11-2.3.1.jar:2.3.1]
        at org.scalatra.ScalatraBase$$anonfun$handle$1.apply(ScalatraBase.scala:113) [org.scalatra.scalatra_2.11-2.3.1.jar:2.3.1]
        at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58) [org.scala-lang.scala-library-2.11.8.jar:na]
        at org.scalatra.DynamicScope$class.withResponse(DynamicScope.scala:80) [org.scalatra.scalatra_2.11-2.3.1.jar:2.3.1]
        at org.scalatra.ScalatraServlet.withResponse(ScalatraServlet.scala:49) [org.scalatra.scalatra_2.11-2.3.1.jar:2.3.1]
        at org.scalatra.DynamicScope$$anonfun$withRequestResponse$1.apply(DynamicScope.scala:60) [org.scalatra.scalatra_2.11-2.3.1.jar:2.3.1]
        at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58) [org.scala-lang.scala-library-2.11.8.jar:na]
        at org.scalatra.DynamicScope$class.withRequest(DynamicScope.scala:71) [org.scalatra.scalatra_2.11-2.3.1.jar:2.3.1]
        at org.scalatra.ScalatraServlet.withRequest(ScalatraServlet.scala:49) [org.scalatra.scalatra_2.11-2.3.1.jar:2.3.1]
        at org.scalatra.DynamicScope$class.withRequestResponse(DynamicScope.scala:59) [org.scalatra.scalatra_2.11-2.3.1.jar:2.3.1]
        at org.scalatra.ScalatraServlet.withRequestResponse(ScalatraServlet.scala:49) [org.scalatra.scalatra_2.11-2.3.1.jar:2.3.1]
        at org.scalatra.ScalatraBase$class.handle(ScalatraBase.scala:111) [org.scalatra.scalatra_2.11-2.3.1.jar:2.3.1]
        at org.scalatra.ScalatraServlet.org$scalatra$servlet$ServletBase$$super$handle(ScalatraServlet.scala:49) [org.scalatra.scalatra_2.11-2.3.1.jar:2.3.1]
        at org.scalatra.servlet.ServletBase$class.handle(ServletBase.scala:43) [org.scalatra.scalatra_2.11-2.3.1.jar:2.3.1]
        at org.scalatra.ScalatraServlet.handle(ScalatraServlet.scala:49) [org.scalatra.scalatra_2.11-2.3.1.jar:2.3.1]
        at org.scalatra.ScalatraServlet.service(ScalatraServlet.scala:54) [org.scalatra.scalatra_2.11-2.3.1.jar:2.3.1]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [javax.servlet.javax.servlet-api-3.1.0.jar:3.1.0]
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:800) [org.eclipse.jetty.jetty-servlet-9.2.5.v20141112.jar:9.2.5.v20141112]
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:587) [org.eclipse.jetty.jetty-servlet-9.2.5.v20141112.jar:9.2.5.v20141112]
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:517) [org.eclipse.jetty.jetty-servlet-9.2.5.v20141112.jar:9.2.5.v20141112]
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [org.eclipse.jetty.jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [org.eclipse.jetty.jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]
        at org.eclipse.jetty.server.Server.handle(Server.java:497) [org.eclipse.jetty.jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310) [org.eclipse.jetty.jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:248) [org.eclipse.jetty.jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]
        at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540) [org.eclipse.jetty.jetty-io-9.2.5.v20141112.jar:9.2.5.v20141112]
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:620) [org.eclipse.jetty.jetty-util-9.2.5.v20141112.jar:9.2.5.v20141112]
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:540) [org.eclipse.jetty.jetty-util-9.2.5.v20141112.jar:9.2.5.v20141112]
        at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181]
    ```