Kafka Messaging From AWS VPC To Private Network

Overview

Kafka is a messaging bus using TCP as its underpinnings. It may be desirable to run a Kafka broker inside a private cloud environment (e.g. an AWS VPC) and connect to that broker from a remote private network. Likewise it can be desirable to run a Kafka broker inside a remote private network and connect as a client from a remote private cloud such as AWS VPC. These instructions work equally for each case.

For this example, we will use two hosts. Cube exists inside a private network with a restrictive firewall. Only HTTPS/443 is allowed, and only in the outbound direction. On this host runs a Kafka broker on port 9092.

Office runs inside a Virtual Private Cloud. It two has outbound-only access: no inbound, no public IP.

For the sake of this example, we will use docker-compose to bring up the Broker on Cube. We will then use kafkacat to send and receive traffic via the broker.

Steps (Cube):

1. docker-compose up
2. apt-get install kafkacat
3. Install Agilicus Connector
4. Create a Network Service called kafka-cube, on port 9092, localhost (or the IP of the docker container if you prefer).
5. Run kafkacat -C -b localhost:9092 -t test which will sit and listen for messages

Steps (Office):

1. Install Agilicus Connector
2. Create Service Forwarder, source-connector == Office, Destination-network == kafka-cube, source-ip localhost, source-port 9092
3. Assign permissions to Service Forwarder to office connector service-account
4. apt-get install kafkacat
5. run echo foo | kafkacat -P -b localhost:9092 -t test

At this stage you should observe the kafkacat on cube wake up and say foo. We are done.

You may now wish to try e.g. change the server-forwarder to listen on 127.0.10.1:9092, THen we can run echo foo | kafkacat -P -b 127.0.10.1:9092 -t test. We can then add an entry to /etc/hosts and call it ‘Cube’ and repeat with a hostname.

docker-compose.yml

version: '2'
services:
  zookeeper:
    build: ./zookeeper
    ports:
      - "2181:2181"
  kafka:
    build: ./kafka
    ports:
      - "9092:9092"

zookeeper/Dockerfile

FROM ubuntu:22.04

EXPOSE 2181

RUN : \
 && apt-get update \
 && apt-get install -y wget vim net-tools default-jre curl

WORKDIR /opt

RUN : \
 && curl -sSL https://dlcdn.apache.org/zookeeper/zookeeper-3.8.0/apache-zookeeper-3.8.0-bin.tar.gz > /tmp/apache-zookeeper-3.8.0-bin.tar.gz \
 && tar xvzf /tmp/apache-zookeeper-3.8.0-bin.tar.gz \
 && rm -f /tmp/apa*gz \
 && cp apache-zookeeper-3.8.0-bin/conf/zoo_sample.cfg apache-zookeeper-3.8.0-bin/conf/zoo.cfg

ENTRYPOINT apache-zookeeper-3.8.0-bin/bin/zkServer.sh start-foreground

kafka/Dockerfile

FROM ubuntu:22.04

EXPOSE 9020

RUN : \
 && apt-get update \
 && apt-get install -y wget vim net-tools default-jre curl

WORKDIR /opt

RUN : \
 && curl -sSL https://downloads.apache.org/kafka/3.2.1/kafka_2.12-3.2.1.tgz > /tmp/kafka_2.12-3.2.1.tgz \
 && tar xvzf /tmp/kafka_2.12-3.2.1.tgz \
 && rm -f /tmp/*tgz

COPY server.properties /opt/kafka_2.12-3.2.1/config/server.properties

ENTRYPOINT /opt/kafka_2.12-3.2.1/bin/kafka-server-start.sh /opt/kafka_2.12-3.2.1/config/server.properties

kafka/server.properties:

broker.id=0
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/tmp/kafka-logs
num.partitions=1
num.recovery.threads.per.data.dir=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=zookeeper:2181
zookeeper.connection.timeout.ms=6000
group.initial.rebalance.delay.ms=0
offsets.topic.replication.factor=1
ssl.endpoint.identification.algorithm=

advertised.listeners=PLAINTEXT://localhost:9092
listeners=PLAINTEXT://0.0.0.0:9092