Sample Kafka Messaging From AWS VPC To Private Network

format

topic


Kafka Messaging From AWS VPC To Private Network

Sample Setup

Kafka is a messaging bus using TCP as its underpinnings. It may be desirable to run a Kafka broker inside a private cloud environment (e.g. an AWS VPC) and connect to that broker from a remote private network. Likewise it can be desirable to run a Kafka broker inside a remote private network and connect as a client from a remote private cloud such as AWS VPC. These instructions work equally for each case.

98208b64 messaging

Overview

Kafka is a messaging bus using TCP as its underpinnings. It may be desirable to run a Kafka broker inside a private cloud environment (e.g. an AWS VPC) and connect to that broker from a remote private network. Likewise it can be desirable to run a Kafka broker inside a remote private network and connect as a client from a remote private cloud such as AWS VPC. These instructions work equally for each case.

c3fde201 image

For this example, we will use two hosts. Cube exists inside a private network with a restrictive firewall. Only HTTPS/443 is allowed, and only in the outbound direction. On this host runs a Kafka broker on port 9092.

Office runs inside a Virtual Private Cloud. It two has outbound-only access: no inbound, no public IP.

For the sake of this example, we will use docker-compose to bring up the Broker on Cube. We will then use kafkacat to send and receive traffic via the broker.

Steps (Cube):

  1. docker-compose up
  2. apt-get install kafkacat
  3. Install Agilicus Connector
  4. Create a Network Service called kafka-cube, on port 9092, localhost (or the IP of the docker container if you prefer).
  5. Run kafkacat -C -b localhost:9092 -t test which will sit and listen for messages

Steps (Office):

  1. Install Agilicus Connector
  2. Create Service Forwarder, source-connector == Office, Destination-network == kafka-cube, source-ip localhost, source-port 9092
  3. Assign permissions to Service Forwarder to office connector service-account
  4. apt-get install kafkacat
  5. run echo foo | kafkacat -P -b localhost:9092 -t test

At this stage you should observe the kafkacat on cube wake up and say foo. We are done.

You may now wish to try e.g. change the server-forwarder to listen on 127.0.10.1:9092, THen we can run echo foo | kafkacat -P -b 127.0.10.1:9092 -t test. We can then add an entry to /etc/hosts and call it ‘Cube’ and repeat with a hostname.

a7d3bb5c cube network
3f7e7184 office forwarder
482b5ddb service account perm

docker-compose.yml

version: '2'
services:
  zookeeper:
    build: ./zookeeper
    ports:
      - "2181:2181"
  kafka:
    build: ./kafka
    ports:
      - "9092:9092"

zookeeper/Dockerfile

FROM ubuntu:22.04

EXPOSE 2181

RUN : \
 && apt-get update \
 && apt-get install -y wget vim net-tools default-jre curl

WORKDIR /opt

RUN : \
 && curl -sSL https://dlcdn.apache.org/zookeeper/zookeeper-3.8.0/apache-zookeeper-3.8.0-bin.tar.gz > /tmp/apache-zookeeper-3.8.0-bin.tar.gz \
 && tar xvzf /tmp/apache-zookeeper-3.8.0-bin.tar.gz \
 && rm -f /tmp/apa*gz \
 && cp apache-zookeeper-3.8.0-bin/conf/zoo_sample.cfg apache-zookeeper-3.8.0-bin/conf/zoo.cfg

ENTRYPOINT apache-zookeeper-3.8.0-bin/bin/zkServer.sh start-foreground

kafka/Dockerfile

FROM ubuntu:22.04

EXPOSE 9020

RUN : \
 && apt-get update \
 && apt-get install -y wget vim net-tools default-jre curl

WORKDIR /opt

RUN : \
 && curl -sSL https://downloads.apache.org/kafka/3.2.1/kafka_2.12-3.2.1.tgz > /tmp/kafka_2.12-3.2.1.tgz \
 && tar xvzf /tmp/kafka_2.12-3.2.1.tgz \
 && rm -f /tmp/*tgz

COPY server.properties /opt/kafka_2.12-3.2.1/config/server.properties

ENTRYPOINT /opt/kafka_2.12-3.2.1/bin/kafka-server-start.sh /opt/kafka_2.12-3.2.1/config/server.properties

kafka/server.properties:

broker.id=0
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/tmp/kafka-logs
num.partitions=1
num.recovery.threads.per.data.dir=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=zookeeper:2181
zookeeper.connection.timeout.ms=6000
group.initial.rebalance.delay.ms=0
offsets.topic.replication.factor=1
ssl.endpoint.identification.algorithm=

advertised.listeners=PLAINTEXT://localhost:9092
listeners=PLAINTEXT://0.0.0.0:9092

Want Assistance?

The Agilicus team is here for you. The ‘Chat‘ icon in the lower left, here, or in the administrative web page, goes to our team.

Or, feel free to email support@agilicus.com


(None)