# Sample Kafka Messaging From AWS VPC To Private Network

- Link: https://www.agilicus.com/anyx-guide/sample-kafka-messaging-from-aws-vpc-to-private-network/
- Published: 2024-05-14T00:25:09+00:00
- Author: Don Bowman

1. [Home](https://www.agilicus.com/)
 2. Sample Kafka Messaging From AWS VPC To Private Network

![98208b64-messaging](https://www.agilicus.com/www/98208b64-messaging.svg)

Sample Setup

---

## Kafka Messaging From AWS VPC To Private Network

Kafka is a messaging bus using TCP as its underpinnings. It may be desirable to 
run a Kafka broker inside a private cloud environment (e.g. an AWS VPC) and connect
to that broker from a remote private network. Likewise it can be desirable to run
a Kafka broker inside a remote private network and connect as a client from a remote
private cloud such as AWS VPC. These instructions work equally for each case.

[CONTACT](https://www.agilicus.com/contact-us/)

## Overview

Kafka is a messaging bus using TCP as its underpinnings. It may be desirable to 
run a Kafka broker inside a private cloud environment (e.g. an AWS VPC) and connect
to that broker from a remote private network. Likewise it can be desirable to run
a Kafka broker inside a remote private network and connect as a client from a remote
private cloud such as AWS VPC. These instructions work equally for each case.

[⌊c3fde201-image⌉⌊c3fde201-image⌉[

For this example, we will use two hosts. **Cube** exists inside a private network
with a restrictive firewall. Only HTTPS/443 is allowed, and only in the outbound
direction. On this host runs a Kafka broker on port 9092.

**Office** runs inside a Virtual Private Cloud. It two has outbound-only access:
no inbound, no public IP.

For the sake of this example, we will use docker-compose to bring up the Broker 
on Cube. We will then use `kafkacat` to send and receive traffic via the broker.

Steps (Cube):

 1. docker-compose up
 2. apt-get install kafkacat
 3. [Install](https://www.agilicus.com/anyx-guide/agilicus-connector/) Agilicus Connector
 4. Create a Network Service called `kafka-cube`, on port 9092, localhost (or the IP
    of the docker container if you prefer).
 5. Run `kafkacat -C -b localhost:9092 -t test` which will sit and listen for messages

Steps (Office):

 1. Install Agilicus Connector
 2. Create Service Forwarder, source-connector == Office, Destination-network == kafka-
    cube, source-ip localhost, source-port 9092
 3. Assign permissions to Service Forwarder to office connector service-account
 4. apt-get install kafkacat
 5. run `echo foo | kafkacat -P -b localhost:9092 -t test`

At this stage you should observe the kafkacat on `cube` wake up and say `foo`. We
are done.

You may now wish to try e.g. change the server-forwarder to listen on 127.0.10.1:
9092, THen we can run `echo foo | kafkacat -P -b 127.0.10.1:9092 -t test`. We can
then add an entry to /etc/hosts and call it ‘Cube’ and repeat with a hostname.

[⌊a7d3bb5c-cube-network⌉⌊a7d3bb5c-cube-network⌉[

[⌊3f7e7184-office-forwarder⌉⌊3f7e7184-office-forwarder⌉[

[⌊482b5ddb-service-account-perm⌉⌊482b5ddb-service-account-perm⌉[

**docker-compose.yml**

    ```language-docker
    version: '2'
    services:
      zookeeper:
        build: ./zookeeper
        ports:
          - "2181:2181"
      kafka:
        build: ./kafka
        ports:
          - "9092:9092"
    ```

**zookeeper/Dockerfile**

    ```language-docker
    FROM ubuntu:22.04

    EXPOSE 2181

    RUN : \
     && apt-get update \
     && apt-get install -y wget vim net-tools default-jre curl

    WORKDIR /opt

    RUN : \
     && curl -sSL https://dlcdn.apache.org/zookeeper/zookeeper-3.8.0/apache-zookeeper-3.8.0-bin.tar.gz > /tmp/apache-zookeeper-3.8.0-bin.tar.gz \
     && tar xvzf /tmp/apache-zookeeper-3.8.0-bin.tar.gz \
     && rm -f /tmp/apa*gz \
     && cp apache-zookeeper-3.8.0-bin/conf/zoo_sample.cfg apache-zookeeper-3.8.0-bin/conf/zoo.cfg

    ENTRYPOINT apache-zookeeper-3.8.0-bin/bin/zkServer.sh start-foreground
    ```

**kafka/Dockerfile**

    ```language-docker
    FROM ubuntu:22.04

    EXPOSE 9020

    RUN : \
     && apt-get update \
     && apt-get install -y wget vim net-tools default-jre curl

    WORKDIR /opt

    RUN : \
     && curl -sSL https://downloads.apache.org/kafka/3.2.1/kafka_2.12-3.2.1.tgz > /tmp/kafka_2.12-3.2.1.tgz \
     && tar xvzf /tmp/kafka_2.12-3.2.1.tgz \
     && rm -f /tmp/*tgz

    COPY server.properties /opt/kafka_2.12-3.2.1/config/server.properties

    ENTRYPOINT /opt/kafka_2.12-3.2.1/bin/kafka-server-start.sh /opt/kafka_2.12-3.2.1/config/server.properties
    ```

**kafka/server.properties:**

    ```language-java
    broker.id=0
    num.network.threads=3
    num.io.threads=8
    socket.send.buffer.bytes=102400
    socket.receive.buffer.bytes=102400
    socket.request.max.bytes=104857600
    log.dirs=/tmp/kafka-logs
    num.partitions=1
    num.recovery.threads.per.data.dir=1
    transaction.state.log.replication.factor=1
    transaction.state.log.min.isr=1
    log.retention.hours=168
    log.segment.bytes=1073741824
    log.retention.check.interval.ms=300000
    zookeeper.connect=zookeeper:2181
    zookeeper.connection.timeout.ms=6000
    group.initial.rebalance.delay.ms=0
    offsets.topic.replication.factor=1
    ssl.endpoint.identification.algorithm=

    advertised.listeners=PLAINTEXT://localhost:9092
    listeners=PLAINTEXT://0.0.0.0:9092
    ```