In this article, we will walk through the steps to set up Apache Kafka on an Amazon EC2 instance (Amazon Linux distribution). We'll start with updating the system, installing Java, and then proceed with the installation and configuration of Kafka. By the end, you'll have a running Kafka instance, ready for producing and consuming messages.

Prerequisites

An Amazon EC2 instance (running Amazon Linux 2 or a similar distribution)
Basic knowledge of using the terminal

Steps

Step 1: Update the System

First, ensure your system is up to date by running:

sudo yum update -y

Step 2: Install Java

Apache Kafka requires Java to run. Install Java 11 using the following command:

sudo yum install java-11-amazon-corretto-devel -y

Step 3: Download and Extract Kafka

Next, download Kafka from the official Apache archive and extract it:

wget https://archive.apache.org/dist/kafka/3.3.1/kafka_2.12-3.3.1.tgz
tar -xzf kafka_2.12-3.3.1.tgz
cd kafka_2.12-3.3.1

Step 4: Start Zookeeper

Kafka requires Zookeeper to manage its cluster. Start Zookeeper with:

nohup bin/zookeeper-server-start.sh config/zookeeper.properties > zookeeper.log 2>&1 &

Step 5: Configure Kafka

Edit the Kafka configuration file to set the advertised listeners. Replace ec2-xx-xxx-xx-xxx.us-west-2.compute.amazonaws.com with your EC2 instance's public DNS:

vi config/server.properties

Add the following line:

advertised.listeners=PLAINTEXT://ec2-xx-xxx-xx-xxx.us-west-2.compute.amazonaws.com:9092

Step 6: Start Kafka Server

Now, start the Kafka server:

nohup bin/kafka-server-start.sh config/server.properties > kafka-server.log 2>&1 &

Step 7: Create a Topic

Create a new Kafka topic named test:

bin/kafka-topics.sh --create --topic test --bootstrap-server ec2-xx-xxx-xx-xxx.us-west-2.compute.amazonaws.com:9092 --partitions 1 --replication-factor 1

Step 8: Produce Messages

Start a Kafka producer to send messages to the test topic:

bin/kafka-console-producer.sh --topic test --bootstrap-server ec2-xx-xxx-xx-xxx.us-west-2.compute.amazonaws.com:9092

You can now type messages into the console, and they will be sent to the Kafka topic.

Step 9: Consume Messages

Start a Kafka consumer to read messages from the test topic:

bin/kafka-console-consumer.sh --topic test --bootstrap-server ec2-xx-xxx-xx-xxx.us-west-2.compute.amazonaws.com:9092 --from-beginning

You should see the messages you produced earlier displayed in the console.

Conclusion

By following these steps, you've successfully set up Apache Kafka on an Amazon EC2 instance. You can now produce and consume messages, enabling you to build real-time data streaming applications. This setup forms the foundation for more advanced Kafka configurations and use cases.

Feel free to explore Kafka's extensive features and tailor the configuration to suit your specific requirements. Happy streaming!