dev-resources.site

for different kinds of informations.

Kafka Connect: FileStreamSourceConnector in distributed mode

Published at

8/23/2024

Categories

kafka

kafkaconnect

filestreamsourceconnector

Author

eidher

Main Article

https://dev.to/eidher/kafka-connect-filestreamsourceconnector-in-distributed-mode-4l8

Categories

3 categories in total

filestreamsourceconnector

Author

6 person written this

Kafka Connect: FileStreamSourceConnector in distributed mode

1) Source connectors: Start our kafka cluster. This is the docker-compose.yml file:

version: '2'

services:

  kafka-cluster:
    image: landoop/fast-data-dev:cp3.3.0
    environment:
      ADV_HOST: 127.0.0.1
      RUNTESTS: 0
    ports:
      - 2181:2181                 # Zookeeper
      - 3030:3030                 # Landoop UI
      - 8081-8083:8081-8083       # REST Proxy, Schema Registry, Kafka Connect
      - 9581-9585:9581-9585       # JMX Ports
      - 9092:9092                 # Kafka Broker

Create and start the container:

docker-compose up kafka-cluster

2) Create the topic we're going to write to:

docker run --rm -it --net=host landoop/fast-data-dev:cp3.3.0 bash
kafka-topics --create --topic demo-2-distributed --partitions 3 --replication-factor 1 --zookeeper 127.0.0.1:2181

3) In a browser go to 127.0.0.1:3030 -> Connect UI
Create a new connector -> File Source
Paste the next configuration and click the Create button:

name=file-stream-demo-distributed
connector.class=org.apache.kafka.connect.file.FileStreamSourceConnector
tasks.max=1
file=demo-file.txt
topic=demo-distributed
key.converter=org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable=true
value.converter=org.apache.kafka.connect.json.JsonConverter
value.converter.schemas.enable=true

4) Create the source file:
Execute docker ps to see the containerId.

docker exec -it <containerId> bash
touch demo-file.txt
echo "add any content" >> demo-file.txt

3) In a browser go to 127.0.0.1:3030 -> Kafka Topics UI
Click the topic defined previously (demo-distributed) and validate the content:

kafkaconnect Article's

30 articles in total

Kafka Connect: FileStreamSourceConnector in distributed mode

currently reading

What is Kafka Connect?

Publish PostgresSQL Data Changes to React with KsqlDB and MQTT

Empowering Your Kafka Connectors: A Guide to Connector Guardian

Running Debezium On Kubernetes

Constant Lag in CDC Pipeline (JDBC Sink Connector)

Kafka Connect sink to OpenSearch/ElasticSearch: how to sink unix timestamps

Kafka 2 CockroachDB via JDBC Sink Connector Blueprint

KSQL with authenticated kafka connect

8 tips to speed up Apache Kafka® Connect development

Showcasing Change Data Capture with Debezium and Kafka

Use your own connector with Twitter and Aiven for Apache Kafka®

Manage Apache Kafka Connect connectors with kcctl

Loading CSV data into Confluent Cloud using the FilePulse connector

Using Kafka Connect JDBC Source: a PostgreSQL example

Kafka Connect JDBC Sink deep-dive: Working with Primary Keys

Kafka Connect: The Magic Behind Mux Data Realtime Exports

An Overview About the Different Kafka Connect Plugins

Heroku Error - H10 App Crashed

Apache Kafka Connect Usage Patterns

Vinted Search Scaling Chapter 1: Indexing

Running a self-managed Kafka Connect worker for Confluent Cloud

Streaming data into Kafka S01/E04 — Parsing log files using Grok Expressions

Kafka Connect - Deep Dive into Single Message Transforms

🎄 Twelve Days of SMT 🎄 - Day 12: Community Transformations

🎄 Twelve Days of SMT 🎄 - Day 11: Predicate and Filter

🎄 Twelve Days of SMT 🎄 - Day 10: ReplaceField

🎄 Twelve Days of SMT 🎄 - Day 9: Cast

🎄 Twelve Days of SMT 🎄 - Day 8: TimestampConverter

🎄 Twelve Days of SMT 🎄 - Day 7: TimestampRouter

Featured ones:

abubakersiddique761