Logo

dev-resources.site

for different kinds of informations.

An Overview About the Different Kafka Connect Plugins

Published at
2/18/2021
Categories
apachekafka
kafkaconnect
Author
rueedlinger
Categories
2 categories in total
apachekafka
open
kafkaconnect
open
Author
11 person written this
rueedlinger
open
An Overview About the Different Kafka Connect Plugins

Kafka Connect is a great framework for connecting Kafka with external systems. In the best case you can use Connect right away. But in some special cases you might have to write your own plugins to add missing functionality to the framework. In this blog post I give a short overview about the different plugin types which can be used to add new functionality to Connect.

In the last part of this blog post I give you a short introduction to my GitHub project Ready, Steady, Connect - A Kafka Connect Quickstart. This project contains example Java code you can use to extend Connect with your own plugins.

The name of the project came from the blog post (Ready, Steady, Connect. Help Your Organization to Appreciate Kafka)I wrote about the experience we had with Connect.

Plugin Types

There are two main plugin categories that can be used to add new functionality to Kafka Connect:

  • Connect Plugins are part of the Connect API and can be used to extend the functionality of Connect.
  • Kafka Client Plugins are part of the Kafka Client (Consumer / Producer API). Kafka Connect is build on top of the Kafka Consumer / Producer API, so we have the possibility to write plugins which are part of these โ€œlower APIโ€™sโ€ and use them with Connect.

plugin types

Connect Plugins (Connect API)

Plugin Description API
Sink Connector A SinkConnector can load data from Kafka and store it into an external system (eg. database). Itโ€™s quite easy to write your own sink connector or take an existing open source version and modify it to your needs. SinkConnector, SinkTask
Source Connector A SourceConnector can load data from an external system and store it into Kafka. A source connector is a bit more complicated to write than a sink connector. But with some inspiration from other open source connectors this should not be to hard. SourceConnector, SourceTask
Single Message Transforms (SMTs) With a Transformation (SMT) you can transform Kafka messages when they are processed by a connector. For example you could write a SMT which appends a UUID to every message that passes trough. Transformation
Predicates A SMT can be configured with a Predicate (KIP-585). The SMT is only applied when the condition of the predicate was true. Predicate
Config Providers A ConfigProvider loads configuration values from external resources. These configuration values can then be referenced in the connector configuration. You could write a ConfigProvider which loads configuration values from a database, rest endpoint or from environment variables. ConfigProvider
Rest Extensions With a RestExtension (KIP-285) you can extend the existing Kafka Connect Rest API. You could write an authorization filter or liveness/readiness endpoints for k8s. ConnectRestExtension
Converter The Converter provides support for translating between Kafka Connectโ€™s runtime data format and the raw payload of the Kafka messages (JSON, Avro, โ€ฆ). Converter

Kafka Client Plugins (Kafka Producer / Consumer API)

Plugin Description API
Kafka Consumer Interceptor The ConsumerInterceptor (KIP-42) can be used to intercept Kafka messages before they are processed by the consumer. ConsumerInterceptor
Kafka Producer Interceptor The ProducerInterceptor (KIP-42) is a neat way to intercept Kafka messages before they are published to Kafka. ProducerInterceptor
Kafka Metrics Reporter The MetricsReporter can be used to listen to Kafka client metrics and process them. MetricsReporter

Create Your Own Connect Plugins

The Docker image and the source code for all plugin examples can be found in the Ready, Steady, Connect - A Kafka Connect Quickstart (rueedlinger/kafka-connect-quickstart) repository.

The first step is to clone the project.

git clone https://github.com/rueedlinger/kafka-connect-quickstart
Enter fullscreen mode Exit fullscreen mode

The main components of the project are the Docker Image , Java source code and the Docker Compose file.

  • The custom Connect Docker image has two parts.
    • The builder part to build the example plugins from the Java source code.
    • The main part to run the Kafka Connect container with all the Kafka Connect plugins.
  • The Java source code contains all the Kafka Connect plugin examples (connectors, transforms, etc.).
  • The Docker Compose file can be used to run the whole infrastructure (Kafka broker, zookeeper, etc).

The next step is to build all the plugins (Java) and start the containers with Docker.

docker-compose up --build
Enter fullscreen mode Exit fullscreen mode

When all containers are started you can access the following services:

Happy Coding

Now that everything is up and running. You can start to play around with Kafka Connect. I hope the kafka-connect-quickstart project is useful and gives you an easy start into the world of Kafka Connect plugins.

kafkaconnect Article's
30 articles in total
Favicon
Kafka Connect: FileStreamSourceConnector in distributed mode
Favicon
What is Kafka Connect?
Favicon
Publish PostgresSQL Data Changes to React with KsqlDB and MQTT
Favicon
Empowering Your Kafka Connectors: A Guide to Connector Guardian
Favicon
Running Debezium On Kubernetes
Favicon
Constant Lag in CDC Pipeline (JDBC Sink Connector)
Favicon
Kafka Connect sink to OpenSearch/ElasticSearch: how to sink unix timestamps
Favicon
Kafka 2 CockroachDB via JDBC Sink Connector Blueprint
Favicon
KSQL with authenticated kafka connect
Favicon
8 tips to speed up Apache Kafkaยฎ Connect development
Favicon
Showcasing Change Data Capture with Debezium and Kafka
Favicon
Use your own connector with Twitter and Aiven for Apache Kafkaยฎ
Favicon
Manage Apache Kafka Connect connectors with kcctl
Favicon
Loading CSV data into Confluent Cloud using the FilePulse connector
Favicon
Using Kafka Connect JDBC Source: a PostgreSQL example
Favicon
Kafka Connect JDBC Sink deep-dive: Working with Primary Keys
Favicon
Kafka Connect: The Magic Behind Mux Data Realtime Exports
Favicon
An Overview About the Different Kafka Connect Plugins
Favicon
Heroku Error - H10 App Crashed
Favicon
Apache Kafka Connect Usage Patterns
Favicon
Vinted Search Scaling Chapter 1: Indexing
Favicon
Running a self-managed Kafka Connect worker for Confluent Cloud
Favicon
Streaming data into Kafka S01/E04 โ€” Parsing log files using Grok Expressions
Favicon
Kafka Connect - Deep Dive into Single Message Transforms
Favicon
๐ŸŽ„ Twelve Days of SMT ๐ŸŽ„ - Day 12: Community Transformations
Favicon
๐ŸŽ„ Twelve Days of SMT ๐ŸŽ„ - Day 11: Predicate and Filter
Favicon
๐ŸŽ„ Twelve Days of SMT ๐ŸŽ„ - Day 10: ReplaceField
Favicon
๐ŸŽ„ Twelve Days of SMT ๐ŸŽ„ - Day 9: Cast
Favicon
๐ŸŽ„ Twelve Days of SMT ๐ŸŽ„ - Day 8: TimestampConverter
Favicon
๐ŸŽ„ Twelve Days of SMT ๐ŸŽ„ - Day 7: TimestampRouter

Featured ones: