dev-resources.site
for different kinds of informations.
Constant Lag in CDC Pipeline (JDBC Sink Connector)
I am running a CDC Pipeline (data streaming) with source DB as PostgreSQL (Debezium Source Connector) and target DB also PostgreSQL with JDBC Sink Connector reading data from Kafka. I am experiencing a constant lag around 40 messages. And publishing rate is 5 messages per second. I also a another JDBC Sink Connector running to a different PostgreSQL DB but lag is around 10 messages.
Even though i tried to modify the following properties still lag didn't reduced:
max.poll.records=5000
max.poll.interval.ms=300000
fetch.min.bytes=100000
fetch.max.wait.ms=500
fetch.min.bytes=100000
consumer.override.auto.commit.interval.ms=100
consumer.override.offset.flush.interval.ms=200
batch.size=4000
How do i reduce the lag, so that pipeline works in real time or near real time?
Featured ones: