I’m using kafka-python library for my fastapi consumer app and I’m consuming messages in batch with maximum of 100 records. Since the topic has huge traffic and have only one partition, consuming, processing and committing should be as quick as possible hence I want to use commit_async(), instead of synchronous commit(). But I’m not able to find a good example
Tag: kafka-python
Python-Kafka: Keep polling topic infinitely
I am using python-kafka to listen to a kafka topic and use that the records. I want to keep it polling infinitely without any exit. This is my code below: This code just reads the data and exits directly. Is there a way to keep listening to topics even if message is not pushed to it? Any relevant example where
kafka-python raise UnrecognizedBrokerVersion Error
I am getting this error when constructing KafkaProducer with the kafka-python package: The code is as follows: I am using Python 3.7 and an AWS MSK cluster. Answer Solved it by just adding security_protocol=”SSL” to the KafkaProducer as follows:
Python: how to mock a kafka topic for unit tests?
We have a message scheduler that generates a hash-key from the message attributes before placing it on a Kafka topic queue with the key. This is done for de-duplication purposes. However, I am not sure how I could possibly test this deduplication without actually setting up a local cluster and checking that it is performing as expected. Searching online for