Kafka consumer/producer with Python in WSL2 Ubuntu

This is a follow-up question from this thread.

As per advised from the thread, the possible cause for why my Python code is not working is because I was to connect to a remote server in WSL2. And there could be unknown issues with WSL2 Ubuntu.

So I am testing that hypothesis with the following two approaches of communicating within WLS2 Ubuntu locally (i.e. via localhost:9092):

Note that, for both approaches below, I already have zookeeper running in one terminal (T1) with:

bin/zookeeper-server-start.sh config/zookeeper.properties

Approach 1: communicating with producer/consumer in console as described by the following commands (taken from this tutorial)

Step 1: create a topic TutorialTopic (in terminal T2)

~/kafka/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic TutorialTopic

Step 2: produce a message in a terminal T3

echo "Hello, World" | ~/kafka/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic TutorialTopic

Step 3: consume the message in another terminal T4

~/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic TutorialTopic --from-beginning

Outcome: I see a message Hello, World in terminal T4

Approach 2: communicate through two Python modules, consumer.py and producer.py as in this tutorial:

Step 1: create a topic sample in terminal T5

bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic sample

Step 2: run the producer module in terminal T6

producer.py

from kafka import KafkaProducer

producer = KafkaProducer(bootstrap_servers='localhost:9092')
producer.send('sample', value='Hello, World!')

Step 3: consume that message in terminal T7

consumer.py

from kafka import KafkaConsumer

consumer = KafkaConsumer('sample', bootstrap_servers=['localhost: 9092'])

for message in consumer:
    print (message.value)

Outcome: Nothing is displayed on T7. The Python console is stuck in the run unless I execute Ctrl+C. No other errors or messages are produceds. Unlike the previous thread where I get No Broker error.

However, within this Approach 2, if I produce the message through a command in T6 as following, I surprisingly receive it in the consumer terminal T7:

echo "Hello" | ~/kafka/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic sample

My ultimate goal is to consume the messages by Python app from a remote server and work with WSL2 Ubuntu. This experiment seems to imply WSL2 is not the issue. If someone could enlighten as to what is happening here.

Answer

produce the message through a command … I surprisingly receive it in the consumer terminal T7

No surprise here since you’ve not called producer.flush() or producer.close() in your Python producer app after starting the consumer loop.

The console producer blocks on every record by calling get() on the future – source, effectively flushing its buffer

Alternatively, you are missing the matching option for --from-beginning in the Python consumer if you wanted to see the previously sent records

Ultimately, testing a local client/server within the same network adapter/subnet isn’t going to help resolve an external network connection

Advertisement

Answer