Writing Kafka Producers using Java

Producers are processes that publish messages to Kafka.

Writing Producers basic concepts

• Producers can be implemented in different languages such as Java, C, and Python; we will however only look into Java for now. 

• The producer connects to any of the live nodes, requests metadata about the leaders for the partitions of a topic and then the producers can put the message directly to the lead broker.

• The producer can control which partition it publishes messages to. The producer can specify a key to partition; if customer ID is selected as a key, then all data for a given customer will be sent to the same partition.

• The Kafka producer configuration event.handler also provides the ability to define custom event handlers.

Publishing the messages in batches

Producers can also publish the messages in batches. 

• However publishing the messages in batches work in asynchronous mode only.

• The producer works either with a fixed number of messages or fixed latency defined by producer configuration, queue.time or batch.size, respectively.

• Data is accumulated in memory at the producer’s end and published in batches in a single request.

• Asynchronous mode also has the risk of losing the data in the case of a producer crash with accumulated non-published, in-memory data.

The Java producer API

kafka.javaapi.producer.Producer<K, V>

• for creating messages for single or multiple topics

• message partition is an optional feature.

• The default message partitioner is based on the hash of the key.

• K and V specify the types for the partition key and message value, respectively.

kafka.producer.KeyedMessage<K,V>

• takes the topic name, partition key, and the message value that need to be passed from the producer.

• K and V specify the type for the partition key and message value, respectively, and the topic is of type String.

kafka.producer.ProducerConfig

• encapsulates the values required for establishing the connection with the brokers such as

  • the broker list,

  • message partition class,

  • serializer class for the message, and

  • partition key.