Kafka Architecture


First, if you are not sure what Kafka is, see this article.

Kafka consists of records, topics, consumers, producers, brokers, logs, partitions, and clusters. Records can have keys (optional), values, and timestamps. Kafka records are immutable. A Kafka Topic is a stream of records ("/orders""/user-signups"). You can think of a topic as a feed name. A topic has a log which is the topic’s storage on disk. A topic log is broken up into partitions and segments. The Kafka Producer API is used to produce streams of data records. The Kafka Consumer API is used to consume a stream of records from Kafka. A broker is a Kafka server that runs in a Kafka cluster. Kafka brokers form a cluster. The Kafka cluster consists of many Kafka brokers on many servers. Broker some times refer to more of a logical system or as Kafka as a whole.

Kafka uses ZooKeeper to manage the cluster. ZooKeeper is used to coordinate the brokers/cluster topology. ZooKeeper is a consistent file system for configuration information. ZooKeeper is used for leadership election for broker topic partition leaders.

Read more at DZone