Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
kafka [2019/09/06 09:49]
root
kafka [2019/11/26 06:41] (current)
root [Kafka Setup and Concepts]
Line 1: Line 1:
 Back to [[Home]] or [[Kafka]] ​ Back to [[Home]] or [[Kafka]] ​
 +
 +== Overview == 
 +
 +Kafka is organized around a few key terms: topics, partitions, producers, consumers, and brokers.
 +* All Kafka messages are organized into **topics**. Messages are sent or received from topics
 +* A **consumer** pulls messages off of a Kafka topic while **producers** push messages into a Kafka topic. ​
 +Lastly, Kafka, as a distributed system, runs in a cluster. Each node in the cluster is called a Kafka **broker**.
 +
 +
 +Kafka topics are divided into a number of partitions. Partitions allow you to parallelize a topic by splitting the data in a particular topic across multiple brokers — each partition can be placed on a separate machine to allow for multiple consumers to read from a topic in parallel. ​
 +
 +Consumers can also be parallelized so that multiple consumers can read from multiple partitions in a topic allowing for very high message processing throughput.
 +
 +Each message within a partition has an identifier called its offset. The offset the ordering of messages as an immutable sequence. Kafka maintains this message ordering for you. Consumers can read messages starting from a specific offset and are allowed to read from any offset point they choose, allowing consumers to join the cluster at any point in time they see fit. Given these constraints,​ each specific message in a Kafka cluster can be uniquely identified by a tuple consisting of the message’s topic, partition, and offset within the partition.
 +
 +
  
 == Kafka Setup and Concepts ==  == Kafka Setup and Concepts == 
Line 6: Line 22:
 * --partitions 1 * --partitions 1
  
 +* Kafka Connect cluster (which is a cluster of workers) is completely different from the  
 +* Kafka cluster (which is a cluster of Kafka brokers).
 == Kafka/​Confluent ==  == Kafka/​Confluent == 
  
Line 95: Line 112:
 * An alternate elastic https://​github.com/​Stratio/​kafka-elasticsearch-sink * An alternate elastic https://​github.com/​Stratio/​kafka-elasticsearch-sink
  
-== Dashboard == +== Dashboard ​and Managers ​== 
  
 * Confluent control center ​ * Confluent control center ​
 https://​www.confluent.io/​blog/​dawn-of-kafka-devops-managing-kafka-clusters-at-scale-with-confluent-control-center https://​www.confluent.io/​blog/​dawn-of-kafka-devops-managing-kafka-clusters-at-scale-with-confluent-control-center
 +* Kafka Manager created at Yahoo https://​github.com/​yahoo/​kafka-manager
 +
  
  
Line 107: Line 126:
 == KQL == == KQL ==
 Kafka Window SQL Kafka Window SQL
 +
 +== Kafka Consumer Groups ==
 +
 +allow monitoring of micro-service health and workload distribution.
 +* Different instances of a micro-service are placed in consumer group. ​
 +* Kafka will automatically distribute the workload, provided that the customers are well balanced across the partitions.
 +* Same mechanism used for 24/7 with rolling upgrade.
  
 == Kafka (Cloud) Alternatives ==  == Kafka (Cloud) Alternatives == 
Line 117: Line 143:
 == Links and Reference == == Links and Reference ==
 * [[https://​www.slideshare.net/​ConfluentInc/​connecting-kafka-across-multiple-aws-vpcs?​ref=https://​www.confluent.io/​kafka-summit-sf18/​connecting-kafka-across-multiple-aws-vpcs ​ | Kafka Across Multiple AWS VPCs]] * [[https://​www.slideshare.net/​ConfluentInc/​connecting-kafka-across-multiple-aws-vpcs?​ref=https://​www.confluent.io/​kafka-summit-sf18/​connecting-kafka-across-multiple-aws-vpcs ​ | Kafka Across Multiple AWS VPCs]]
 +* [[https://​sookocheff.com/​post/​kafka/​kafka-in-a-nutshell/​ | Kafka In a Nutshell]] ​
 
kafka.1567788546.txt.gz · Last modified: 2019/09/06 09:49 by root
 
RSS - 200 © CrosswireDigitialMedia Ltd