Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
Last revision Both sides next revision
kafka [2019/09/06 09:47]
root [Dashboard]
kafka [2019/11/26 03:14]
root [Dashboard]
Line 1: Line 1:
 +Back to [[Home]] or [[Kafka]] ​
 +
 +== Overview == 
 +
 +Kafka is organized around a few key terms: topics, partitions, producers, consumers, and brokers.
 +* All Kafka messages are organized into **topics**. Messages are sent or received from topics
 +* A **consumer** pulls messages off of a Kafka topic while **producers** push messages into a Kafka topic. ​
 +Lastly, Kafka, as a distributed system, runs in a cluster. Each node in the cluster is called a Kafka **broker**.
 +
 +
 +Kafka topics are divided into a number of partitions. Partitions allow you to parallelize a topic by splitting the data in a particular topic across multiple brokers — each partition can be placed on a separate machine to allow for multiple consumers to read from a topic in parallel. ​
 +
 +Consumers can also be parallelized so that multiple consumers can read from multiple partitions in a topic allowing for very high message processing throughput.
 +
 +Each message within a partition has an identifier called its offset. The offset the ordering of messages as an immutable sequence. Kafka maintains this message ordering for you. Consumers can read messages starting from a specific offset and are allowed to read from any offset point they choose, allowing consumers to join the cluster at any point in time they see fit. Given these constraints,​ each specific message in a Kafka cluster can be uniquely identified by a tuple consisting of the message’s topic, partition, and offset within the partition.
 +
 +
 +
 == Kafka Setup and Concepts ==  == Kafka Setup and Concepts == 
  
Line 93: Line 111:
 * An alternate elastic https://​github.com/​Stratio/​kafka-elasticsearch-sink * An alternate elastic https://​github.com/​Stratio/​kafka-elasticsearch-sink
  
-== Dashboard == +== Dashboard ​and Managers ​== 
  
 * Confluent control center ​ * Confluent control center ​
 https://​www.confluent.io/​blog/​dawn-of-kafka-devops-managing-kafka-clusters-at-scale-with-confluent-control-center https://​www.confluent.io/​blog/​dawn-of-kafka-devops-managing-kafka-clusters-at-scale-with-confluent-control-center
 +* Kafka Manager created at Yahoo https://​github.com/​yahoo/​kafka-manager
 +
  
  
Line 105: Line 125:
 == KQL == == KQL ==
 Kafka Window SQL Kafka Window SQL
 +
 +== Kafka Consumer Groups ==
 +
 +allow monitoring of micro-service health and workload distribution.
 +* Different instances of a micro-service are placed in consumer group. ​
 +* Kafka will automatically distribute the workload, provided that the customers are well balanced across the partitions.
 +* Same mechanism used for 24/7 with rolling upgrade.
  
 == Kafka (Cloud) Alternatives ==  == Kafka (Cloud) Alternatives == 
Line 115: Line 142:
 == Links and Reference == == Links and Reference ==
 * [[https://​www.slideshare.net/​ConfluentInc/​connecting-kafka-across-multiple-aws-vpcs?​ref=https://​www.confluent.io/​kafka-summit-sf18/​connecting-kafka-across-multiple-aws-vpcs ​ | Kafka Across Multiple AWS VPCs]] * [[https://​www.slideshare.net/​ConfluentInc/​connecting-kafka-across-multiple-aws-vpcs?​ref=https://​www.confluent.io/​kafka-summit-sf18/​connecting-kafka-across-multiple-aws-vpcs ​ | Kafka Across Multiple AWS VPCs]]
 +* [[https://​sookocheff.com/​post/​kafka/​kafka-in-a-nutshell/​ | Kafka In a Nutshell]] ​
 
kafka.txt · Last modified: 2019/11/26 06:41 by root
 
RSS - 200 © CrosswireDigitialMedia Ltd