Kafka/Confluent

Confluent the commercial/supported version of Kafka quick start

Starting up:

$ confluent start schema-registry
Starting zookeeper
zookeeper is [UP]
Starting kafka
kafka is [UP]
Starting schema-registry
schema-registry is [UP]

Piping in a file for testing:

kafka-console-producer.sh --broker-list localhost:9092 --topic my_topic < my_file.txt

Consuming a File -> Console and Elastic

With kafka/confluent running, start the console consumer:

./kafka-avro-console-consumer --zookeeper localhost:2181 --topic catalog --from-beginning

we are watching a topic called catalog

Now injest data into the catalog topic. The connect-file-source.properties properties has been setup as:

name=catalog
connector.class=FileStreamSource
tasks.max=1
file=/home/richard/devTools/confluent-4.0.0/testData/testData.json
topic=catalog
connect-file-source

And a simple script to copy data to the testData.json to slowly stream the data:

#! /usr/bin/python
import os
import sys
import time
 
 
i = 0
f = open("20180126022806-20180126022131-full-taxonomy.json","r")
copy = open("testData.json","wt")
#loop that copies file line by line and terminates loop when i reaches 10
for line in f: 
     copy.write(str(line))
     copy.flush()
     i = i +1
     sys.stdout.write('.')
     sys.stdout.flush()
     time.sleep(1)
f.close()
copy.close()

Now we can load the file source ..

./confluent load file-source

Logging

As well as the /logs files .. one can connect to the running log files with:

 confluent log connect

Making a Topic

this should create the catalog topic automatically. However if you want to manually make a topic:

./kafka-topics --create --zookeeper localhost:2181 --replication-factor 1   --partitions 1 --topic catalog

Kafka Plugins

 
kafka.txt · Last modified: 2018/01/29 01:30 by root
 
RSS - 200 © CrosswireDigitialMedia Ltd