Forum: Building VoltDB Clients

Post: how we process the data from kafkaloader through procedure

how we process the data from kafkaloader through procedure
rsureshrajah
Apr 30, 2015
Hi,

I want to process the data which comes from kafkaloader i.e in realtime. how will i do that??

i saw that in kafkaloader i have a option of procedure which can process the data which comes to kafka one by one.
how should the procedure be? can you get me some examples for it.
In the UsingVoltdb Pdf i did not find any example for a procedure which can be used in kafkaloader.

Kindly help as it is very urgent.

Thanks
Sureshrajah. R
nshi
Apr 30, 2015
Hi Sureshrajah,

The kafkaloader expects each Kafka message to contain a CSV string. It will decode the CSV string and pass the decoded values as parameters to the procedure you specified. Here's an example of using the kafkaloader with the default insert procedure.

http://voltdb.com/kafka-voltdb

Hope it helps.
madhab_p
Feb 14, 2017
Hi,

The link
http://voltdb.com/kafka-voltdb opens up the link https://www.voltdb.com/products/kafka-importer-and-export . The later gives a general idea only. Can you share a sample procedure used with kafkaloader
bballard
Feb 14, 2017
You can use any procedure with kafkaloader or kafka importer as long as the input parameters of the procedure correspond exactly with the columns in the CSV or TSV data coming from the kafka topic.

For example, suppose you have a table FOO:

CREATE TABLE foo (
id BIGINT NOT NULL,
col1 VARCHAR(100),
col2 VARCHAR(100)
);
PARTITION TABLE foo ON COLUMN id;

There is an auto-generated procedure FOO.insert that you can call to insert into this table. You can configure KafkaLoader or KafkaImporter to call this procedure so long as the format of the topic matches the input parameters, which are the columns of the table in the order they are defined within the DDL. For example:

1, "Hello", "World"
2, "Something", "Something else"

You could also write a java stored procedure, following instructions in Using VoltDB Ch. 5 (https://docs.voltdb.com/UsingVoltDB/DesignProc.php), if you wanted to add additional validation or reformatting of values, or call multiple SQL statements.