Results 1 to 4 of 4

Thread: how we process the data from kafkaloader through procedure

  1. #1
    New Member
    Join Date
    Apr 2015
    Posts
    1

    Post how we process the data from kafkaloader through procedure

    Hi,

    I want to process the data which comes from kafkaloader i.e in realtime. how will i do that??

    i saw that in kafkaloader i have a option of procedure which can process the data which comes to kafka one by one.
    how should the procedure be? can you get me some examples for it.
    In the UsingVoltdb Pdf i did not find any example for a procedure which can be used in kafkaloader.

    Kindly help as it is very urgent.

    Thanks
    Sureshrajah. R

  2. #2
    Super Moderator
    Join Date
    Feb 2010
    Posts
    106
    Hi Sureshrajah,

    The kafkaloader expects each Kafka message to contain a CSV string. It will decode the CSV string and pass the decoded values as parameters to the procedure you specified. Here's an example of using the kafkaloader with the default insert procedure.

    http://voltdb.com/kafka-voltdb

    Hope it helps.
    Ning

  3. #3
    New Member
    Join Date
    Feb 2017
    Posts
    1
    Hi,

    The link
    http://voltdb.com/kafka-voltdb opens up the link https://www.voltdb.com/products/kafk...ter-and-export . The later gives a general idea only. Can you share a sample procedure used with kafkaloader

  4. #4
    Super Moderator
    Join Date
    Dec 2011
    Posts
    224
    You can use any procedure with kafkaloader or kafka importer as long as the input parameters of the procedure correspond exactly with the columns in the CSV or TSV data coming from the kafka topic.

    For example, suppose you have a table FOO:

    CREATE TABLE foo (
    id BIGINT NOT NULL,
    col1 VARCHAR(100),
    col2 VARCHAR(100)
    );
    PARTITION TABLE foo ON COLUMN id;

    There is an auto-generated procedure FOO.insert that you can call to insert into this table. You can configure KafkaLoader or KafkaImporter to call this procedure so long as the format of the topic matches the input parameters, which are the columns of the table in the order they are defined within the DDL. For example:

    1, "Hello", "World"
    2, "Something", "Something else"

    You could also write a java stored procedure, following instructions in Using VoltDB Ch. 5 (https://docs.voltdb.com/UsingVoltDB/DesignProc.php), if you wanted to add additional validation or reformatting of values, or call multiple SQL statements.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •