Announcement

Collapse
No announcement yet.

how we process the data from kafkaloader through procedure

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • how we process the data from kafkaloader through procedure

    Hi,

    I want to process the data which comes from kafkaloader i.e in realtime. how will i do that??

    i saw that in kafkaloader i have a option of procedure which can process the data which comes to kafka one by one.
    how should the procedure be? can you get me some examples for it.
    In the UsingVoltdb Pdf i did not find any example for a procedure which can be used in kafkaloader.

    Kindly help as it is very urgent.

    Thanks
    Sureshrajah. R

  • #2
    Hi Sureshrajah,

    The kafkaloader expects each Kafka message to contain a CSV string. It will decode the CSV string and pass the decoded values as parameters to the procedure you specified. Here's an example of using the kafkaloader with the default insert procedure.

    http://voltdb.com/kafka-voltdb

    Hope it helps.
    Ning

    Comment


    • #3
      Hi,

      The link
      http://voltdb.com/kafka-voltdb opens up the link https://www.voltdb.com/products/kafk...ter-and-export . The later gives a general idea only. Can you share a sample procedure used with kafkaloader

      Comment


      • #4
        You can use any procedure with kafkaloader or kafka importer as long as the input parameters of the procedure correspond exactly with the columns in the CSV or TSV data coming from the kafka topic.

        For example, suppose you have a table FOO:

        CREATE TABLE foo (
        id BIGINT NOT NULL,
        col1 VARCHAR(100),
        col2 VARCHAR(100)
        );
        PARTITION TABLE foo ON COLUMN id;

        There is an auto-generated procedure FOO.insert that you can call to insert into this table. You can configure KafkaLoader or KafkaImporter to call this procedure so long as the format of the topic matches the input parameters, which are the columns of the table in the order they are defined within the DDL. For example:

        1, "Hello", "World"
        2, "Something", "Something else"

        You could also write a java stored procedure, following instructions in Using VoltDB Ch. 5 (https://docs.voltdb.com/UsingVoltDB/DesignProc.php), if you wanted to add additional validation or reformatting of values, or call multiple SQL statements.

        Comment

        Working...
        X