Announcement

Collapse
No announcement yet.

What next after EXPORT?

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • interntest
    started a topic What next after EXPORT?

    What next after EXPORT?

    I believe VoltDB has not yet implemented "Capped Collection" or is using EXPORT feature as a workaround. Topics on export is covered quite well. However I have few questions in mind

    Q1: If I export a table (or a subset of it), does that part still remain in memory. I believe its not, but still want a confirmation.

    Q2: What if I need to import the 'exported' data. Does VoltDB allow this? If so, what is/are the condition(s) that has to be met and the do's and dont's?

    Q3: Only INSERT can be performed to an 'exported' table and SELECT, DELETE, UPDATE will throw an error. So these errors might only pop up when I used the same VoltDB instance to perform such actions and should not be the case when it is exported to Hadoop. etc.

  • bballard
    replied
    VoltDB is an in-memory database. The working data set needs to be stored in tables in memory. It can scale to many nodes in a cluster so it is not limited to the memory of a single server.

    Leave a comment:


  • interntest
    replied
    @pzhao and @bballard: Thank you for pointing to jdbcloader and kafkaloader.

    Is there a way to perform loading-on-demand (similar to what happens when a page fault occurs). By 'loading-on-demand', I mean to load the exported table as soon as its referenced by some program/query and not run a separate thread for csv, jdbc or kafka loaders.

    Leave a comment:


  • bballard
    replied
    Hi interntest,

    Export tables in VoltDB are insert-only and are only buffered in memory until they are loaded by an export connector to a target external data system, which is typically within seconds or less. You might think of export tables as a queue for records that will be pushed downstream.

    Typically VoltDB applications use a combination of persistent and export tables. For example, you might use a procedure to insert each record into both a persistent table and an export table with the same column definitions. You might lookup related values or do some other transformations on the records within the procedure before inserting into the export table, to enrich the data. For "sessionization" you might store related records such as disparate events for the same customer in persistent tables, and have some checks in a procedure so that when some combination of events have been received, a summary event is inserted into an export table.

    With version 5.1, you can now configure multiple export streams to different targets. For example, you might push enriched raw data to Hadoop and notifications of detected events to Kafka or RabbitMQ.

    Hopefully this helps give you a picture of some of the possibilities using export tables in VoltDB.

    Regards,
    Ben

    Leave a comment:


  • pzhao
    replied
    Interntest,

    VoltDB has a capped collection via create table's limit constraint. More information is located here.

    Q1: Correct, it is not in memory. Just a clarification, export tables are exported as a whole.
    Q2: VoltDB has a few loaders. Importing data can be done via csvloader, jdbcloader, or kafkaloader.
    Q3: I am not sure exactly what your question is but your first statement is correct.

    For your reference, Export chapter on VoltDB is located here.
    If you have any questions feel free to let me know.

    Peter Zhao

    Leave a comment:

Working...
X