Forum: Building VoltDB Applications

Post: Export monitor

Export monitor
kdombeck
Dec 22, 2010
We have a ExportToFileClient that is exporting our export only table to a file. What is the best way to monitor this process to ensure it is not falling behind with writing the data to a file?
Is a process that runs every x minutes that does a count(*) of the export table the best way to see if records are not being written out or is there a better way to monitor the backlog of data in the export only table?
As implemented, there are a
rbetts
Dec 23, 2010
As implemented, there are a few ways to discover this information, but none are great.
Note, export tables are write-only from SQL, as documented. You can't execute count or select against them.
1. The export data includes the transaction ID of the transaction that wrote the exported row. You could monitor the transaction id written to the table versus the last transaction to update the export-only table.
2. The export only table counts the total number of rows written to it. This statistic could be compared to the total rows written by the file client.
3. The system supports flushing all queued export data and the client can be run until all queues are empty. This requires pausing the system, though. This workflow is only realistic for shutdown operations.
4. The export file client knows when it polled an empty data source; however it does not record or log this event.
This is a clear shortcoming - I'd prefer to simply add the statistics you require. I'll discuss this internally today.
Update
seven
Jan 4, 2011
As implemented, there are a few ways to discover this information, but none are great.
Note, export tables are write-only from SQL, as documented. You can't execute count or select against them.
1. The export data includes the transaction ID of the transaction that wrote the exported row. You could monitor the transaction id written to the table versus the last transaction to update the export-only table.
2. The export only table counts the total number of rows written to it. This statistic could be compared to the total rows written by the file client.
3. The system supports flushing all queued export data and the client can be run until all queues are empty. This requires pausing the system, though. This workflow is only realistic for shutdown operations.
4. The export file client knows when it polled an empty data source; however it does not record or log this event.
This is a clear shortcoming - I'd prefer to simply add the statistics you require. I'll discuss this internally today.


Any update on how to monitor the export only tables?
Updated table stats to track queued byte count.
rbetts
Jan 4, 2011
Any update on how to monitor the export only tables?


I added / fixed the allocated data memory counter for export only tables. This change is on our trunk. I updated the support issue and the engineering defect: https://issues.voltdb.com/browse/ENG-947
With this change, the table statistics selector can be polled to determine how much data is currently queued for export only tables.
Would you like this change merged to the 1.2.1.x branch?
Release
seven
Jan 6, 2011
I added / fixed the allocated data memory counter for export only tables. This change is on our trunk. I updated the support issue and the engineering defect: https://issues.voltdb.com/browse/ENG-947
With this change, the table statistics selector can be polled to determine how much data is currently queued for export only tables.
Would you like this change merged to the 1.2.1.x branch?


When is the next release scheduled? If it is not too far out we can wait until then.
Export fix and the next release
bheath
Jan 7, 2011
When is the next release scheduled? If it is not too far out we can wait until then.


We are planning to add this fix (Eng-947) to the 1.2.1.x branch as well as trunk.
In answer to your other question, we plan to ship version 1.3 in March. I've added some details about the contents on the Roadmap page: http://community.voltdb.com/roadmap
Bobbi Heath