Forum: Building VoltDB Applications

Post: Autosnapshot loading

Autosnapshot loading
dark5tar
Sep 9, 2016
Hi

we used to run save command to create snapshot , that creates csv files to load later
Now i've implemented autosnapshot , but this creates .vpt files. can somebody help me how to convert this to .csv
or how to load these when i create new voltdb?

I tried using snapshotconverter tool, but that doesn't work, keep giving trouble one or the other.
I'm using ansible to automate this, If anybody worked on this loading table data from autosnapshot, can you please gibe me some direction?

thanks
Andy
rmorgenstein
Sep 12, 2016
Use the 'voltadmin restore' command to load a snapshot. See https://docs.voltdb.com/UsingVoltDB/ChapSaveRestore.php for more information.

Ruth
dark5tar
Sep 13, 2016
Hi
Somehow this doesn't work. I can get AUTO snapshot generated when some data is in the schema.
After that I drop DB and create DB with empty tables
then I'm running voltadmin restore --host=localhost:<port> <snapshot location> <snapshot ID>
this fails with Error
Result: The database crashes.
voltadmin: Snapshot restore has been started. Check the server logs for ongoing status of the restore operation.
ERROR: "@SnapshotRestore" procedure call failed.
ERROR: Status: -1
ERROR: Information: Connection broken
ERROR:
In the logs: INFO [Snapshot Restore - 0] SNAPSHOT: Performing restore for table: T
FATAL [SP 0 site - 0:0] HOST: Attempted violation of constraint
Constraint violation type: Unique
on table: T
New tuple:

if i fix one table then other table fails, where as same data as CSV saved file works fine.
rmorgenstein
Sep 13, 2016
Andy, what version are you using? I know we fixed a snapshotconverter problem in V5.5. But, really, the fastest way to load everything is using a native-format (i.e. .vpt) snapshot.

If you are loading an entire snapshot, then ALL tables must be empty. In V6.6, which we just released, VoltDB will no longer crash - all constraint violations are logged. But in previous versions it was fatal.

Is there any chance that a user or application is loading a little data after the database starts. For example, we had a customer that had a cron job that would init/update some small dimension tables periodically and this would cause problems like what you see. The best way to handle this is to always bring the database up in admin mode if you plan to load a snapshot.

Ruth
dark5tar
Sep 14, 2016
Hi Ruth

Thanks for the information. My Volt version is 5.8.1.
I use to load entire snapshot with 33 tables in it.
Also three of the tables are created with some default data when I run the Ansible to create voltdb.
But this didn't give me problem when I load CSV snapshot which was created with Save command with CSV format.
I just tried restoring using Admin mode thru web console (.vpt snapshot).
Still same problem, DB crashed when try to load the table which has some default data in it.
See the steps i followed and the result

103104105106107

Now I found out how to start in admin mode and worked that way both commands
a) voltadmin restore

and logged into db
b) exec @SnapshotRestore

Both commands same result with constraint key failure..
rmorgenstein
Sep 14, 2016
Andy,

Loading from CSV allows for conflicts and errors - they go to a logfile, which you are welcome to ignore. Until recently snapshot restore was much less forgiving. It looks to me like you brought up the database, then Paused it. At some point before the pause, Ansible or another process loaded the data.

To bring up the database paused from the beginning, you need to put the following in your deployment file:

<deployment>
<admin-mode adminstartup="true">
</deployment>

If Ansible is trying to populate these tables AND if it is using the client port (21212 by default), then it will not be able to do so.

As an alternative, if you don't want to manage having 2 deployment.xml files, you could simply truncate the tables that ansible populated before doing the restore.

BTW, in V6.6, which we just released, we introduced a much simplified command-line interface that makes it much easier to use containers and scripts for VoltDB. Check out the release notes at https://docs.voltdb.com/ReleaseNotes/. If you upgrade, one of the things you'd get is the ability to use --pause on the command-line so that you don't have to keep an extra deployment-adminmode.xml around to come up paused when you need to.
dark5tar
Oct 13, 2016
Hi Ruth
I've succeed in loading autosnapshot thru ansible, I'm truncating tables before loading autosnapshot thru "restore" command
Now first time it works fine cleaning tables and loading snapshot, if i run the same script again it fails with the following error.
Can you please help

108

it appears to me that cluster has been restored once (in first run), how do i capture this as flag and reset when running for second time.
rmorgenstein
Oct 14, 2016
Restoring a complete snapshot is allowed once. There is no reset flag. You should shutdown the cluster and then 'voltdb create' again to start fresh with no data and restore the snapshot.

Ruth