Forum: Managing VoltDB

Post: Snapshot Utilities

Snapshot Utilities
ajgent
May 13, 2010
Two utilities are included in the most recent VoltDB release (0.9.01)
but not covered in the documentation. The command line is likely to
change in future versions. But for anyone who would like to try them
now, the following explains how to use the current implementation.

--Andrew



Managing VoltDB Snapshots

VoltDB provides two additional utilities (in addition to the system
procedures) for managing snapshot files. These utilities verify that a
snapshot is complete and usable and convert the snapshot contents to a
text representation that can be useful for uploading or reloading data
in case of severe errors.

It is possible, as the result of a design flaw or failed program
logic, for a database application to become unusable. However, the data
is still of value. In such emergency cases, it is desirable to extract
the data from the database and possibly reload it. The goal of these
utilities is to simplify that process.

The SnapshotVerifier verifies that a VoltDB snapshot is complete and
usable. The SnapshotConverter converts a snapshot -- or part of a
snapshot -- into text files, creating one file for each table in
snapshot.

Running the Utilities

Unlike system procedures, that must be run within the context of an
existing database connection, the snapshot utilities can be run from the
Linux shell without a running database present. However, the utilities
are still dependent on the Java classes and library for VoltDB. So you
must be sure to define your Java classpath and library path
appropriately to invoke the classes.

To run either SnapshotVerifier or SnapshotConverter, use the Java
command to invoke the class, specifying the appropriate classpath and
library path based on where your VoltDB software is installed. For
example, if VoltDB is installed in /opt/voltdb, the command to invoke
SnapshotVerifier is as follows:


$ Java -classpath /opt/voltdb/voltdb/*.jar \ -Djava.library.path=/opt/voltdb/voltdb \ org.voltdb.utils.SnapshotVerifier
You may find it easier to add an alias to your shell script startup file to abbreviate these commands:


$ alias snapshotverify="-classpath /opt/voltdb/voltdb/*.jar \ -Djava.library.path=/opt/voltdb/voltdb \ org.voltdb.utils.SnapshotVerifier " $ alias snapshotconvert="-classpath /opt/voltdb/voltdb/*.jar \ -Djava.library.path=/opt/voltdb/voltdb \ org.voltdb.utils.SnapshotConverter "
The following section assumes these aliases have been defined.

Command Line Syntax

Each command accepts a different set of arguments. Using the --help
argument displays a list the allowable arguments and qualifiers. For
example:


$ snapshotverify --help
Snapshotverify

For SnapshotVerifier, there are two possible commands:


snapshotverify --all --dirs {directory-list} snapshotverify --single --name {snapshot-id} --dirs {directory-list}
All arguments are required. So if you want to verify all of the
snapshots in the current directory, the command would be as follows:


$ snapshotverify --all --dirs ./
If you want to verify a specific snapshot, use the unique identifier
specified when the snapshot was created (not the filename) in the --name
argument, like so:


$ snapshotverify --single --name mysnaps --dirs /etc/voltdbsave/
Snapshotconvert

SnapshotConverter converts the tables in a valid VoltDb snapshot into
either comma-delimited (csv) or tab-delimited (tsv) text files. The
syntax is:


snapshotconvert ---name {snapshot-id} --tables {table-list} \ --type {csv|tsv} --dirs {directory-list} -- outdir {directory}
Where:



  • {snapshot-id} is the unique identifier specified when the snapshot
    was created. (It is also the name of the .digest file that is part of
    the snapshot.)
  • {table-list} is a comma-separated list of the tables that you want to export to text files. The table names must be uppercase.
  • {csv|tsv} is either "csv" or "tsv" and specifies whether the output
    file is comma-separated or tab-separated. This argument is also used as
    the filetype of the output files.
  • {directory-list} is the list of directories to search for the snapshot.
  • {directory} is the location where you want the text files to be created.



For example, the following command exports all of the tables of a
snapshot of the flight reservation example used in the Using VoltDB
manual as comma-separated files (one file per table) to the user's home
directory:


$ snapshotconvert ---name flightsnap --tables FLIGHT,CUSTOMER,RESERVATION \ --type csv --dirs /etc/voltdbsave/ -- outdir ~/
ArrayIndexOutOfBoundsException
tuancao
May 15, 2010
Hi,

I tried to compile on my Redhat Linux Enterprise box, it gave the following error:

[cat82@wl01 voltdb]$ ls

COPYING doc examples README tools voltdb

[cat82@wl01 voltdb]$ java -classpath ~/tuandev/voltdb/voltdb/*.jar
-Djava.library.path=~/tuandev/voltdb/voltdb
org.voltdb.utils.SnapshotVerifier

Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 0

at org.voltdb.utils.SnapshotVerifier.main(SnapshotVerifier.java:37)

[cat82@wl01 voltdb]$ ls -al ~/tuandev/voltdb/voltdb/

total 121584

drwxr-xr-x 2 cat82 Domain Users 4096 May 15 00:48 .

drwxr-xr-x 6 cat82 Domain Users 4096 May 15 00:48 ..

-rw-r--r-- 1 cat82 Domain Users 35146 May 3 11:25 COPYING

-rw-r--r-- 1 cat82 Domain Users 121225418 May 3 11:25 libvoltdb-0.9.01.so

-rw-r--r-- 1 cat82 Domain Users 3099298 May 3 11:25 voltdb-0.9.01.jar

Could you please take a look?

Thanks,

Tuan
Hi Tuan, You must either use
ajgent
May 15, 2010
Hi,

I tried to compile on my Redhat Linux Enterprise box, it gave the following error:

[cat82@wl01 voltdb]$ ls





Hi Tuan,

You must either use the --help qualifier or specify the complete
command line. Unfortunately, there are currently no default values and
you get the error you found if you do not specify any arguments.

So your command should be one of the following:


$ java -classpath ~/tuandev/voltdb/voltdb/*.jar \-Djava.library.path=~/tuandev/voltdb/voltdb \org.voltdb.utils.SnapshotVerifier \--help$ java -classpath ~/tuandev/voltdb/voltdb/*.jar \-Djava.library.path=~/tuandev/voltdb/voltdb \org.voltdb.utils.SnapshotVerifier \--all --dirs {some-directory-list}$ java -classpath ~/tuandev/voltdb/voltdb/*.jar \-Djava.library.path=~/tuandev/voltdb/voltdb \org.voltdb.utils.SnapshotVerifier \--single --name {snapshot-id} --dirs {directory-list}
Hi, Thanks for replying.
tuancao
May 15, 2010


Hi Tuan,

You must either use the --help qualifier or specify the complete
command line. Unfortunately, there are currently no default values and
you get the error you found if you do not specify any arguments.




Hi,

Thanks for replying.

That was my fault. I did not point the SnapshotVerifier to the directory i stored the snapshot.

Tuan
I wanted to pop in here and
aweisberg
Apr 4, 2011
I wanted to pop in here and say that the documentation for
these utilities here is out of date. You can invoke them with --help to
get the current command line args. They do require that you set
-Djava.library.path=.

Up to date documentation for the utilities will be in the guide for the enterprise management tool in the upcoming 1.3 release.
backup
kookee01
Jul 10, 2012
A full backup of a large data set may take a long time to
complete. On multi-tasking or multi-user systems, there may be writes to
that data while it is being backed up.-Arthur van der Vant