Forum: VoltDB Architecture

Post: Data At Rest Compression

Data At Rest Compression
arthurz
Mar 12, 2015
I am curious whether VoltDb compresses the data on disk/at rest.

If it does, what is the algorithm used and if there are options for 3rd party compression methods. E.g. a loss permitted proprietary video stream compression algorithm.
bballard
Mar 12, 2015
Hi Arthur,

VoltDB uses the Snappy compression algorithm for writing snapshots to disk. Snappy is optimized for speed vs. compression ratio, but the compression is still pretty good. There aren't any options for configuring or customizing VoltDB to use a different compression algorithm.

Thanks,
Ben
arthurz
Mar 13, 2015
Thank you Ben!
bballard
Mar 13, 2015
Hi Arthur,

I may have misunderstood your question. I posted this on Stack Overflow as well:

Data stored in VoltDB (e.g. when you insert records) is stored 100% in RAM and is not compressed. There is a sizing worksheet built in to the web interface that can help estimate the RAM required based on the specific datatypes of the tables, and whatever indexes you may define.

One of the datatypes that is supported is VARBINARY which stores byte arrays, i.e. any binary data. You could store pre-compressed data in VARBINARY columns, or use a third-party java compression library within stored procedures to compress and decompress inputs. There is a maximum size limit of 1MB per column, and 2MB per record, however a procedure could store larger sized binary data by splitting it across multiple records. There is a maximum size of 50MB for the inputs to or the results from a stored procedure. You could potentially store and retrieve larger sized binary data using multiple transactions.

We're interested in what you're trying to build. Ok if I contact you at the email address from your Forum account?

Thanks,
Ben