Forum: Building VoltDB Applications

Post: Append only OLTP applications

Append only OLTP applications
ccherng
Mar 29, 2013
Consider an application where data is added to the database. That is, it is appended and never updated and after it is persisted it is read very infrequently. With Voltdb this data which is read infrequently would normally be constantly using memory. Is there any notion in Voltdb of keeping infrequently used data on disk instead of in memory. That situation intuitively seems like it would be common; how prevalent are applications that I am describing?
jpiekos
Mar 29, 2013
You are correct, that infrequently read data stays in memory when it resides in VoltDB. We also offer the ability, with our Export functionality, to automatically export data no longer needed in VoltDB to downstream systems (hadoop, csv file, jdbc, vertica, netezza). That being said, the situation you describe, where you keep infrequently read (cold) data in VoltDB, residing in some "cold" storage like disk, is a scenario we've considered for a future release, though not near-term.

John
pmartel
Mar 29, 2013
The VoltDB engineering team has also discussed providing automatic support for "capped collections". With this feature, the memory size of a table is managed by the automatic eviction of older/colder data. This future feature (like "cold storage", not yet scheduled for development AFAIK) could provide an alternative to use of the VoltDB native cold storage that John described.

The capped collections feature could be supplemented with use of the current export feature and possibly a small amount of custom client programming to re-import past data of interest (in small amounts, infrequently, on demand). This might be easier (than the cold storage feature) to approximate with the current release of VoltDB by assigning old/cold data cleanup responsibilities to client programs, possibly special-purpose garbage collector daemons. This seems to me to be a viable approach even today for highly scalable append-only (or even append-mostly) OLTP applications.

--paul