Forum: VoltDB Architecture

Post: Would is be difficult to extend the Export-Only API to be bidirectional ?

Would is be difficult to extend the Export-Only API to be bidirectional ?
monster
Nov 7, 2010
I still have this idea that it would be nice to have the clients only communicate with a single database (VoltDB), but have another storage engine handle large and infrequently accessed data (BLOBs) that is saved on a persistent medium, to save memory.


With an Export-Only table, we already have half of it. The client can save data to that table, and it is committed immediately, without the stored procedure having to wait for the "slow" external storage engine. I was just thinking that one could actually do the same thing for SELECTs. If a stored procedure was allowed to run a SELECT on a "Import-Export-Only" table, but without accessing any VoltDB tables at the same time, then it would not have to go in the "VoltDB Thread" at all, blocking everything, but could just be queued into the external storage engine API, until the SELECT result arrives. Combining this with asynchronous client calls would make it "almost transparent". The client and wire protocol would not have to change, and I think the C++ code would not have to change either.


Does my idea makes sense, from your architectural point-of-view?


As a side-effect, it would become easy to create independent VoltDB clusters, but still allow asynchronous exchange of data. For example, "shards" in a game site, where players can "move" from one shard to another.
Interesting
jhugg
Nov 9, 2010
I think I see what you're asking for and it sounds neat, but I'm not sure it's something we can do soon.


The export functionality we provide solves a problem that isn't easy to be solved in another way. This isn't true (in general) for getting data into VoltDB. It seems like it would be possible to run a process such that a query to an external system could be followed by an insert into VoltDB, then a transaction in VoltDB.


Automating this process would be a neat feature for a later version.