Forum: VoltDB Architecture

Post: Db Replication (failure of agent or slave)

Db Replication (failure of agent or slave)
gambitg
Jun 12, 2012
Is it true that failure of dragent or slave requires the contents of the Replica to be truncated and an empty replica cluster to be created ?

From my tests it seems if you don't do this, the dragent will not start again.
"FATAL: Replica cluster is not empty"


Thanks.
Yes
nshi
Jun 12, 2012
You are right. In that case, both the DR agent and the replica need to be restarted. Section 12.2.4.2 in Using VoltDB lists all cases replication needs to be restarted.DbRepHowTo#DbRepManage

Ning
Re: Db Replication
gambitg
Jun 13, 2012
You are right. In that case, both the DR agent and the replica need to be restarted. Section 12.2.4.2 in Using VoltDB lists all cases replication needs to be restarted.DbRepHowTo#DbRepManage

Ning


Snapshotting the full DB and transferring from primary DC to secondary DC every time the DR agent or replica fails seems like an overkill. Any idea if enhancement on way to avoid the full db transfer incase of a failure of replica cluster ?
Hi, The current WAN
aweisberg
Jun 13, 2012
Snapshotting the full DB and transferring from primary DC to secondary DC every time the DR agent or replica fails seems like an overkill. Any idea if enhancement on way to avoid the full db transfer incase of a failure of replica cluster ?


Hi,


The current WAN replication implementation can't resume if the WAN agent or node the WAN agent has connected to fails. That we plan on fixing outright so that any node in any cluster can fail without breaking replication as long as both clusters stay up.


For an ACID database with distributed transactions it comes down to log shipping. If the the slave goes down we can buffer logs and then ship/replay indefinitely, but then the question is at what point is it faster to just ship a snapshot and a new log.


It probably makes sense to buffer on disk the same amount of data as a snapshot or some low multiple of it. Beyond that shipping a snapshot wins. Right now the value isn't tunable although I personally would like it to be.


I think many of the WAN replication uses cases will have largish amounts of cold data that doesn't need to be shipped. so shipping partial data sets is another possible enhancement.


I don't know if that has been explored in the context of ACID relational databases. Cassandra uses a tree of hashes to diff large data sets and then sync the portions that don't match, but ACID (and distributed ACID) makes that a little more involved. Cassandra is up front about associating a timestamp with everything all the way down to the storage layer. VoltDB isn't doing that with transaction ids although it is something I think will be useful in the long run.


-Ariel