Forum: Building VoltDB Applications

Post: Config for bringing up the VoltDB in cluster mode

Config for bringing up the VoltDB in cluster mode
shetty_ritesh
May 6, 2010
Hi,
I gave following config to bring up the leader node. Here I intend to have 2 nodes and each having 2 partitions. I havent specified the k -Safety.
I get the following error while starting the server.
************************************************************************************************
nohup: ignoring input
Buildfile: build.xml

server:
[java] [main] INFO HOST - Loading...
[java]
[java] _ __ ____ ____ ____
[java] | | / /___ / / /_/ __ \/ __ )
[java] | | / / __ \/ / __/ / / / __ |
[java] | |/ / /_/ / / /_/ /_/ / /_/ /
[java] |___/\____/_/\__/_____/_____/
[java]
[java] Initialization Log Output:
[java] --------------------------------
[java]
[java] [main] INFO HOST - Build: 0.9.01 https://svn.voltdb.com/eng/trunk?revision=475
[java] [main] INFO HOST - HTTP admin console listening on port 8080
[java] [main] INFO HOST - Loading application catalog jarfile from /home/shettyr/amitabh/VoltDBMapper/subject.jar
[java] [main] INFO HOST - Creating host manager for 2 hosts using leader /192.168.160.129
[java] [Thread-4] INFO HOST - Connecting to the VoltDB cluster leader...
[java] [Thread-4] INFO HOST - Maximum clock/network skew is 2081 milliseconds (according to leader)
[java] [Thread-4] INFO HOST - Catalog checksums do not match across cluster
[java] java.lang.Thread.dumpThreads(Native Method)
[java] java.lang.Thread.getAllStackTraces(Thread.java:1487)
[java] org.voltdb.VoltDB.crashVoltDB(VoltDB.java:210)
[java] org.voltdb.messaging.SocketJoiner.runNonPrimary(SocketJoiner.java:413)
[java] org.voltdb.messaging.SocketJoiner.run(SocketJoiner.java:137)
[java] VoltDB has encountered an unrecoverable error and is exiting.
[java] The log may contain additional information.
[java] Java Result: 255
BUILD SUCCESSFUL
Total time: 1 second
************************************************************************
The config is as follows
************************************************************************************************


<target name="catalog" depends="clientjar" description="Create the catalog.">
<java fork="yes" classname="org.voltdb.compiler.VoltCompiler">
<arg value="project.xml" />
<!-- project file -->
<arg value="2" />
<!-- hosts -->
<arg value="2" />
<!-- sites -->
<arg value="192.168.160.129" />
<!-- leader -->
<arg value="subject.jar" />
<!-- output -->
<classpath refid='project.classpath' />
<assertions>
<disable />
</assertions>
</java>
</target>
Hopefully two problems.
jhugg
May 6, 2010
Hi Ritesh,

We've added a check in 0.9.01 that computes a CRC checksum of the catalog JAR as seen by each cluster member. If the catalog is not bytewise-identical, then the cluster won't start and will print the error you see.

Additionally, we've tightened the restrictions on clock skew to 100ms. Your cluster is reporting a 2 second skew which would also cause the system not to start, if it got past the CRC check. It's unfortunate we don't show both errors, just the first one we run into. It's possible this is a bug and we've got the messages for CRC-fail and clock-skew-fail reversed. I peeked at the code and I don't think so, but it's possible.

So my advice would be to first double check you have the exact same catalog jar file available to both cluster nodes. If you're sure the file is identical, let me know and I'll treat it as a bug. Second, try running NTP on the nodes configured to sync to the same NTP server. This should bring the skew down to acceptable levels.

-John Hugg
VoltDB Engineering
John, I get the error when i
shetty_ritesh
May 6, 2010
Hi Ritesh,

We've added a check in 0.9.01 that computes a CRC checksum of the catalog JAR as seen by each cluster member. If the catalog is not bytewise-identical, then the cluster won't start and will print the error you see...
-John Hugg
VoltDB Engineering

John,
I get the error when i start the leader node. the second node is not up yet. Also i have the same catalog jar on both nodes.
do i have to bring up all
shetty_ritesh
May 6, 2010
John,
I get the error when i start the leader node. the second node is not up yet. Also i have the same catalog jar on both nodes.



do i have to bring up all nodes at the same time ?
i will try to synch the time
shetty_ritesh
May 6, 2010
do i have to bring up all nodes at the same time ?

i will try to synch the time on the 2 servers and let you know
Two notes
jhugg
May 7, 2010
i will try to synch the time on the 2 servers and let you know




1) The catalog CRC check and network skew check shouldn't run until all VoltDB processes on all nodes have started (though not fully finished initializing).
2) The output from your original post is from a non-leader node. That's why it says it's trying to connect to the leader. Or at least it doesn't think it's the leader.

So do you think only one process on one node was running when this output happened?
If so, is it possible that another VoltDB process was running from a previous test?

If there really is only one VoltDB process running on your cluster and you got this message, then you've found a bug in VoltDB. Let me know.

-John
thanks John. You were right.
shetty_ritesh
May 7, 2010
1) The catalog CRC check and network skew check shouldn't run until all VoltDB processes on all nodes have started (though not fully finished initializing).
2) The output from your original post is from a non-leader node. That's why it says it's trying to connect to the leader. Or at least it doesn't think it's the leader...

-John



thanks John. You were right. 2 problems

1>Even though the catalog was the same it was build on different nodes and hence checksum failed
2>VoltDB didnt tolerate a time gap of even 170 milli sec. after synching the 2 servers using NTP the servers are up on clustered node

Thanks
Ritesh
Checksum
henning
Aug 5, 2010
I ran into the same issue. Is there anything to be done about it?

If I have three systems set up as similar as could be, it's counter intuitive to begin with to accept the system's conclusion that the catalogues look different. I was not quite clear if only the .jar or maybe the entire directory was meant because I was so sure the jars must be identical, for the file lengths matched and I had done the exact same steps and not much more on the servers. So I did not *believe* the server and wasted time installing the sample all over.

Eventually I checked md5 and, of course VoltDB is right and Java buffs may instantly know why. I also understand that a parameter to ignore the checksum might produce more pain than good in the long run. But is there no better way? For now, while experimenting, I am stuck with a rather senseless manual task of distributing catalogues. And before I get some rsynch or virtual drive in place, I'll be hacking away with sftp.

Maybe for catalogues < 10K VoltDB could accept differing md5s if the file length is the same? Just a pragmatic thought to make starting with VoltDB yet easier. Sure you don't have a secret parameter to disable the check in place anyway?

Thanks,
Henning
We're working on improving this process.
jhugg
Aug 5, 2010
I ran into the same issue. Is there anything to be done about it?

If I have three systems set up as similar as could be, it's counter intuitive to begin with to accept the system's conclusion that the catalogues look different. I was not quite clear if only the .jar or maybe the entire directory was meant because I was so sure the jars must be identical, for the file lengths matched and I had done the exact same steps and not much more on the servers. So I did not *believe* the server and wasted time installing the sample all over...

Thanks,
Henning



There are two tickets in the issue tracker to improve this:
https://issues.voltdb.com/browse/ENG-543 (from 5/15/10) and
https://issues.voltdb.com/browse/ENG-682 (from today).

Our primary, long term path is to make it easier for nodes to share a single catalog via a definitive URL over http. If you're hosting a catalog at a URL, you should currently be able to fire up a cluster that way, rather than using a local file. To make this path easier in the future, one thought would be to make the leader node serve the catalog over http to the other nodes in the cluster, and to any nodes rejoining after a failure.

In the short term, we've figured out why creating a catalog isn't deterministic. It turns out the contents of the jar are bytewise identical, but the jar process itself injects some differences in the resulting jar file. So our plan is to compute checksums based on the contents, rather than the bytes of the jar file itself. Hopefully we'll get that fix on the trunk soon and have it working for 1.2.