Forum: Building VoltDB Clients

Post: Implementing VoltDB in a virtualized environment (using vmware)

Implementing VoltDB in a virtualized environment (using vmware)
aris_sety
Apr 26, 2010
Hi,
In order to simplify testing process in "High Availability" feature, I build VoltDB in a virtualized environment with the compositions like:


  • VMwareŽ Player 3.0.1 build-227600
  • Intel Core 2 Solo processor SU3500, 64-bit, 4 G memory
  • The host OS is Windows 7 Home Premium, 32-bit (Build 7600) 6.1.7600, ip address 192.168.0.100, netmask 255.255.255.0
  • Two guests OS Ubuntu-9.10 64 bit, memory 512 M each, ip address 192.168.0.96 and 192.168.0.97, netmask 255.255.255.0
  • For networking, I use custom/specific virtual network, using Host only (connect VM's internally in a private network)
  • I'm using "Hello World Application" wich was described in "Getting Started" document

I'm not using recomended memory configuration for simplify testing process. I got a benefit from testing cluster with virtualized environment because with virtualized node, I can increase the volume of node in a cluster just as simple as copy and paste the virtualize node file's.
I build the catalog with 2 node configuration wich host 192.168.0.96 as the leader. An error raised in the leader "VoltDB has encountered an unrecoverable error and is exiting. The log may contain additional", it was said that "INFO HOST - Maximum clock/network is 18337% higher than allowable limit".
In the other node the error was "INFO HOST - Maximum clock/network is 18337% higher than allowable limit". Was that error raised because of I use virtual network adapter, so that the clock was different with real network adapter?
The complete stack trace in leader node was:

root@aris-desktop:~/helloworld# java -Djava.library.path=/usr/voltdb/voltdb-0.6.02/voltdb org.voltdb.VoltDB catalog helloworld.jar No logging configuration supplied via -Dlog4j.configuration. Supplying default config that logs to INFO or higher to STDOUT 257 [main] INFO HOST - Loading... Initialization Log Output: -------------------------------- 334 [main] INFO HOST - Build: 0.6.02 https://svn.voltdb.com/eng/branches/voltdb-0.6?revision=212 714 [main] INFO HOST - HTTP admin console listening on port 8080 714 [main] INFO HOST - Loading application catalog jarfile from /root/helloworld/helloworld.jar 4378 [main] INFO HOST - Creating host manager for 2 hosts using leader /192.168.0.96 4528 [Thread-3] INFO HOST - Connecting to VoltDB cluster as the leader... java.lang.Thread.dumpThreads(Native Method) java.lang.Thread.getAllStackTraces(Thread.java:1503) org.voltdb.VoltDB.crashVoltDB(VoltDB.java:210) org.voltdb.messaging.impl.SocketJoiner.runPrimary(SocketJoiner.java:256) org.voltdb.messaging.impl.SocketJoiner.run(SocketJoiner.java:119) VoltDB has encountered an unrecoverable error and is exiting. The log may contain additional information. 4764 [Thread-3] INFO HOST - Maximum clock/network skew is 366749 milliseconds (according to leader) 4764 [Thread-3] INFO HOST - Maximum clock/network is 18337% higher than allowable limit root@aris-desktop:~/helloworld# The complete stack trace in another node was:

root@aris-desktop:~/helloworld# java -Djava.library.path=/usr/voltdb/voltdb-0.6.02/voltdb org.voltdb.VoltDB catalog helloworld.jar No logging configuration supplied via -Dlog4j.configuration. Supplying default config that logs to INFO or higher to STDOUT 91 [main] INFO HOST - Loading... Initialization Log Output: -------------------------------- 101 [main] INFO HOST - Build: 0.6.02 https://svn.voltdb.com/eng/branches/voltdb-0.6?revision=212 180 [main] INFO HOST - HTTP admin console listening on port 8080 180 [main] INFO HOST - Loading application catalog jarfile from /root/helloworld/helloworld.jar 794 [main] INFO HOST - Creating host manager for 2 hosts using leader /192.168.0.96 844 [Thread-3] INFO HOST - Connecting to the VoltDB cluster leader... 921 [Thread-3] WARN org.voltdb.messaging.impl.SocketJoiner - Joining primary failed: Connection refused retrying.. 1422 [Thread-3] WARN org.voltdb.messaging.impl.SocketJoiner - Joining org.voltdb.messaging.impl.SocketJoiner - Joining primary failed: Connection refused retrying.. 8491 [Thread-3] WARN org.voltdb.messaging.impl.SocketJoiner - Joining primary failed: Connection refused retrying.. 8992 [Thread-3] WARN org.voltdb.messaging.impl.SocketJoiner - Joining primary failed: Connection refused retrying.. java.lang.Thread.dumpThreads(Native Method) java.lang.Thread.getAllStackTraces(Thread.java:1503) org.voltdb.VoltDB.crashVoltDB(VoltDB.java:210) org.voltdb.messaging.impl.SocketJoiner.runNonPrimary(SocketJoiner.java:373) org.voltdb.messaging.impl.SocketJoiner.run(SocketJoiner.java:124) VoltDB has encountered an unrecoverable error and is exiting. The log may contain additional information. 25931 [Thread-3] INFO HOST - Maximum clock/network skew is 366749 milliseconds (according to leader) 25931 [Thread-3] INFO HOST - Maximum clock/network is 18337% higher than allowable limit
re: Clock Skew on Servers
tcallaghan
Apr 26, 2010
Aris,
Your log is showing that the two servers in your configuration have a difference in time of 366749 milliseconds. It is critical in VoltDB that the servers clocks are synchronized with as little skew as possible.
To accomplish this run an NTP client on the servers, with all servers getting time updates from the same NTP server (a local NTP time server if possible, although a remote NTP time server is fine for your development purposes).
After starting NTP clients on all your servers please retest and let me know if you are successful.
-Tim
re re: Clock Skew on Servers
aris_sety
Apr 27, 2010
Aris,
Your log is showing that the two servers in your configuration have a difference in time of 366749 milliseconds. It is critical in VoltDB that the servers clocks are synchronized with as little skew as possible.
To accomplish this run an NTP client on the servers, with all servers getting time updates from the same NTP server (a local NTP time server if possible, although a remote NTP time server is fine for your development purposes).
After starting NTP clients on all your servers please retest and let me know if you are successful.
-Tim


I see. That's true, I'm not using NTP protocol in my environments.
Thank's you
Yes, It was because high time skew
aris_sety
May 6, 2010
I see. That's true, I'm not using NTP protocol in my environments.
Thank's you


It worked, Tim. With virtualized environment, I can flexibel in tunning the amount of node in a cluster, memory size of a node, and core size of the processor.