Forum: Installation

Post: cluster -skew error

cluster -skew error
dav
May 12, 2011
Hi,
I am trying to create a cluster of two Voltdb servers on two different nodes - i.e. 64 bit Ubuntu VM's using the helloworld example by modifying the deployment.xml and having the same deployment.xml on both the nodes. I get the following error:
Buildfile: build.xml



srccompile:



proccompile:


catalog:
[java] 11/05/12 11:06:23 INFO COMPILER: Path to catalog helloworld.sql
[java] 11/05/12 11:06:23 INFO COMPILER: [project.xml]: Created index: SYS_IDX_SYS_PK_10018_10019 on table: HELLOWORLD of type: BALANCED_TREE
[java] 11/05/12 11:06:24 INFO COMPILER: [Insert.class]: Compiling Statement: INSERT INTO HELLOWORLD VALUES (?, ?, ?);
[java] 11/05/12 11:06:24 INFO COMPILER: [Select.class]: Compiling Statement: SELECT HELLO, WORLD FROM HELLOWORLD WHERE DIALECT = ?;
[java] ------------------------------------------
[java] Successfully created helloworld.jar
[java] Includes schema: helloworld.sql
[java]
[java] [SP][RW] Insert
[java] INSERT INTO HELLOWORLD VALUES (?, ?, ?);
[java]
[java] [SP][RO] Select
[java] SELECT HELLO, WORLD FROM HELLOWORLD WHERE DIALECT = ?;
[java]
[java] ------------------------------------------


server:
[java] 11/05/12 18:06:24 INFO HOST: Initializing VoltDB...
[java]
[java] _ __ ____ ____ ____
[java] | | / /___ / / /_/ __ \/ __ )
[java] | | / / __ \/ / __/ / / / __ |
[java] | |/ / /_/ / / /_/ /_/ / /_/ /
[java] |___/\____/_/\__/_____/_____/
[java]
[java] --------------------------------
[java]
[java] 11/05/12 18:06:25 INFO HOST: Build: 1.3.1 https://svn.voltdb.com/eng/tags/voltdb-1.3.1?revision=1793
[java] 11/05/12 18:06:25 INFO HOST: Loading application catalog jarfile from /home/deepak/voltdb/voltdb-1.3.1/examples/helloworld/helloworld.jar
[java] 11/05/12 18:06:25 INFO HOST: URL of deployment info: deployment.xml
[java] 11/05/12 18:06:26 INFO HOST: Cluster has 2 hosts with leader hostname: "10.244.196.156". 2 sites per host. K = 0.
[java] 11/05/12 18:06:26 INFO HOST: The entire cluster has 1 copy of each of the 4 logical partitions.
[java] 11/05/12 18:06:26 INFO HOST: Detection of network partitions in the cluster is not enabled.
[java] 11/05/12 18:06:26 INFO HOST: Using "/home/deepak/voltdb/voltdb-1.3.1/examples/helloworld/voltdbroot" for voltdbroot directory.
[java] 11/05/12 18:06:26 INFO HOST: Beginning inter-node communicaton on port 3021.
[java] 11/05/12 18:06:26 INFO HOST: Connecting to VoltDB cluster as the leader...
[java] 11/05/12 18:06:44 INFO HOST: Maximum clock/network skew is 234 milliseconds (according to leader)
[java] 11/05/12 18:06:44 ERROR HOST: Maximum clock/network is 234% higher than allowable limit
[java] java.lang.Thread.dumpThreads(Native Method)
[java] java.lang.Thread.getAllStackTraces(Thread.java:1530)
[java] org.voltdb.VoltDB.crashVoltDB(VoltDB.java:353)
[java] org.voltdb.messaging.SocketJoiner.runPrimary(SocketJoiner.java:375)
[java] org.voltdb.messaging.SocketJoiner.run(SocketJoiner.java:189)
[java] VoltDB has encountered an unrecoverable error and is exiting.
[java] The log may contain additional information.
[java] Java Result: 255


default:

BUILD SUCCESSFUL
Total time: 22 seconds


I tried the commands -'ntpd -q' and 'ntpd -x' for numerous times. The ntpd is running but I still see the above exceptions when I I try to start the voltdb servers. Is there any good way to synchronize the time.


I am using sunjdk:


deepak@ubuntu:~/voltdb/voltdb-1.3.1/examples/helloworld$ java -version
java version "1.6.0_25"
Java(TM) SE Runtime Environment (build 1.6.0_25-b06)
Java HotSpot(TM) 64-Bit Server VM (build 20.0-b11, mixed mode)



I was able to run the server properly on single node and was able to connect to it. Thanks for your help.

Thanks,
Deepak
I have exactly same copy
dav
May 12, 2011
I have exactly same copy of deployment.xml on both the nodes :


<?xml version="1.0"?>
<deployment>
<cluster hostcount="2"
sitesperhost="2"
leader="10.244.196.156" />
</deployment>

This should help
sebc
May 13, 2011
Hey,
ntpd -q will help but doesn't necessarily guarantee you will get within the 100ms limit in a single call, especially if your VMs are naturally very skewed from the start. Additionally, you need to make sure the VMs aren't somehow ignoring the local NTP service by synchronizing with the host. For instance, when you run on EC2, you will want to make sure you disabled the Xen wallclock, or your NTP setup is essentially useless, right?

I'm going to assume you're running your virtualization on Xen for now, if that's not the case, you'll need to find out how to make sure your VM does NOT synchronize time with the host.

With Xen, your first step is the following:echo 1 > /proc/sys/xen/independent_wallclock


(You'll need to be root for this to get accepted)

Setup NTP to perform an aggressive time correction at startup:


echo NTPD_OPTS='-g -x' > /etc/default/ntp



THat said, as mentioned, once might not be enough, so the following script stops the service, performs serveral calls until we're satified the skew is limited (compared to the server feeding us the time), then restarts the service:

service ntp stop # Aggressively force time sync slew=`ntpd -q | sed 's/ntpd\: time slew //gi' | sed 's/[-+s]//gi'` slew=`echo ${slew:0:4}``echo ${slew:4:1} | sed 's/[0-5]/0/gi'` counter=10 while [ "$slew" != "0.000" -a $counter -gt 1 ]; do sleep 1 slew=`ntpd -q | sed 's/ntpd\: time slew //gi' | sed 's/[-+s]//gi'` slew=`echo ${slew:0:4}``echo ${slew:4:1} | sed 's/[0-5]/0/gi'` let counter-=1 done service ntp start


Ideally, you will be using some internal NTP server. If that is not possible, you can use NTP.org's servers, but you will want to make a careful selection.

FOr instance, selecting 0.pool.ntp.org means "give me any server in the world", and that's likely to give you more variance than you really want. So you'll want to go (for instance, if you're in the US) with 0.us.pool.ntp.org

The following is a sample ntp.conf file that should get your cluster in good shape:

driftfile /var/lib/ntp/ntp.drift server 0.us.pool.ntp.org iburst server 1.us.pool.ntp.org iburst server 2.us.pool.ntp.org iburst server 3.us.pool.ntp.org iburst restrict -4 default kod notrap nomodify nopeer noquery restrict -6 default kod notrap nomodify nopeer noquery restrict 127.0.0.1 restrict ::1


Hope this helps,
Seb
Thanks for the help. I was
dav
May 13, 2011
Hey,
ntpd -q will help but doesn't necessarily guarantee you will get within the 100ms limit in a single call, especially if your VMs are naturally very skewed from the start. Additionally, you need to make sure the VMs aren't somehow ignoring the local NTP service by synchronizing with the host. For instance, when you run on EC2, you will want to make sure you disabled the Xen wallclock, or your NTP setup is essentially useless, right?...


Thanks for the help. I was able to resolve that issue as I made one VM the ntpd server and the other VM as it's client and was able to sync the time. I do not see any skew errors from the voltdb. However I do see timeouts in the non-leader db trying to connect to the leader.

Non-Leader output:
[java] 11/05/13 21:51:07 INFO HOST: Build: 1.3.1 https://svn.voltdb.com/eng/tags/voltdb-1.3.1?revision=1793
[java] 11/05/13 21:51:07 INFO HOST: Loading application catalog jarfile from /home/deepak/voltdb/examples/helloworld/helloworld.jar
[java] 11/05/13 21:51:07 INFO HOST: URL of deployment info: deployment.xml
[java] 11/05/13 21:51:08 INFO HOST: Cluster has 2 hosts with leader hostname: "10.244.196.156". 2 sites per host. K = 0.
[java] 11/05/13 21:51:08 INFO HOST: The entire cluster has 1 copy of each of the 4 logical partitions.
[java] 11/05/13 21:51:08 INFO HOST: Detection of network partitions in the cluster is not enabled.
[java] 11/05/13 21:51:08 INFO HOST: Using "/home/deepak/voltdb/examples/helloworld/voltdbroot" for voltdbroot directory.
[java] 11/05/13 21:51:08 INFO HOST: Beginning inter-node communicaton on port 3021.
[java] 11/05/13 21:51:53 WARN messaging.SocketJoiner: Joining primary failed: Connection timed out retrying..
[java] 11/05/13 21:52:38 WARN messaging.SocketJoiner: Joining primary failed: Connection timed out retrying..
[java] 11/05/13 21:53:24 WARN messaging.SocketJoiner: Joining primary failed: Connection timed out retrying..
[java] 11/05/13 21:51:08 INFO HOST: Connecting to the VoltDB cluster leader...


Leader output:
[java] 11/05/13 21:50:29 INFO HOST: Build: 1.3.1 https://svn.voltdb.com/eng/tags/voltdb-1.3.1?revision=1793
[java] 11/05/13 21:50:29 INFO HOST: Loading application catalog jarfile from /home/deepak/voltdb/voltdb-1.3.1/examples/helloworld/helloworld.jar
[java] 11/05/13 21:50:30 INFO HOST: URL of deployment info: deployment.xml
[java] 11/05/13 21:50:30 INFO HOST: Cluster has 2 hosts with leader hostname: "10.244.196.156". 2 sites per host. K = 0.
[java] 11/05/13 21:50:30 INFO HOST: The entire cluster has 1 copy of each of the 4 logical partitions.
[java] 11/05/13 21:50:30 INFO HOST: Detection of network partitions in the cluster is not enabled.
[java] 11/05/13 21:50:30 INFO HOST: Using "/home/deepak/voltdb/voltdb-1.3.1/examples/helloworld/voltdbroot" for voltdbroot directory.
[java] 11/05/13 21:50:31 INFO HOST: Beginning inter-node communicaton on port 3021.
[java] 11/05/13 21:50:31 INFO HOST: Connecting to VoltDB cluster as the leader...


The deployment.xml is as follows:

<?xml version="1.0"?>
<deployment>
<cluster hostcount="2"
sitesperhost="2"
leader="10.244.196.156" />
</deployment>



Is there anything that I have missed out here. I am also including the output of the basic commands where by I am able to ping leader VM from non-leader VM and the ports that are in use.

Non-leader VM:
deepak@ubuntu:~/voltdb/examples/helloworld$ lsof -i TCP
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
java 9547 deepak 34u IPv6 434668 0t0 TCP 10.244.196.67:35677->10.244.196.156:3021 (SYN_SENT)

deepak@ubuntu:~/voltdb/examples/helloworld$ ping 10.244.196.156
PING 10.244.196.156 (10.244.196.156) 56(84) bytes of data.
64 bytes from 10.244.196.156: icmp_seq=1 ttl=64 time=0.446 ms
64 bytes from 10.244.196.156: icmp_seq=2 ttl=64 time=0.421 ms
64 bytes from 10.244.196.156: icmp_seq=3 ttl=64 time=0.338 ms
^C
--- 10.244.196.156 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1998ms
rtt min/avg/max/mdev = 0.338/0.401/0.446/0.051 ms

Leader VM:
deepak@ubuntu:~/voltdb-ent/management$ lsof -i TCP | fgrep LISTEN
java 12914 deepak 34u IPv6 616169 0t0 TCP 10.244.196.156:3021 (LISTEN)
DNS issue, maybe?
sebc
May 13, 2011
Hey,
There is a long story around this, but for the brief version, nodes must be able to resolve each other by hostname as well as IP address - from the log output, I'm thinking that this could be your issue - do you have a DNS server for those machines? If not, you can work around this by entering each node with its name in /etc/hosts for each machine.

What are you using for virtualization, incidentally? I'm assuming that both VMs are Bridged onto the physical network where they get their IP, correct?
Seb
I am using the two VM's of
dav
May 16, 2011
Hey,
There is a long story around this, but for the brief version, nodes must be able to resolve each other by hostname as well as IP address - from the log output, I'm thinking that this could be your issue - do you have a DNS server for those machines? If not, you can work around this by entering each node with its name in /etc/hosts for each machine.

What are you using for virtualization, incidentally? I'm assuming that both VMs are Bridged onto the physical network where they get their IP, correct?
Seb




I am using the two VM's of ubuntu on windows box installed using vmplayer of vmware and both vm's are bridged onto the physical network from which they get their IP addresses.


I also entered the host names and corresponding IP adresses in the /etc/hosts but it is not working.I did a 'tcpdump -n host ' and I see the IP address getting resolved.You can see below that 10.244.198.67 is my non-leader machine and it is being replied by 10.244.198.64 which is a leader VM.


19:47:20.462079 ARP, Request who-has 10.244.198.64 tell 10.244.196.67, length 28
19:47:20.462321 ARP, Reply 10.244.198.64 is-at 00:0c:29:fa:4a:47, length 46
19:47:39.713250 IP 10.244.196.67.42300 > 10.244.198.64.3021: Flags , seq 2220909268, win 5840, options [mss 1460], length 0
19:47:42.712042 IP 10.244.196.67.42300 > 10.244.198.64.3021: Flags , seq 2220909268, win 5840, options [mss 1460], length 0

Or is it the case that log messages given by voltdb on non-client machine is mis-leading. I will attach the log messages from non-leader machine for your reference (sorry to ask this question as I have seen that log messages will be misleading sometimes).


[java] 11/05/16 02:51:25 WARN messaging.SocketJoiner: Joining primary failed: Connection timed out retrying..
[java] 11/05/16 02:52:11 WARN messaging.SocketJoiner: Joining primary failed: Connection timed out retrying..
Well this message: "WARN
sebc
May 16, 2011
I am using the two VM's of ubuntu on windows box installed using vmplayer of vmware and both vm's are bridged onto the physical network from which they get their IP addresses...


Well this message:
"WARN messaging.SocketJoiner: Joining primary failed: Connection timed out retrying..."

Will tend to happen on the non-leader host if you start it up before you startup the leader, but should self-heal as soon as the leader is found (that's why it's only a "WARN").

That said, it still seems you might have some network resolution issues. Sorry if this is what you did, but I am not entirely clear from your message:

Can you do the following:

ping nameofmachine1

And get an adequate response from BOTH machine1 and machine2

And then also:

ping nameofmachine2

From both machines as well?

Forget all the fancy network checks (that's beyond what I know and shouldn't even be needed).

If this does work, can you check that the firewall isn't somehow active with:

sudo ufw status
(should return: Status: inactive)

If you're all "green" on those things, can you paste in the full standard output log from both nodes?
Thanks for the information. I
dav
May 17, 2011
Thanks for the information. I had two problems. I had to updated my iptables in both the VM's to allow communication on all the ports used by voltdb. Secondly, the two VMs were out of sync wrt time. The ntpdate or ntpd did not work for me and so I wrote a simple client server program to sync one VM with the time of the other VM and got the cluster working. But still I need to fix the issue of ntpd. However my goal of getting the Voltdb cluster is done :-).
Excellent, glad it all worked
sebc
May 18, 2011
Thanks for the information. I had two problems. I had to updated my iptables in both the VM's to allow communication on all the ports used by voltdb. Secondly, the two VMs were out of sync wrt time. The ntpdate or ntpd did not work for me and so I wrote a simple client server program to sync one VM with the time of the other VM and got the cluster working. But still I need to fix the issue of ntpd. However my goal of getting the Voltdb cluster is done :-).


Excellent, glad it all worked out!