Forum: Managing VoltDB

Post: NETWORK: VoltPort died due to an unexpected exception

NETWORK: VoltPort died due to an unexpected exception
qianxt
Nov 27, 2013
Hi
My voltdb got exited automatically with an exception .
I have 8 nodes in my voltdb and each nodes has 10 partitions.

The errors:

2013-11-21 13:32:30,162 INFO [ZooKeeperServer] JOIN: Agreement, Sending fault data 3:-1 to 0:-1, 1:-1, 2:-1, 4:-1, 5:-1, 6:-1, 7:-1 survivors
2013-11-21 13:32:30,175 INFO [ZooKeeperServer] JOIN: Agreement, Sent fault data. Expecting 7 responses.
2013-11-21 13:32:30,175 INFO [ZooKeeperServer] JOIN: Agreement, Received failure message from 4:-1 for failed sites 3:-1 safe txn id 1559401454421147651 failed site 3:-1
2013-11-21 13:32:30,175 INFO [ZooKeeperServer] JOIN: Agreement, Received failure message from 6:-1 for failed sites 3:-1 safe txn id 1559401463254351875 failed site 3:-1
2013-11-21 13:32:30,175 INFO [ZooKeeperServer] JOIN: Agreement, Received failure message from 5:-1 for failed sites 3:-1 safe txn id 1559401463254351875 failed site 3:-1
2013-11-21 13:32:30,176 INFO [ZooKeeperServer] JOIN: Agreement, Received failure message from 2:-1 for failed sites 3:-1 safe txn id 1559401462784589827 failed site 3:-1
2013-11-21 13:32:30,176 INFO [ZooKeeperServer] JOIN: Agreement, Received failure message from 7:-1 for failed sites 3:-1 safe txn id 1559401463254351875 failed site 3:-1
2013-11-21 13:32:30,176 INFO [ZooKeeperServer] JOIN: Agreement, Received failure message from 1:-1 for failed sites 3:-1 safe txn id 1559401463254351875 failed site 3:-1
2013-11-21 13:32:30,176 INFO [ZooKeeperServer] JOIN: Agreement, Received failure message from 0:-1 for failed sites 3:-1 safe txn id 1559401463254351875 failed site 3:-1
2013-11-21 13:32:30,176 INFO [ZooKeeperServer] JOIN: Agreement, handling site faults for newly failed sites 3:-1 initiatorSafeInitPoints {3:-11559401463254351875}
2013-11-21 13:32:30,176 INFO [ZooKeeperServer] ZK-SERVER: Initiating close of session 0x159401d316000003
2013-11-21 13:32:30,200 INFO [ZooKeeperServer] ZK-SERVER: Processed session termination for sessionid: 0x159401d316000003
2013-11-21 13:32:30,201 INFO [ZooKeeperServer] ZK-SERVER: Processed session termination for sessionid: 0x159401d316000003
2013-11-21 13:32:30,201 INFO [ZooKeeperServer] ZK-SERVER: Processed session termination for sessionid: 0x159401d316000003
2013-11-21 13:32:30,201 INFO [ZooKeeperServer] ZK-SERVER: Processed session termination for sessionid: 0x159401d316000003
2013-11-21 13:32:30,201 INFO [ZooKeeperServer] ZK-SERVER: Processed session termination for sessionid: 0x159401d316000003
2013-11-21 13:32:30,201 INFO [ZooKeeperServer] ZK-SERVER: Processed session termination for sessionid: 0x159401d316000003
2013-11-21 13:32:30,216 INFO [ZooKeeperServer] ZK-SERVER: Processed session termination for sessionid: 0x159401d316000003
2013-11-21 13:32:30,265 ERROR [Volt Network - 4] NETWORK: VoltPort died due to an unexpected exception
java.lang.RuntimeException: java.lang.NullPointerException
at org.voltdb.ClientInterface$ClientInputHandler.handleMessage(ClientInterface.java:755)
at org.voltcore.network.VoltPort.run(VoltPort.java:189)
at org.voltcore.network.VoltNetwork.callPort(VoltNetwork.java:402)
at org.voltcore.network.VoltNetwork.invokeCallbacks(VoltNetwork.java:430)
at org.voltcore.network.VoltNetwork.run(VoltNetwork.java:311)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.NullPointerException
at org.voltdb.dtxn.SimpleDtxnInitiator.createSinglePartitionTxn(SimpleDtxnInitiator.java:346)
at org.voltdb.dtxn.SimpleDtxnInitiator.createTransaction(SimpleDtxnInitiator.java:196)
at org.voltdb.dtxn.SimpleDtxnInitiator.createTransaction(SimpleDtxnInitiator.java:145)
at org.voltdb.ClientInterface.createTransaction(ClientInterface.java:1025)
at org.voltdb.ClientInterface.createTransaction(ClientInterface.java:952)
at org.voltdb.ClientInterface.handleRead(ClientInterface.java:1759)
at org.voltdb.ClientInterface$ClientInputHandler.handleMessage(ClientInterface.java:747)
... 5 more
2013-11-21 13:32:30,265 ERROR [Volt Network - 2] NETWORK: VoltPort died due to an unexpected exception
java.lang.RuntimeException: java.lang.NullPointerException
at org.voltdb.ClientInterface$ClientInputHandler.handleMessage(ClientInterface.java:755)
at org.voltcore.network.VoltPort.run(VoltPort.java:189)
at org.voltcore.network.VoltNetwork.callPort(VoltNetwork.java:402)
at org.voltcore.network.VoltNetwork.invokeCallbacks(VoltNetwork.java:430)
at org.voltcore.network.VoltNetwork.run(VoltNetwork.java:311)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.NullPointerException
at org.voltdb.dtxn.SimpleDtxnInitiator.createSinglePartitionTxn(SimpleDtxnInitiator.java:346)
at org.voltdb.dtxn.SimpleDtxnInitiator.createTransaction(SimpleDtxnInitiator.java:196)
at org.voltdb.dtxn.SimpleDtxnInitiator.createTransaction(SimpleDtxnInitiator.java:145)
at org.voltdb.ClientInterface.createTransaction(ClientInterface.java:1025)
at org.voltdb.ClientInterface.createTransaction(ClientInterface.java:952)
at org.voltdb.ClientInterface.handleRead(ClientInterface.java:1759)
at org.voltdb.ClientInterface$ClientInputHandler.handleMessage(ClientInterface.java:747)
... 5 more
2013-11-21 13:32:30,466 ERROR [Volt Network - 2] NETWORK: VoltPort died due to an unexpected exception
java.lang.RuntimeException: java.lang.NullPointerException
at org.voltdb.ClientInterface$ClientInputHandler.handleMessage(ClientInterface.java:755)
at org.voltcore.network.VoltPort.run(VoltPort.java:189)
at org.voltcore.network.VoltNetwork.callPort(VoltNetwork.java:402)
at org.voltcore.network.VoltNetwork.invokeCallbacks(VoltNetwork.java:430)
at org.voltcore.network.VoltNetwork.run(VoltNetwork.java:311)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.NullPointerException
at org.voltdb.dtxn.SimpleDtxnInitiator.createSinglePartitionTxn(SimpleDtxnInitiator.java:346)
at org.voltdb.dtxn.SimpleDtxnInitiator.createTransaction(SimpleDtxnInitiator.java:196)
at org.voltdb.dtxn.SimpleDtxnInitiator.createTransaction(SimpleDtxnInitiator.java:145)
at org.voltdb.ClientInterface.createTransaction(ClientInterface.java:1025)
at org.voltdb.ClientInterface.createTransaction(ClientInterface.java:952)
at org.voltdb.ClientInterface.handleRead(ClientInterface.java:1759)
at org.voltdb.ClientInterface$ClientInputHandler.handleMessage(ClientInterface.java:747)
... 5 more
2013-11-21 13:32:30,489 ERROR [Volt Network - 2] NETWORK: VoltPort died due to an unexpected exception
java.lang.RuntimeException: java.lang.NullPointerException
at org.voltdb.ClientInterface$ClientInputHandler.handleMessage(ClientInterface.java:755)
at org.voltcore.network.VoltPort.run(VoltPort.java:189)
at org.voltcore.network.VoltNetwork.callPort(VoltNetwork.java:402)
at org.voltcore.network.VoltNetwork.invokeCallbacks(VoltNetwork.java:430)
at org.voltcore.network.VoltNetwork.run(VoltNetwork.java:311)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.NullPointerException
at org.voltdb.dtxn.SimpleDtxnInitiator.createSinglePartitionTxn(SimpleDtxnInitiator.java:346)
at org.voltdb.dtxn.SimpleDtxnInitiator.createTransaction(SimpleDtxnInitiator.java:196)
at org.voltdb.dtxn.SimpleDtxnInitiator.createTransaction(SimpleDtxnInitiator.java:145)
at org.voltdb.ClientInterface.createTransaction(ClientInterface.java:1025)
at org.voltdb.ClientInterface.createTransaction(ClientInterface.java:952)
at org.voltdb.ClientInterface.handleRead(ClientInterface.java:1759)
at org.voltdb.ClientInterface$ClientInputHandler.handleMessage(ClientInterface.java:747)
... 5 more
2013-11-21 13:32:30,578 ERROR [Volt Network - 3] NETWORK: VoltPort died due to an unexpected exception
java.lang.RuntimeException: java.lang.NullPointerException
at org.voltdb.ClientInterface$ClientInputHandler.handleMessage(ClientInterface.java:755)
at org.voltcore.network.VoltPort.run(VoltPort.java:189)
at org.voltcore.network.VoltNetwork.callPort(VoltNetwork.java:402)
at org.voltcore.network.VoltNetwork.invokeCallbacks(VoltNetwork.java:430)
at org.voltcore.network.VoltNetwork.run(VoltNetwork.java:311)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.NullPointerException
at org.voltdb.dtxn.SimpleDtxnInitiator.createSinglePartitionTxn(SimpleDtxnInitiator.java:346)
at org.voltdb.dtxn.SimpleDtxnInitiator.createTransaction(SimpleDtxnInitiator.java:196)
at org.voltdb.dtxn.SimpleDtxnInitiator.createTransaction(SimpleDtxnInitiator.java:145)
at org.voltdb.ClientInterface.createTransaction(ClientInterface.java:1025)
at org.voltdb.ClientInterface.createTransaction(ClientInterface.java:952)
at org.voltdb.ClientInterface.handleRead(ClientInterface.java:1759)
at org.voltdb.ClientInterface$ClientInputHandler.handleMessage(ClientInterface.java:747)
... 5 more
2013-11-21 13:32:30,763 FATAL [Mailbox tracker] HOST: Configured number of partitions 80 is not the same as the number of partitions present 70
2013-11-21 13:32:30,763 FATAL [Mailbox tracker] HOST: Stack trace from crashLocalVoltDB() method:
2013-11-21 13:32:30,764 FATAL [Mailbox tracker] HOST: java.lang.Thread.dumpThreads(Native Method)
2013-11-21 13:32:30,764 FATAL [Mailbox tracker] HOST: java.lang.Thread.getAllStackTraces(Thread.java:1530)
2013-11-21 13:32:30,764 FATAL [Mailbox tracker] HOST: org.voltdb.VoltDB.crashLocalVoltDB(VoltDB.java:558)
2013-11-21 13:32:30,764 FATAL [Mailbox tracker] HOST: org.voltdb.VoltDB.crashGlobalVoltDB(VoltDB.java:691)
2013-11-21 13:32:30,764 FATAL [Mailbox tracker] HOST: org.voltdb.RealVoltDB.handleMailboxUpdate(RealVoltDB.java:2145)
2013-11-21 13:32:30,764 FATAL [Mailbox tracker] HOST: org.voltdb.dtxn.MailboxTracker.getMailboxes(MailboxTracker.java:192)
2013-11-21 13:32:30,764 FATAL [Mailbox tracker] HOST: org.voltdb.dtxn.MailboxTracker.access$000(MailboxTracker.java:54)
2013-11-21 13:32:30,764 FATAL [Mailbox tracker] HOST: org.voltdb.dtxn.MailboxTracker$EventTask.run(MailboxTracker.java:75)
2013-11-21 13:32:30,764 FATAL [Mailbox tracker] HOST: java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
2013-11-21 13:32:30,764 FATAL [Mailbox tracker] HOST: java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
2013-11-21 13:32:30,764 FATAL [Mailbox tracker] HOST: java.util.concurrent.FutureTask.run(FutureTask.java:138)
2013-11-21 13:32:30,765 FATAL [Mailbox tracker] HOST: java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
2013-11-21 13:32:30,765 FATAL [Mailbox tracker] HOST: java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
2013-11-21 13:32:30,765 FATAL [Mailbox tracker] HOST: java.lang.Thread.run(Thread.java:662)


Why does this happen?
nshi
Nov 27, 2013
Hi Qian,

The stack traces look like it was from a version prior to 3.0. Are you running with v2.x?

We have re-architected how transactions are managed in the system since 3.0. This should no longer be an issue in 3.0 or above. The latest released version is v3.7. You can download it from our website http://voltdb.com/download/software.
qianxt
Nov 28, 2013
Hi Nshi

My voltdb is from the version on 2.8.2
Can you tell me why this exception happened?
nshi
Nov 28, 2013
Hi Qian,

The NullPointerExceptions may be caused by the node failures happened immediately before that. The log indicated that some other nodes in the cluster had failed at 2013-11-21 13:32:30, which possibly made the cluster unviable.

Were there other node failures in your cluster? What was the k-factor?
qianxt
Dec 2, 2013
Hi Nshi

There were no other failures . This is my deployment.xml .

<deployment>
<cluster hostcount="8" sitesperhost="10" />
<systemsettings>
<temptables maxsize="500"/>
</systemsettings>
</deployment>
nshi
Dec 2, 2013
Hi Qian,

The following log message indicated that either a node in the cluster had failed or the network connection between the two nodes broke, which caused the NullPointerExceptions afterwards.

"2013-11-21 13:32:30,162 INFO [ZooKeeperServer] JOIN: Agreement, Sending fault data 3:-1 to 0:-1, 1:-1, 2:-1, 4:-1, 5:-1, 6:-1, 7:-1 survivors"

SimpleDtxnInitiator has been deprecated since v3.0 when we introduced the new transaction management system, so it no longer has this bug. We strongly recommend you upgrade to the latest version of VoltDB, which has a lot of performance improvements and new features.