Forum: Building VoltDB Clients

Post: NoConnectionsException in the middle of a test

NoConnectionsException in the middle of a test
pmahlen
Nov 16, 2011
Hi,
When running a test that is doing some 50k+ writes/sec, I suddenly get the terminal flooded with messages like these:
Connection to database host (127.0.0.1) was lost before a response was received
Connection to database host (127.0.0.1) was lost before a response was received
Connection to database host (127.0.0.1) was lost before a response was received
2011-11-16 13:36:37,469 ERROR [BatchInserter.java:52] caught exception in batch: 480
java.lang.RuntimeException: org.voltdb.client.NoConnectionsException: No connections.
at com.shopzilla.inventory.rdr.test.repository.voltdb.VoltCrudRepository.asynchronousCall(VoltCrudRepository.java:124)
at com.shopzilla.inventory.rdr.test.repository.voltdb.VoltCrudRepository.save(VoltCrudRepository.java:36)
at com.shopzilla.inventory.rdr.test.repository.voltdb.VoltCrudRepository.save(VoltCrudRepository.java:21)
at com.shopzilla.inventory.rdr.test.generation.BatchInserter.run(BatchInserter.java:43)
at com.shopzilla.inventory.rdr.test.generation.DataInsertionThread$2.run(DataInsertionThread.java:99)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)
Caused by: org.voltdb.client.NoConnectionsException: No connections.
at org.voltdb.client.Distributer.queue(Distributer.java:555)
at org.voltdb.client.ClientImpl.callProcedure(ClientImpl.java:295)
at org.voltdb.client.ClientImpl.callProcedure(ClientImpl.java:237)
at com.shopzilla.inventory.rdr.test.repository.voltdb.VoltCrudRepository.asynchronousCall(VoltCrudRepository.java:121)
... 10 more
There are many thousands of the 'Connection to database host' lines.
Once, when I hit control-C to stop the load generation, there was this additional message at the end of the log:
Exception in thread "Volt Network" java.lang.NoSuchFieldError: TRACE
at org.voltdb.logging.VoltLog4jLogger.getPriorityForLevel(VoltLog4jLogger.java:63)
at org.voltdb.logging.VoltLog4jLogger.isEnabledFor(VoltLog4jLogger.java:77)
at org.voltdb.logging.VoltLogger.isTraceEnabled(VoltLogger.java:95)
at org.voltdb.network.NIOWriteStream.backpressureStarted(NIOWriteStream.java:165)
at org.voltdb.network.NIOWriteStream.drainTo(NIOWriteStream.java:372)
at org.voltdb.network.VoltPort.drainWriteStream(VoltPort.java:291)
at org.voltdb.network.VoltPort.call(VoltPort.java:214)
at org.voltdb.network.VoltNetwork$4.run(VoltNetwork.java:598)
at org.voltdb.network.VoltNetwork.invokeCallbacks(VoltNetwork.java:632)
at org.voltdb.network.VoltNetwork.run(VoltNetwork.java:451)
at java.lang.Thread.run(Thread.java:680)
This is intermittent, so some runs will be successful, whereas others fail. There are no messages on the server terminal. Any idea what this could be? I'm running this on Mac OS X Lion.
Thanks,
Petter
Here's a stack trace - this
pmahlen
Nov 17, 2011
Here's a stack trace - this happens all the time when I run enough asynchronous statements, whether this is done single-threaded or multi-threaded. Usually, it works fine for a couple of seconds, and then something seems to get overloaded and blow up.
java.lang.RuntimeException: CrudRepo for entity ProductVariant failed with message: Connection to database host (127.0.0.1) was lost before a response was received
at com.shopzilla.inventory.rdr.test.repository.voltdb.VoltCrudRepository$VoltDBCrudCallback.clientCallback(VoltCrudRepository.java:179)
at org.voltdb.client.ClientImpl$1.clientCallback(ClientImpl.java:290)
at org.voltdb.client.Distributer$NodeConnection.stopping(Distributer.java:331)
at org.voltdb.network.VoltPort.unregistering(VoltPort.java:402)
at org.voltdb.network.VoltNetwork.unregisterChannel(VoltNetwork.java:411)
at org.voltdb.network.VoltNetwork.p_shutdown(VoltNetwork.java:490)
at org.voltdb.network.VoltNetwork.run(VoltNetwork.java:460)
at java.lang.Thread.run(Thread.java:680)
Then, after probably thousands of these, the following starts happening:
java.lang.RuntimeException: org.voltdb.client.NoConnectionsException: No connections.
at com.shopzilla.inventory.rdr.test.repository.voltdb.VoltCrudRepository.asynchronousCall(VoltCrudRepository.java:146)
at com.shopzilla.inventory.rdr.test.repository.voltdb.VoltCrudRepository.save(VoltCrudRepository.java:39)
at com.shopzilla.inventory.rdr.test.repository.voltdb.VoltProductRepository.save(VoltProductRepository.java:31)
at com.shopzilla.inventory.rdr.test.repository.voltdb.VoltProductRepository.save(VoltProductRepository.java:21)
at com.shopzilla.inventory.rdr.test.repository.voltdb.VoltCrudRepository.save(VoltCrudRepository.java:47)
at com.shopzilla.inventory.rdr.test.generation.BatchInserter.run(BatchInserter.java:40)
at com.shopzilla.inventory.rdr.test.generation.DataInsertionThread$2.run(DataInsertionThread.java:99)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)
Caused by: org.voltdb.client.NoConnectionsException: No connections.
at org.voltdb.client.Distributer.queue(Distributer.java:555)
at org.voltdb.client.ClientImpl.callProcedure(ClientImpl.java:295)
at org.voltdb.client.ClientImpl.callProcedure(ClientImpl.java:237)
at com.shopzilla.inventory.rdr.test.repository.voltdb.VoltCrudRepository.asynchronousCall(VoltCrudRepository.java:143)
... 12 more
Hi Petter,The client library
aweisberg
Nov 17, 2011
Hi Petter,
The client library throws a NoConnectionsException if you attempt to submit a transaction when it is not connected to a server. Because you are catching the exception and continuing to submit new requests there is a flood of messages from the caught exception.
The server is hanging up on the client because there is an exception in the network code while operating on the connection to the client. The exception is caused by some logging code that is checking if trace level logging is enabled. It looks like the version of log4j you are running with doesn't have org.apache.log4j.Level.TRACE defined. You may have multiple log4j jars in your classpath?
log4j hasn't changed in a long time so it is strange that you have a version that doesn't define that field.
The reason it appears to happen under load is that the trace statement is in code that is only run when the server is flooded with requests and decides to stop reading new ones until the existing ones are processed. That is the backpressure being referred to.
-Ariel
That was it - for whatever
pmahlen
Nov 17, 2011
Hi Petter,
The client library throws a NoConnectionsException if you attempt to submit a transaction when it is not connected to a server. Because you are catching the exception and continuing to submit new requests there is a flood of messages from the caught exception.
The server is hanging up on the client because there is an exception in the network code while operating on the connection to the client. The exception is caused by some logging code that is checking if trace level logging is enabled. It looks like the version of log4j you are running with doesn't have org.apache.log4j.Level.TRACE defined. You may have multiple log4j jars in your classpath?
log4j hasn't changed in a long time so it is strange that you have a version that doesn't define that field.
The reason it appears to happen under load is that the trace statement is in code that is only run when the server is flooded with requests and decides to stop reading new ones until the existing ones are processed. That is the backpressure being referred to.
-Ariel


That was it - for whatever reason, we were using log4j 1.2.9. Updating to 1.2.16 solved the problem, thanks!
Samuel
Oct 7, 2014
Hi all,

I am encountering the similar problem.

I am using voltdb version 3.5.0.1 and was encountering a problem where the client application ran into some connection error and keep printing "Connection to host (ip of one of the 3 nodes) was lost before a response was received". While this is being printed, the client application continued to clear the messages that are incoming, but it did not produce the expected results (Possibly due to the lack of 1 node of data?). This does not happen all the time, usually after a few days of running.

The post pointed to an outdated log4j library, but I have checked that I am already using the updated version (1.2.16) that came with the voltdb package.

Hope someone in the forum could advise me on this. Thanks.

Regards,
Samuel