Forum: Installation

Post: VoltDB cluster Error

VoltDB cluster Error
mauso
Apr 26, 2011
hi,
I am install cluster, I get this error,
I have not password on either side of the configuration.


ERROR
------------------------------
1310 [main] INFO HOST - HTTP admin console unable to bind to port 8080. Exiting.
java.io.IOException: Unauthenticated connection was rejected due to too many recent rejected attempts. Wait 1 seconds and try again.
at org.voltdb.client.AuthenticatedConnectionCache.getClient(AuthenticatedConnectionCache.java:155)
at org.voltdb.HTTPClientInterface.process(HTTPClientInterface.java:168)
at org.voltdb.utils.HTTPAdminListener$RequestHandler.handle(HTTPAdminListener.java:78)
at org.eclipse.jetty_voltpatches.server.handler.HandlerWrapper.handle(HandlerWrapper.java:113)
at org.eclipse.jetty_voltpatches.server.Server.handle(Server.java:352)
at org.eclipse.jetty_voltpatches.server.HttpConnection.handleRequest(HttpConnection.java:595)
at org.eclipse.jetty_voltpatches.server.HttpConnection$RequestHandler.content(HttpConnection.java:1060)
at org.eclipse.jetty_voltpatches.http.HttpParser.parseNext(HttpParser.java:764)
at org.eclipse.jetty_voltpatches.http.HttpParser.parseAvailable(HttpParser.java:217)
at org.eclipse.jetty_voltpatches.server.HttpConnection.handle(HttpConnection.java:425)
at org.eclipse.jetty_voltpatches.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:241)
at org.eclipse.jetty_voltpatches.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:436)
at java.lang.Thread.run(Thread.java:636)
{"status":-3,"appstatus":-128,"statusstring":"Unauthenticated connection was rejected due to too many recent rejected attempts. Wait 1 seconds and try again.","appstatusstring":
THE deployment.xml NODE 1
------------------------------
<?xml version="1.0"?>
THE deployment.xml NODE 2
------------------------------
<?xml version="1.0"?>

THANKS
THE deployment.xml NODE
mauso
Apr 26, 2011
THE deployment.xml NODE 1
------------------------------
<?xml version="1.0"?>
<deployment>
<httpd>
<jsonapi enabled="true"/>
</httpd>

<snapshot prefix="prueba" frequency="2m" retain="10" enabled="true"/>

<cluster hostcount="2"
sitesperhost="1"
leader="127.0.0.1" />

<partition-detection enabled="true">
<snapshot prefix="netfaultsave"/>
</partition-detection>

<admin-mode adminstartup="false"/>

</deployment>




THE deployment.xml NODE 2
------------------------------
<?xml version="1.0"?>
<deployment>
<httpd>
<jsonapi enabled="true"/>
</httpd>

<snapshot prefix="prueba" frequency="2m" retain="10" enabled="true"/>

<cluster hostcount="2"
sitesperhost="1"
leader="172.16.155.180" />

<partition-detection enabled="true">
<snapshot prefix="netfaultsave"/>
</partition-detection>

<admin-mode adminstartup="false"/>

</deployment>
Something else is using port 8080
sebc
Apr 26, 2011
Hi Mauso,
From the log detail message: "HTTP admin console unable to bind to port 8080. Exiting." it looks like something else on your server is already using port 8080, which is why you get into this problem.


You can change the JSON port this way:


<deployment>
<httpd port="8081">
<jsonapi enabled="true"/>
</httpd>
...



Hope this helps,
Seb
Deployment files do not match
sebc
Apr 26, 2011
Mauso,


Another thing I had not noticed earlier is that your two deployment files do not match on each node. I do not get the same error here, but definitely the database will not start in this case. Right now you have:

Node 1: leader="127.0.0.1"
Node 2: leader="172.16.155.180"


Both files should point the the same IP address (and not localhost), so here, you probably mean to be using 'leader="172.16.155.180"'.


Keep in mind that your catalog and deployment files need to be exactly identical on all your nodes for a cluster deployment. Indeed, one easy way to go about it is to use a NFS mount, or have a replication system in place to make sure this is always the case).

Seb
Change xml file
mauso
Apr 27, 2011
Mauso,

Another thing I had not noticed earlier is that your two deployment files do not match on each node. I do not get the same error here, but definitely the database will not start in this case...


Ok, the xml is


<deployment>
<httpd port="8090">

<jsonapi enabled="true"/>
</httpd>

<snapshot prefix="prueba" frequency="2m" retain="10" enabled="true"/>

<cluster hostcount="2"
sitesperhost="1"
leader="172.16.155.180"
kfactor="1"
/>

<partition-detection enabled="true">
<snapshot prefix="netfaultsave"/>
</partition-detection>

<admin-mode adminstartup="false"/>



The system is 2 node cluster,
consider some configuration?

Thanks
Unauthenticated connection was rejected
mauso
Apr 27, 2011
The node 2 says:


java.io.IOException: Unauthenticated connection was rejected due to too many recent rejected attempts. Wait 1 seconds and try again.
at org.voltdb.client.AuthenticatedConnectionCache.getClient(AuthenticatedConnectionCache.java:155)
at org.voltdb.HTTPClientInterface.process(HTTPClientInterface.java:168)
at org.voltdb.utils.HTTPAdminListener$RequestHandler.handle(HTTPAdminListener.java:78)
at org.eclipse.jetty_voltpatches.server.handler.HandlerWrapper.handle(HandlerWrapper.java:113)
at org.eclipse.jetty_voltpatches.server.Server.handle(Server.java:352)
at org.eclipse.jetty_voltpatches.server.HttpConnection.handleRequest(HttpConnection.java:595)
at org.eclipse.jetty_voltpatches.server.HttpConnection$RequestHandler.content(HttpConnection.java:1060)
at org.eclipse.jetty_voltpatches.http.HttpParser.parseNext(HttpParser.java:764)
at org.eclipse.jetty_voltpatches.http.HttpParser.parseAvailable(HttpParser.java:217)
at org.eclipse.jetty_voltpatches.server.HttpConnection.handle(HttpConnection.java:425)
at org.eclipse.jetty_voltpatches.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:241)
at org.eclipse.jetty_voltpatches.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:436)
at java.lang.Thread.run(Thread.java:636)



but file project.xml says


<?xml version="1.0"?>
<project>
<security enabled="false"/>
<database>
<schemas>
<schema path='prueba.sql' />
</schemas>
<procedures>
<procedure class='insert' />
<procedure class='select' />
<procedure class='update' />
<procedure class='delete' />
<procedure class='getNextUniqueID' />
<procedure class='incrementSales' />
<procedure class='decrementSales' />
</procedures>
<partitions>
<partition table='PRODUCTSALES' column='PRODUCTID' />
<partition table='SEQUENCE' column='TABLE_NAME' />
</partitions>
</database>
</project>
Client app?
sebc
Apr 27, 2011
Mauso,
I need a little more information from you on this:
- Are you using the current version of VoltDB (1.3.01)?
- Can you share a bit about your client application? Clearly, from the message, it looks like you're using the JSON but somehow flooding the interface and getting rejected - does *anything* work? What language are you using to submit your requests (Python, Java, etc.)
Seb
we using php and JSON the
mauso
Apr 28, 2011
Mauso,
I need a little more information from you on this:
- Are you using the current version of VoltDB (1.3.01)?
- Can you share a bit about your client application? Clearly, from the message, it looks like you're using the JSON but somehow flooding the interface and getting rejected - does *anything* work? What language are you using to submit your requests (Python, Java, etc.)
Seb


we using php and JSON


the version is 1.3.01


now we have problem with 3 node
Maximum clock/network skew is 5 milliseconds (according to leader)
java.lang.Thread.dumpThreads(Native Method)
java.lang.Thread.getAllStackTraces(Thread.java:1503)
org.voltdb.VoltDB.crashVoltDB(VoltDB.java:353)
org.voltdb.messaging.SocketJoiner.runNonPrimary(SocketJoiner.java:523)
org.voltdb.messaging.SocketJoiner.run(SocketJoiner.java:193)
VoltDB has encountered an unrecoverable error and is exiting.
The log may contain additional information.


but we don't get or set the time using NTP, because 5 milliseconds is very little

the server's virtual

can you help me?
Mauso, Well, I am trying (to
sebc
Apr 28, 2011
Mauso,
Well, I am trying (to help you) :)
5ms skew isn't your issue (maximum acceptable is 100ms and 5ms is considered good).
Could you get me the *full log* on all 3 nodes - that will get me more physical details about your VMs and might point me in a specific direction.
Cheers,
Seb
the problem is file
mauso
Apr 29, 2011
Mauso,
Well, I am trying (to help you) :)
5ms skew isn't your issue (maximum acceptable is 100ms and 5ms is considered good).
Could you get me the *full log* on all 3 nodes - that will get me more physical details about your VMs and might point me in a specific direction.
Cheers,
Seb


the problem is file configure, is solved.
currently we have 3 servers connected
but we detected that:


-if we reboot the master not re-connect with the other 2 server
-if restart any of the other two servers not reconect to the master
are left with the message:


93 [main] INFO HOST - Build: 1.3.01 https://svn.voltdb.com/eng/tags/voltdb-1.3.01?revision=1723
93
[main] INFO HOST - Loading application catalog jarfile from /opt/voltdb/GROUPALIA/groupalia.jar
437 [main] INFO HOST - URL of deployment info: deployment.xml
689 [main] INFO HOST - Cluster has 3 hosts with leader hostname: "172.16.155.180". 1 sites per host. K = 1.
689 [main] INFO HOST - The entire cluster has 2 copies of 1 logical partition.
691 [main] INFO HOST - Detection of network partitions in the cluster is enabled.
691 [main] INFO HOST - Using "/opt/voltdb/prueba/voltdbroot" for voltdbroot directory.
1276 [main] INFO HOST - Beginning inter-node communicaton on port 3021.
1285 [Thread-13] INFO HOST - Connecting to VoltDB cluster as the leader...



the solution we've found is to restart the whole cluster.

But we think this not be correct
Rejoin operation
sebc
Apr 29, 2011
Good to hear your original problem was fixed.
As for the node restart: all you need is to "rejoin" the node you restarted. Keep in mind the "leader" node is only considered a "master" at the very beginning of the cluster's lifetime: at initialization. Once this is complete, there is no longer a "master" in the cluster and all nodes are actually peers, such that any failed node can be rejoined to any live node remaining in the cluster. The "Using VoltDB" manual will give you more details about this - specifically, have a look here:


[TD="class: xl66, width: 467"]http://www.voltdb.com/community/documentation.php



Hope this helps,
Seb