Forum: Building VoltDB Applications

Post: connection limit

connection limit
Vic
Sep 6, 2012
We limit the maximum number of connections to the database - the starting value is 800 and should be adjusted to your rlimit - 300; we do this to protect the database process from exhausting its file descriptor limit which could lead to failures to log important user data.
You can see the code here: https://github.com/VoltDB/voltdb/blob/master/src/frontend/org/voltdb/ClientInterface.java#L278
You should be able to increase your OS's fd limit and see the corresponding increase in accepted connections to the database.
https://github.com/VoltDB/voltdb/blob/master/src/frontend/org/voltdb/ClientInterface.java#L2041


Hi Rbetts,
Thanks for you reply. I change the limit to 10K for my test and with 3K client voltdb reject the others clients.
The log of the server is:
2012-09-07 17:03:52,989 WARN [Heartbeat] NETWORK: Closing connection to org.voltcore.network.VoltPort@1f707a6a:localhost/127.0.0.1:41731 at Fri Sep 07 17:03:52 GMT+00:00 2012 because it refuses to read responses
2012-09-07 17:03:52,989 WARN [Heartbeat] NETWORK: Closing connection to org.voltcore.network.VoltPort@cda8b1d:localhost/127.0.0.1:41733 at Fri Sep 07 17:03:52 GMT+00:00 2012 because it refuses to read responses
2012-09-07 17:03:52,989 WARN [Heartbeat] NETWORK: Closing connection to org.voltcore.network.VoltPort@4dc6640c:localhost/127.0.0.1:41751 at Fri Sep 07 17:03:52 GMT+00:00 2012 because it refuses to read responses
Do you know if is necesarry make a change in VoltPort to increase the limit?
Thanks
connection limit
Vic
Sep 6, 2012
Hi,
I do a test of concurrency with a client in php. I create 775 connections to a database and voltdb reject the other clients that try to connect.
The log of the server is :
WARN: Rejected connection from /127.0.0.1:55986 because the connection limit of 800 has been reached
WARN: Rejected connection from /127.0.0.1:55987 because the connection limit of 800 has been reached
WARN: Rejected connection from /127.0.0.1:60353 because the connection limit of 800 has been reached
WARN: Rejected connection from /127.0.0.1:60354 because the connection limit of 800 has been reached
WARN: Rejected connection from /127.0.0.1:60355 because the connection limit of 800 has been reached
WARN: Rejected connection from /127.0.0.1:60356 because the connection limit of 800 has been reached
WARN: Rejected connection from /127.0.0.1:60357 because the connection limit of 800 has been reached
WARN: Rejected connection from /127.0.0.1:60358 because the connection limit of 800 has been reached
WARN: Rejected connection from /127.0.0.1:60359 because the connection limit of 800 has been reached
Does Anybody know how change it?
Thanks
We limit the maximum number
rbetts
Sep 6, 2012
We limit the maximum number of connections to the database - the starting value is 800 and should be adjusted to your rlimit - 300; we do this to protect the database process from exhausting its file descriptor limit which could lead to failures to log important user data.
You can see the code here: https://github.com/VoltDB/voltdb/blob/master/src/frontend/org/voltdb/ClientInterface.java#L278
You should be able to increase your OS's fd limit and see the corresponding increase in accepted connections to the database.
https://github.com/VoltDB/voltdb/blob/master/src/frontend/org/voltdb/ClientInterface.java#L2041
Refuses to read responses..
rbetts
Sep 7, 2012
Hi Rbetts,
Thanks for you reply. I change the limit to 10K for my test and with 3K client voltdb reject the others clients.
The log of the server is:
2012-09-07 17:03:52,989 WARN [Heartbeat] NETWORK: Closing connection to org.voltcore.network.VoltPort@1f707a6a:localhost/127.0.0.1:41731 at Fri Sep 07 17:03:52 GMT+00:00 2012 because it refuses to read responses
2012-09-07 17:03:52,989 WARN [Heartbeat] NETWORK: Closing connection to org.voltcore.network.VoltPort@cda8b1d:localhost/127.0.0.1:41733 at Fri Sep 07 17:03:52 GMT+00:00 2012 because it refuses to read responses
2012-09-07 17:03:52,989 WARN [Heartbeat] NETWORK: Closing connection to org.voltcore.network.VoltPort@4dc6640c:localhost/127.0.0.1:41751 at Fri Sep 07 17:03:52 GMT+00:00 2012 because it refuses to read responses
Do you know if is necesarry make a change in VoltPort to increase the limit?
Thanks


This is actually a different error from your first post. In this case, you have a client that created a request on the server but did not read the response from the server. In this case, we limit the number of bytes of responses a client can queue without reading and kick-out clients that aren't reading their data. We do this to prevent the JVM from OOMing as a result of badly implemented clients.
PHP Fatal error
Vic
Sep 9, 2012
This is actually a different error from your first post. In this case, you have a client that created a request on the server but did not read the response from the server. In this case, we limit the number of bytes of responses a client can queue without reading and kick-out clients that aren't reading their data. We do this to prevent the JVM from OOMing as a result of badly implemented clients.


Hi Rbetts,
I change my client to read the response and it work better but now each client use 20MB of RAM. This is a big problem because with 3000 clients use all memory of the server. Do you know the reason of this problem?
Thanks,
Which PHP client?
rbetts
Sep 10, 2012
Hi Rbetts,
I change my client to read the response and it work better but now each client use 20MB of RAM. This is a big problem because with 3000 clients use all memory of the server. Do you know the reason of this problem?
Thanks,


Are you using the new PHP client that was recently announced? Are you running PHP from the command line or hosted within Apache?
PHP Client
Vic
Sep 11, 2012
Are you using the new PHP client that was recently announced? Are you running PHP from the command line or hosted within Apache?


I am running from the command line and use the client for php 5.3 (http://voltdb.com/products-services/downloads).
I do some changes to my client. Now it use only the volt class and a log. Now use 4,5 MB per client.
3.4 MiB + 1.8 MiB = 5.3 MiB magent
5.2 MiB + 912.0 KiB = 6.1 MiB postgres (5)
6.6 MiB + 1.2 MiB = 7.8 MiB bash (28)
8.1 MiB + 788.0 KiB = 8.9 MiB Xvnc
4.2 MiB + 7.2 MiB = 11.5 MiB httpd (12)
12.0 MiB + 1.3 MiB = 13.3 MiB cvd
37.6 MiB + 1.0 MiB = 38.6 MiB dsm_om_connsvcd (2)
220.7 MiB + 4.4 MiB = 225.1 MiB savscand (2)
227.7 MiB + 848.0 KiB = 228.5 MiB cpan
2.4 GiB + 2.1 MiB = 2.4 GiB java (3)
8.9 GiB + 5.9 MiB = 8.9 GiB php (2000)
Private + Shared = RAM used Program
Is it normal?. How can i reduce it?
Application layout
awilson
Sep 11, 2012
I am running from the command line and use the client for php 5.3 (http://voltdb.com/products-services/downloads).
I do some changes to my client. Now it use only the volt class and a log. Now use 4,5 MB per client.
3.4 MiB + 1.8 MiB = 5.3 MiB magent
5.2 MiB + 912.0 KiB = 6.1 MiB postgres (5)
6.6 MiB + 1.2 MiB = 7.8 MiB bash (28)
8.1 MiB + 788.0 KiB = 8.9 MiB Xvnc
4.2 MiB + 7.2 MiB = 11.5 MiB httpd (12)
12.0 MiB + 1.3 MiB = 13.3 MiB cvd
37.6 MiB + 1.0 MiB = 38.6 MiB dsm_om_connsvcd (2)
220.7 MiB + 4.4 MiB = 225.1 MiB savscand (2)
227.7 MiB + 848.0 KiB = 228.5 MiB cpan
2.4 GiB + 2.1 MiB = 2.4 GiB java (3)
8.9 GiB + 5.9 MiB = 8.9 GiB php (2000)
Private + Shared = RAM used Program
Is it normal?. How can i reduce it?


Hi Vic,
I've been following your thread and I am hoping that I can give you a bit of a hand, though I think I need a clear understanding of the application and the way you are testing it.
Let's start out with the basics.
Are you running a command line php app like "php foo.php" to do your testing or are you running a web application? What are you using to drive your application?
If you are running a web application, what is your application stack (apache, nginx, php, fascgi, apc, etc.)?
How many volt servers are you trying to connect to? We usually run in clusters so you could have a several machines.
Are you reconnecting to volt on each invocation of your script or are you using a global instance or cached instance of the driver?
When does the memory allocation being to escalate? In other words, you could be just running a connect and then closing the connection with no queries and getting this memory footprint. Running a query could return a large result and then we need to take a look at the size of the result.
I realize that these are a lot of questions, but you are making a great effort to evaluate volt and I want to get you the answers you are looking for as efficiently as possible and the more we know about your use case or app then the better we can help you.
Thanks,
Andy
My Application
Vic
Sep 11, 2012
Hi Vic,
I've been following your thread and I am hoping that I can give you a bit of a hand, though I think I need a clear understanding of the application and the way you are testing it.
Let's start out with the basics.
Are you running a command line php app like "php foo.php" to do your testing or are you running a web application? What are you using to drive your application?
If you are running a web application, what is your application stack (apache, nginx, php, fascgi, apc, etc.)?
How many volt servers are you trying to connect to? We usually run in clusters so you could have a several machines.
Are you reconnecting to volt on each invocation of your script or are you using a global instance or cached instance of the driver?
When does the memory allocation being to escalate? In other words, you could be just running a connect and then closing the connection with no queries and getting this memory footprint. Running a query could return a large result and then we need to take a look at the size of the result.
I realize that these are a lot of questions, but you are making a great effort to evaluate volt and I want to get you the answers you are looking for as efficiently as possible and the more we know about your use case or app then the better we can help you.
Thanks,
Andy


Hi Andy,
Thanks for help me. I am very interesting to use voltdb for a high recurrency app.
I did a script in linux to test recurrency. Mi script open ten connections per second and send it to background. It is like to "php foo.php &" ten times per second. I did it to use one console to make the test.
For this test i have:
PHP Version 5.3.3
Apache 2.0
LINUX voltdb 2.8.1
One voltdb server (I will use more servers when i have finished this test)
I create a new connection for each new client. All client execute the same procedure. All the connections are open and never close.
Thanks again,
Vic
My Application
awilson
Sep 11, 2012
Hi Andy,
Thanks for help me. I am very interesting to use voltdb for a high recurrency app.
I did a script in linux to test recurrency. Mi script open ten connections per second and send it to background. It is like to "php foo.php &" ten times per second. I did it to use one console to make the test.
For this test i have:
PHP Version 5.3.3
Apache 2.0
LINUX voltdb 2.8.1
One voltdb server (I will use more servers when i have finished this test)
I create a new connection for each new client. All client execute the same procedure. All the connections are open and never close.
Thanks again,
Vic


Hi Vic,
This test is an interesting approach however it may not give you the results that you are really looking for. Another way to test this is to use curl calling a url that has your test case. This will simulate your application's latency and throughput behaviors like your final application.
Next, take a look at APC. You can cache the voltclient object within APC and maintain your connection to the volt cluster. This will lower your page latency and memory requirements for maintaining connections as you will be reusing the existing connections.
Thanks,
Andy
APC with Voltdb
Vic
Sep 14, 2012
Hi Vic,
This test is an interesting approach however it may not give you the results that you are really looking for. Another way to test this is to use curl calling a url that has your test case. This will simulate your application's latency and throughput behaviors like your final application.
Next, take a look at APC. You can cache the voltclient object within APC and maintain your connection to the volt cluster. This will lower your page latency and memory requirements for maintaining connections as you will be reusing the existing connections.
Thanks,
Andy


Hi Andy,
I have completed the instalation of APC. The object Client is recover for the others clients but the object donīt work. Have you test APC with Voltdb?
Thanks,
Vic
APC with Voltdb
awilson
Sep 14, 2012
Hi Andy,
I have completed the instalation of APC. The object Client is recover for the others clients but the object donīt work. Have you test APC with Voltdb?
Thanks,
Vic


Hi Vic,
Have you switched to using web requests, through ab or curl, or are you running the same command line script as before?
What is the error that you get when you load the APC cached object?
Thanks,
Andy
APC with Voltd
Vic
Sep 15, 2012
Hi Vic,
Have you switched to using web requests, through ab or curl, or are you running the same command line script as before?
What is the error that you get when you load the APC cached object?
Thanks,
Andy


Hi Andy
Yes, i have switched to web request. Now all the clients use curl and less memory that before. The error with APC is:
[Sat Sep 15 00:37:33 2012] [error] [client 127.0.0.1] PHP Fatal error: No matching function for overloaded 'ClientNative_invoke' in /voltdb/test/voltdb.php on line 2309
Thanks,
Vic
New PHP driver
nshi
Sep 15, 2012
Hi Andy
Yes, i have switched to web request. Now all the clients use curl and less memory that before. The error with APC is:
[Sat Sep 15 00:37:33 2012] [error] [client 127.0.0.1] PHP Fatal error: No matching function for overloaded 'ClientNative_invoke' in /voltdb/test/voltdb.php on line 2309
Thanks,
Vic


Hi Vic,
voltdb.php is part of the old PHP driver. If you are using voltdb.so from the new PHP driver, please remove voltdb.php from /voltdb/test/.
The API for the new PHP driver has changed substantially from the old one. For an example of how to use the new driver, you can take a look at this example code and the API reference.
Ning
Hi, Also keep in mind that
aweisberg
Sep 17, 2012
Hi Vic,
voltdb.php is part of the old PHP driver. If you are using voltdb.so from the new PHP driver, please remove voltdb.php from /voltdb/test/.
The API for the new PHP driver has changed substantially from the old one. For an example of how to use the new driver, you can take a look at this example code and the API reference.
Ning


Hi,
Also keep in mind that connection pooling only works if the PHP process is reused. If you start PHP processes in a loop each one will create a fresh connection. The server isn't going to see the clients disconnecting immediately which means you can run out of permits even if you only have N clients at any given time.
Ariel
Conexion php
Vic
Sep 18, 2012
Hi,
Also keep in mind that connection pooling only works if the PHP process is reused. If you start PHP processes in a loop each one will create a fresh connection. The server isn't going to see the clients disconnecting immediately which means you can run out of permits even if you only have N clients at any given time.
Ariel


Thanks for your recomendation.
Could you help me with some problems? please
How can i do to execute some procedures in the same php file?
I run mi test again and the client show "Exception 11" and the log of the server show:
WARN: Rejected accepting new connection due to too many open files
WARN: Rejected accepting new connection due to too many open files
WARN: Rejected accepting new connection due to too many open files
WARN: Rejected accepting new connection due to too many open files
WARN: Rejected accepting new connection due to too many open files
WARN: Rejected accepting new connection due to too many open files
WARN: Rejected accepting new connection due to too many open files
This error ocurre with 1156 concurrent clients.
Vic
Hi Vic, Can you send the code
aweisberg
Sep 19, 2012
Thanks for your recomendation.
Could you help me with some problems? please
How can i do to execute some procedures in the same php file?
I run mi test again and the client show "Exception 11" and the log of the server show:
WARN: Rejected accepting new connection due to too many open files
WARN: Rejected accepting new connection due to too many open files
WARN: Rejected accepting new connection due to too many open files
WARN: Rejected accepting new connection due to too many open files
WARN: Rejected accepting new connection due to too many open files
WARN: Rejected accepting new connection due to too many open files
WARN: Rejected accepting new connection due to too many open files
This error ocurre with 1156 concurrent clients.
Vic


Hi Vic,
Can you send the code you are trying to run to
support@voltdb.com. That is probably the easiest way to figure out what you are doing.
That warning is printed when ServerSocket.accept throws an IOException with the message too many open files. It's a little surprising because the server is supposed to avoid running into that limit by accepting fewer connections then the open file limit, but FDs might stay in use for a short time after being closed.
You can invoke multiple procedures from inside a PHP script by invoking another script in a loop and pulling a connection from the connection pool in the script. Are you using the new PHP client? It does connection pooling transparently.
Ariel
Compile volt
Vic
Sep 14, 2012
Hi Rbetts,
I have compiled the jars of voltdb. I change the limit to 10000. When i connect 7000 clients this happens:
WARN: Rejected connection from /127.0.0.1:47889 because the connection limit of 3796 has been reached
WARN: Rejected connection from /127.0.0.1:47890 because the connection limit of 3796 has been reached
WARN: Rejected connection from /127.0.0.1:47891 because the connection limit of 3796 has been reached
And the othes clients are reject.
concurrency test
Vic
Sep 7, 2012
Do you know if voltdb use persistent connection?
Persistent connections
rbetts
Sep 7, 2012
Do you know if voltdb use persistent connection?


Are you using the HTTP interface to volt? We haven't tested that much with persistent connections (though that has long been on our want-to-do list). Most people use native volt client drivers and not the HTTP interface. The native drivers (java, .NET, C++, python, php, ...) are much more performant than the HTTP interface (especially java, .net, c++).
All of those native drivers use a persistent connection to the database -- they multiplex lots of stored procedure invocations over a single socket.
Ryan.