VoltDB 3.6: JSON interface: No connections.
Feb 16, 2014
we are using 3.6 community edition only via the JSON interfrace. After running for rdb. two weeks, the JSON clients do not get any responses any more, and the log contains this message hundreds of times:
JSON interface: No connections.
After the usual pause+save+shutdown+create+restore sequence, the system is back up agains, but obviously, there was some outage and this is just a coarse mitigation.
Any ideas on how to prevent the JSON interface running out of connections?
Thank you in advance.
Feb 17, 2014
It may be a problem with connection/error handling in the HTTP/JSON code? I found a couple of tickets, https://issues.voltdb.com/browse/ENG-5545
, being one, that may account for what you are seeing. Do you see any other errors in the log?
Feb 18, 2014
thanks for your response. I have issues accessing the issue you cited from JIRA; concerning your point, I found some other warnings in the log - namely lines like this:
HOST: JSON interface: Incorrect authorization credentials.
I guessed this comes from some unmanaged user, but by itself does not indicate a problem to me.
But do you think this is related to the issue?
Feb 19, 2014
Not sure if that is related - but if we can have a reproducer on the issue, we can take a deeper look at it.
From looking at the code (https://github.com/VoltDB/voltdb/blob/master/src/frontend/org/voltdb/HTTPClientInterface.java
it looks like the cached client connection may be being cleaned up underneath the HTTP client. When you hit this situation, try connecting with a new JSON request with a different/new username/pwd, and see if that creates a new working connection (that would test this theory).
Feb 27, 2014
thanks for your ideas. Actually, since the problem only occurs in production, and only sporadic it's a little difficult to analyse.
In the last days, I tried to reproduce this within our test environment via massively creating authentication errors via incorrect credentials, and successful JSON requests in parallel (using SoapUI load tests). But actually, no OutOfConnections error appeared..
I get the impression that the the incorrect credentials are not origin of the issue. The question remains what it could be..
I'll provide an update as soon as I've found something.
Any additional ideas anyone?
Feb 27, 2014
I've done some code review and system testing of our JSON/HTTP interface. Here's what I've found: When you specify a username (credentials), the JSON/HTTP interface caches the VoltDB client session and re-uses it for subsequent matching-credentialed requests. From code review, it appears that there's an issue with a client request, it is possible that the connection within the client session gets closed but leaves the client session still in the cache. I'm trying to figure out what "issue" could occur to make that happen, but have been unsuccessful so far. Would love a reproducer, if you happen to find one.
Jun 26, 2014
finally I was able to dig deeper. Namely, I created a second user account at one breakdown. At the latest today, I checked whether this user can still query via the JSON interface while the standard user cannot anymore. And: it's true. A second user can still use the JSON interface.
So, after validating this: is this a bug somewhere? Do have any ideas on what we could do?
Thanks in advance,
Jun 27, 2014
Is this on v3.6 or the latest, v4.4? I think this problem may have been fixed this past March.
Jul 2, 2014
thanks for the hint. You are right: we are still on 3.6.
From what you say, I think we can hope that it is resolved with the new version.
We will upgrade and check.