No announcement yet.

C++ client with multiple nodes

  • Filter
  • Time
  • Show
Clear All
new posts

  • C++ client with multiple nodes


    In the readme file of the cpp client the V4 release notes says: 'When connected to multiple nodes in the cluster, the client library will enable client affinity and can auto-reconnect to restarted nodes.'.

    I wanted to try out these features but I kinda stuck. I was using VoltDB v5.6 and cpp client v5.2. My problems are:
    • If I use the createConnection() method it is establishing the connections to the nodes. Then if a set the client affinity true, it is working, but I have to set it manually. From the release notes I assumed that it will happen automatically. I was trying to find the actual code in the client's source which do some automatic client affinity settings but I probably was not thorough. Maybe I misunderstood the feature?
    • The createConnection() method needs a running/initialized VoltDB server when it is called and based on my observations it does not do any auto-reconnecting. If I use the createPendingConnection() method it will wait until the VoltDB starts up at the given host and can auto-reconnect to it but if I call it e.g. twice for two nodes then my program either aborts or the client throws an OverflowUnderflowException. What am I missing?

    Any advice would be greatly appreciated,
    Last edited by RolandM; 10-06-2015, 01:46 PM.

  • #2

    I'm Bill White, an engineer at VoltDB. I'd like to clarify your first point, and ask for details about the second point.
    1. You appear to be right about the default setting for client affinity. It seems as if client affinity is disabled by default. There was a lot of churn in this area going from version 4 to version 5, so perhaps this got lost. It doesn't seem to be in the test suite, but I may have missed it.
    2. I'm looking into the differences between Client::createConnection and Client::createPendingConnection. In particular, I'm trying to figure out how these two do reconnection. I had thought the former is exactly like the latter, except that it waits for the connection to be established, but I'm not completely certain now.
    3. I want to make sure about your second question. You try to create two pending connections to two separate hosts with no database running, then start the database on those hosts. Sometime after the databases are initialized the client applications crash, sometimes with an OverflowUnderflowException and sometimes with an abort. Are the connections in different threads? The C++ client is not thread safe, so if they are in different threads of the same Unix process hilarious things, like exceptions, may easily occur. Are the connections in the same or different Unix processes? Do you have a stack frame? I will try to reproduce this with a couple different scenarios.

      I think your expectation is that two client connections in two different Unix processes, or two client connections in the same thread should be able to connect to two different hosts reliably. Is that right? That sounds completely reasonable to me.

    Thanks for the note. I'll generate a trouble ticket about the documentation.


    • #3
      Hi Bill,

      Thank you for the quick response.

      Looking at the code again you are probably right. The createConnection() method is also using the PendingConnection object for the initialization. It seems I made some hasty assumptions about their behavior. I will run my tests again and I will get back with the results.

      Note: You get my scenario right. I am having a VoltDB cluster with two nodes and my single threaded application is establishing the (right now not pending) connections to each of these nodes. Then I run test cases like:
      • No database running at the given addresses when the application starts
      • The given addressess are not accessible
      • Only one of the given address is accessible
      • One of the node restarts after the connection is established

      Thanks again,


      • #4

        I managed to re-run some tests and make some debugging.

        So in my code I wrote the following:

        client.createConnection(host1,21212);client.create Connection(host2,21212); //hostx is an IP address (std::string)
        If VoltDB is initialized it is working fine.

        If I replace that with:
        client.createPendingConnection(host1,21212);client .createPendingConnection(host2,21212); then I got a 'voltdb::OverflowUnderflowException' even if VoltDB is initialized.

        #0 0x00007ffff505b5d7 in raise () from /lib64/
        #1 0x00007ffff505ccc8 in abort () from /lib64/
        #2 0x00007ffff595f9b5 in __gnu_cxx::__verbose_terminate_handler() () from /lib64/
        #3 0x00007ffff595d926 in ?? () from /lib64/
        #4 0x00007ffff595d953 in std::terminate() () from /lib64/
        #5 0x00007ffff595db73 in __cxa_throw () from /lib64/
        #6 0x00007ffff6cc0c95 in voltdb::ByteBuffer::checkGetPutIndex (this=<optimized out>, this=<optimized out>, length=<optimized out>) at include/ByteBuffer.hpp:104
        #7 0x00007ffff6cc9d17 in checkGetPutIndex (this=<optimized out>, this=<optimized out>, length=<optimized out>) at include/ByteBuffer.hpp:266
        #8 getByReference (length=<optimized out>, this=0x7fffffffe0c0) at include/ByteBuffer.hpp:435
        #9 voltdb::ByteBuffer::getString (this=this@entry=0x7fffffffe0c0, wasNull=@0x7fffffffe0a0: false) at include/ByteBuffer.hpp:264
        #10 0x00007ffff6cc842e in AuthenticationResponse (buf=..., this=0x7fffffffe0f0) at include/AuthenticationResponse.hpp:43
        #11 voltdb::authenticationReadCallback (bev=0x65c7a0, ctx=0x6531c0) at src/ClientImpl.cpp:139
        #12 0x00007ffff6cfd662 in bufferevent_readcb (fd=23, event=<optimized out>, arg=0x65c7a0) at bufferevent_sock.c:183
        #13 0x00007ffff6cf557d in event_process_active_single_queue (activeq=<optimized out>, base=<optimized out>) at event.c:1340
        #14 event_process_active (base=<optimized out>) at event.c:1407
        #15 event_base_loop (base=0x652880, flags=2) at event.c:1604
        #16 0x00007ffff6cc3916 in voltdb::ClientImpl::runOnce (this=0x652540) at src/ClientImpl.cpp:742
        #17 0x00007ffff6cc1651 in voltdb::Client::runOnce (this=<optimized out>) at src/Client.cpp:83



        • #5
          Sounds like a bug to me.

          Thanks. That certainly sounds like a bug. The client should not crash like this, or any way really.

          I'll submit a ticket and we can track progress on this.

          I'll also try to reproduce it here and see if I can shed any more light on it.

          Thanks again.


          • #6
            Can you please tell me the versions of Voltdb and the C++ client you are using? Also, is it possible for you to send me your reproducer code?

            We have one particular customer who uses the C++ client quite extensively, and has done what you are trying to do. It would be interesting to us to see what you are doing differently.



            • #7
              Sorry, I just saw the version numbers in the original posting.

              It would still help us if you could post the code and the VDB application code which is causing the error.


              • #8
                My actual code is:

                int main(int argc, char* argv[])
                std::vector<std::string> hosts;
                for (auto i = 1; i < argc; ++i)

                auto client=voltdb::Client::create();

                for (const auto& host : hosts)

                while (!_terminate) {
                handling some incoming traffic

                while (!client.drain()) {}

                return 0;

                Thanks in advance.
                Last edited by RolandM; 10-15-2015, 10:07 AM.


                • #9
                  Note: The code above is flawless with Client::createConnection


                  • #10
                    Thanks. I'll try this out.


                    • #11
                      For the record, the ticket number for this issue is ENG-9256. The link to the ticket is here.


                      • #12

                        I've seen that you resolved the issue. Thanks, I will try that out. Could you tell me when do you plan to create an official cpp client release with it?

                        Thanks again,


                        • #13
                          The latest c++ client can be obtained from at your leisure.
                          Peter Zhao


                          • #14
                            Thanks Peter. Though maybe I was not clear enough. The code is on the master branch now and I wanted to ask that do you plan to create a github release soon?


                            • #15
                              I apologize, we do have a release that can be found here:
                              Please expect an update early next quarter.
                              Peter Zhao
                              Last edited by pzhao; 12-03-2015, 03:54 PM.