Announcement

Collapse
No announcement yet.

SQLCMD it's not responding

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • SQLCMD it's not responding

    When I'm executing the sqlcmd client, like I always do, I don't get a prompt and can't use it.

  • #2
    albert,
    Could you provide the following information:
    1) What version you are running?
    2) Can you confirm voltdb running?
    3) What does the logs say, volt.log?
    4) Can you access VMC and run a simple sql query?
    Peter Zhao

    Comment


    • #3
      1) 5.4
      2) voltdb is running
      3) Nothing.
      4) I can access VMC and I ran a simple query and worked fine.

      Comment


      • #4
        albert,
        Can you show me your deployment file, command line startup command, and the output of 'netstat -antplo'?
        Peter Zhao

        Comment


        • #5
          Code:
          <deployment>
          <cluster hostcount="3" sitesperhost="4" kfactor="1" elastic="enabled"/>
          <paths>
          <voltdbroot path="/voltdb00/voltdb-ent-5.4/falcon/"/>
          <snapshots path="snapshots"/>
          <exportoverflow path="export_overflow"/>
          <commandlog path="command_log"/>
          <commandlogsnapshot path="command_log_snapshot"/>
          </paths>
          <partition-detection enabled="true">
          <snapshot prefix="voltdb_partition_detection"/>
          </partition-detection>
          <snapshot prefix="votdb_autosnap" frequency="24h" retain="5"/>
          <!-- <admin-mode port="21211" adminstartup="true"/> -->
          <heartbeat timeout="90"/>
          <httpd port="8080">
          <jsonapi enabled="true"/>
          </httpd>
          <commandlog synchronous="false" enabled="true" logsize="1024">
          <frequency time="200" transactions="2147483647"/>
          </commandlog>
          <systemsettings>
          <temptables maxsize="100"/>
          <snapshot priority="6"/>
          </systemsettings>
          </deployment>
          Code:
           /usr/bin/java -Xmx2048m -server -Djava.awt.headless=true -Dsun.net.inetaddr.ttl=300 -Dsun.net.inetaddr.negative.ttl=3600 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+UseTLAB -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCondCardMark -Dsun.rmi.dgc.server.gcInterval=9223372036854775807 -Dsun.rmi.dgc.client.gcInterval=9223372036854775807 -XX:CMSWaitDuration=120000 -XX:CMSMaxAbortablePrecleanTime=120000 -XX:+ExplicitGCInvokesConcurrent -XX:+CMSScavengeBeforeRemark -XX:+CMSClassUnloadingEnabled -XX:PermSize=64m -Dlog4j.configuration=file:///voltdb00/voltdb-ent-5.4/voltdb/log4j.xml -Djava.library.path=dummyvalue -classpath /voltdb00/voltdb-ent-5.4/voltdb/voltdb-5.4.jar:/voltdb00/voltdb-ent-5.4/lib/felix.jar:/voltdb00/voltdb-ent-5.4/lib/tomcat-juli.jar:/voltdb00/voltdb-ent-5.4/lib/commons-logging-1.1.3.jar:/voltdb00/voltdb-ent-5.4/lib/httpclient-4.3.2.jar:/voltdb00/voltdb-ent-5.4/lib/jsch-0.1.51.jar:/voltdb00/voltdb-ent-5.4/lib/httpcore-nio-4.3.2.jar:/voltdb00/voltdb-ent-5.4/lib/slf4j-nop-1.6.2.jar:/voltdb00/voltdb-ent-5.4/lib/avro-1.7.7.jar:/voltdb00/voltdb-ent-5.4/lib/tomcat-jdbc.jar:/voltdb00/voltdb-ent-5.4/lib/slf4j-api-1.6.2.jar:/voltdb00/voltdb-ent-5.4/lib/metrics-core-2.2.0.jar:/voltdb00/voltdb-ent-5.4/lib/jackson-core-asl-1.9.13.jar:/voltdb00/voltdb-ent-5.4/lib/jna.jar:/voltdb00/voltdb-ent-5.4/lib/commons-lang3-3.0.jar:/voltdb00/voltdb-ent-5.4/lib/groovy-all-2.1.7-indy.jar:/voltdb00/voltdb-ent-5.4/lib/jetty-continuation-7.6.1.v20120215.jar:/voltdb00/voltdb-ent-5.4/lib/jetty-server-7.6.1.v20120215.jar:/voltdb00/voltdb-ent-5.4/lib/jackson-mapper-asl-1.9.13.jar:/voltdb00/voltdb-ent-5.4/lib/super-csv-2.1.0.jar:/voltdb00/voltdb-ent-5.4/lib/jetty-io-7.6.1.v20120215.jar:/voltdb00/voltdb-ent-5.4/lib/jline-2.10.jar:/voltdb00/voltdb-ent-5.4/lib/httpasyncclient-4.0.1.jar:/voltdb00/voltdb-ent-5.4/lib/commons-cli-1.2.jar:/voltdb00/voltdb-ent-5.4/lib/protobuf-java-2.5.0.jar:/voltdb00/voltdb-ent-5.4/lib/scala-library.jar:/voltdb00/voltdb-ent-5.4/lib/kafka_2.8.0-0.8.1.jar:/voltdb00/voltdb-ent-5.4/lib/log4j-1.2.16.jar:/voltdb00/voltdb-ent-5.4/lib/servlet-api-2.5.jar:/voltdb00/voltdb-ent-5.4/lib/httpcore-4.3.2.jar:/voltdb00/voltdb-ent-5.4/lib/jetty-http-7.6.1.v20120215.jar:/voltdb00/voltdb-ent-5.4/lib/jetty-util-7.6.1.v20120215.jar:/voltdb00/voltdb-ent-5.4/lib/snappy-java-1.1.0.1.jar:/voltdb00/voltdb-ent-5.4/lib/commons-codec-1.6.jar:/voltdb00/voltdb-ent-5.4/lib/extension/apache-log4j-extras-1.2.17.jar:/voltdb00/voltdb-ent-5.4/lib/extension/voltdb-rabbitmq.jar org.voltdb.VoltDB create deployment deployment-falcon.xml host vdcsis-voltdb1 license license.xml
          I filtered the lines in the netstat output by state=LISTEN because the ESTABLISHEDs were too many

          Code:
          tcp        0      0 0.0.0.0:111                 0.0.0.0:*                   LISTEN      -                   off (0.00/0/0)
          tcp        0      0 0.0.0.0:34869               0.0.0.0:*                   LISTEN      -                   off (0.00/0/0)
          tcp        0      0 0.0.0.0:22                  0.0.0.0:*                   LISTEN      -                   off (0.00/0/0)
          tcp        0      0 127.0.0.1:631               0.0.0.0:*                   LISTEN      -                   off (0.00/0/0)
          tcp        0      0 127.0.0.1:25                0.0.0.0:*                   LISTEN      -                   off (0.00/0/0)
          tcp        0      0 0.0.0.0:5666                0.0.0.0:*                   LISTEN      -                   off (0.00/0/0)
          tcp        0      0 ::ffff:127.0.0.1:7181       :::*                        LISTEN      -                   off (0.00/0/0)
          tcp        0      0 ::ffff:10.150.16.136:3021   :::*                        LISTEN      -                   off (0.00/0/0)
          tcp        0      0 :::111                      :::*                        LISTEN      -                   off (0.00/0/0)
          tcp        0      0 :::8080                     :::*                        LISTEN      -                   off (0.00/0/0)
          tcp        0      0 :::22                       :::*                        LISTEN      -                   off (0.00/0/0)
          tcp        0      0 ::1:631                     :::*                        LISTEN      -                   off (0.00/0/0)
          tcp        0      0 ::1:25                      :::*                        LISTEN      -                   off (0.00/0/0)
          tcp        0      0 :::21211                    :::*                        LISTEN      -                   off (0.00/0/0)
          tcp        0      0 :::21212                    :::*                        LISTEN      -                   off (0.00/0/0)
          tcp        0      0 :::5666                     :::*                        LISTEN      -                   off (0.00/0/0)
          tcp        0      0 :::9090                     :::*                        LISTEN      -                   off (0.00/0/0)
          tcp        0      0 :::46627                    :::*                        LISTEN      -                   off (0.00/0/0)

          Comment


          • #6
            albert,
            Can you show me the output of 'netstat -antplo | grep 212'?
            Peter Zhao

            Comment


            • #7
              Code:
              tcp        0      0 :::21211                    :::*                        LISTEN      -                   off (0.00/0/0)
              tcp        0      0 :::21212                    :::*                        LISTEN      -                   off (0.00/0/0)
              tcp        0      0 ::ffff:10.150.16.136:21212  ::ffff:10.150.12.211:38108  ESTABLISHED -                   keepalive (1011,49/0/0)
              tcp        0      0 ::ffff:10.150.16.136:21212  ::ffff:10.150.55.131:65117  ESTABLISHED -                   keepalive (1570,25/0/0)
              tcp        0      0 ::ffff:10.150.16.136:21212  ::ffff:10.150.55.131:50287  ESTABLISHED -                   keepalive (3504,99/0/0)
              tcp        0      0 ::ffff:10.150.16.136:21212  ::ffff:10.150.55.131:62311  ESTABLISHED -                   keepalive (3137,86/0/0)
              tcp        0      0 ::ffff:10.150.16.136:21212  ::ffff:10.150.55.131:50262  ESTABLISHED -                   keepalive (3503,85/0/0)
              tcp        0      0 ::ffff:10.150.16.136:21212  ::ffff:10.150.55.131:61680  ESTABLISHED -                   keepalive (2478,53/0/0)
              tcp        0      0 ::ffff:10.150.16.136:21212  ::ffff:10.150.12.211:38096  ESTABLISHED -                   keepalive (1011,42/0/0)
              tcp        0      0 ::ffff:10.150.16.136:21212  ::ffff:10.150.55.131:65148  ESTABLISHED -                   keepalive (1570,25/0/0)
              tcp        0      0 ::ffff:10.150.16.136:21212  ::ffff:10.150.55.131:61675  ESTABLISHED -                   keepalive (2480,47/0/0)
              tcp        0      0 ::ffff:10.150.16.136:21212  ::ffff:10.150.55.131:62306  ESTABLISHED -                   keepalive (3137,82/0/0)
              tcp        0      0 ::ffff:10.150.16.136:21212  ::ffff:10.150.55.131:62332  ESTABLISHED -                   keepalive (3138,93/0/0)
              tcp        0      0 ::ffff:10.150.16.136:21212  ::ffff:10.150.55.131:63266  ESTABLISHED -                   keepalive (4849,25/0/0)
              tcp        0      0 ::ffff:10.150.16.136:21212  ::ffff:10.150.12.211:38102  ESTABLISHED -                   keepalive (1011,45/0/0)
              tcp        0      0 ::ffff:10.150.16.136:21212  ::ffff:10.150.55.131:61900  ESTABLISHED -                   keepalive (2749,76/0/0)
              tcp        0      0 ::ffff:10.150.16.136:21212  ::ffff:10.150.55.131:63241  ESTABLISHED -                   keepalive (4850,11/0/0)
              tcp        0      0 ::ffff:10.150.16.136:21212  ::ffff:10.150.55.131:63271  ESTABLISHED -                   keepalive (4850,27/0/0)
              tcp        0      0 ::ffff:10.150.16.136:21212  ::ffff:10.150.55.131:62296  ESTABLISHED -                   keepalive (3137,78/0/0)
              tcp        0      0 ::ffff:10.150.16.136:21212  ::ffff:10.150.55.131:63287  ESTABLISHED -                   keepalive (4872,34/0/0)
              tcp        0      0 ::ffff:10.150.16.136:21212  ::ffff:10.150.55.131:61905  ESTABLISHED -                   keepalive (2751,83/0/0)
              tcp        0      0 ::ffff:10.150.16.136:21212  ::ffff:10.150.55.131:63276  ESTABLISHED -                   keepalive (4850,28/0/0)
              tcp        0      0 ::ffff:10.150.16.136:21212  ::ffff:10.150.55.131:65138  ESTABLISHED -                   keepalive (1563,17/0/0)
              tcp        0      0 ::ffff:10.150.16.136:21212  ::ffff:10.150.55.131:63251  ESTABLISHED -                   keepalive (4871,18/0/0)
              tcp        0      0 ::ffff:10.150.16.136:21212  ::ffff:10.150.12.211:38087  ESTABLISHED -                   keepalive (1011,35/0/0)
              tcp        0      0 ::ffff:10.150.16.136:21212  ::ffff:10.150.55.131:61665  ESTABLISHED -                   keepalive (2480,31/0/0)
              tcp        0      0 ::ffff:10.150.16.136:21212  ::ffff:10.150.55.131:63261  ESTABLISHED -                   keepalive (4868,21/0/0)
              tcp        0      0 ::ffff:10.150.16.136:21212  ::ffff:10.150.55.131:61655  ESTABLISHED -                   keepalive (2478,08/0/0)
              tcp        0      0 ::ffff:10.150.16.136:21212  ::ffff:10.150.55.131:63282  ESTABLISHED -                   keepalive (4872,27/0/0)
              tcp        0      0 ::ffff:10.150.16.136:21212  ::ffff:10.150.55.131:61920  ESTABLISHED -                   keepalive (2752,00/0/0)
              tcp        0      0 ::ffff:10.150.16.136:21212  ::ffff:10.150.55.131:50303  ESTABLISHED -                   keepalive (3499,03/0/0)
              tcp        0      0 ::ffff:10.150.16.136:21212  ::ffff:10.150.55.131:50308  ESTABLISHED -                   keepalive (3500,09/0/0)
              tcp        0      0 ::ffff:10.150.16.136:21212  ::ffff:10.150.55.131:61910  ESTABLISHED -                   keepalive (2749,84/0/0)
              tcp        0      0 ::ffff:10.150.16.136:21212  ::ffff:10.150.55.131:50267  ESTABLISHED -                   keepalive (3498,85/0/0)
              tcp        0      0 ::ffff:10.150.16.136:21212  ::ffff:10.150.12.211:38093  ESTABLISHED -                   keepalive (1010,35/0/0)
              tcp        0      0 ::ffff:10.150.16.136:21212  ::ffff:10.150.55.131:61685  ESTABLISHED -                   keepalive (2478,51/0/0)
              tcp        0      0 ::ffff:10.150.16.136:21212  ::ffff:10.150.55.131:62326  ESTABLISHED -                   keepalive (3137,85/0/0)
              tcp        0      0 ::ffff:10.150.16.136:21212  ::ffff:10.150.55.131:61894  ESTABLISHED -                   keepalive (2751,62/0/0)
              tcp        0      0 ::ffff:10.150.16.136:21212  ::ffff:10.150.55.131:61650  ESTABLISHED -                   keepalive (2478,81/0/0)
              tcp        0      0 ::ffff:10.150.16.136:21212  ::ffff:10.150.55.131:65143  ESTABLISHED -                   keepalive (1570,17/0/0)
              tcp        0      0 ::ffff:10.150.16.136:21212  ::ffff:10.150.55.131:65128  ESTABLISHED -                   keepalive (1569,07/0/0)
              tcp        0      0 ::ffff:10.150.16.136:21212  ::ffff:10.150.12.211:38084  ESTABLISHED -                   keepalive (1011,22/0/0)
              tcp        0      0 ::ffff:10.150.16.136:21212  ::ffff:10.150.55.131:62337  ESTABLISHED -                   keepalive (3138,94/0/0)
              tcp        0      0 ::ffff:10.150.16.136:21212  ::ffff:10.150.55.131:62291  ESTABLISHED -                   keepalive (3136,67/0/0)
              tcp        0      0 ::ffff:10.150.16.136:21212  ::ffff:10.150.55.131:50272  ESTABLISHED -                   keepalive (3497,85/0/0)
              tcp        0      0 ::ffff:10.150.16.136:21212  ::ffff:10.150.55.131:50298  ESTABLISHED -                   keepalive (3503,96/0/0)
              tcp        0      0 ::ffff:10.150.16.136:21212  ::ffff:10.150.55.131:65133  ESTABLISHED -                   keepalive (1569,08/0/0)
              tcp        0      0 ::ffff:10.150.16.136:21212  ::ffff:10.150.55.131:50277  ESTABLISHED -                   keepalive (3506,87/0/0)
              tcp        0      0 ::ffff:10.150.16.136:21212  ::ffff:10.150.55.131:61690  ESTABLISHED -                   keepalive (2479,53/0/0)
              tcp        0      0 ::ffff:10.150.16.136:21212  ::ffff:10.150.55.131:50282  ESTABLISHED -                   keepalive (3499,88/0/0)
              tcp        0      0 ::ffff:10.150.16.136:21212  ::ffff:10.150.55.131:65112  ESTABLISHED -                   keepalive (1563,11/0/0)
              tcp        0      0 ::ffff:10.150.16.136:21212  ::ffff:10.150.55.131:63256  ESTABLISHED -                   keepalive (4849,14/0/0)
              tcp        0      0 ::ffff:10.150.16.136:21212  ::ffff:10.150.55.131:65123  ESTABLISHED -                   keepalive (1569,02/0/0)
              tcp        0      0 ::ffff:10.150.16.136:21212  ::ffff:10.150.55.131:65153  ESTABLISHED -                   keepalive (1570,19/0/0)
              tcp        0      0 ::ffff:10.150.16.136:21212  ::ffff:10.150.55.131:63246  ESTABLISHED -                   keepalive (4872,08/0/0)
              tcp        0      0 ::ffff:10.150.16.136:21212  ::ffff:10.150.12.211:38090  ESTABLISHED -                   keepalive (1011,29/0/0)
              tcp        0      0 ::ffff:10.150.16.136:21212  ::ffff:10.150.12.211:38099  ESTABLISHED -                   keepalive (1010,34/0/0)
              tcp        0      0 ::ffff:10.150.16.136:21212  ::ffff:10.150.12.211:38111  ESTABLISHED -                   keepalive (1011,41/0/0)
              tcp        0      0 ::ffff:10.150.16.136:21212  ::ffff:10.150.55.131:65107  ESTABLISHED -                   keepalive (1569,04/0/0)
              tcp        0      0 ::ffff:10.150.16.136:21212  ::ffff:10.150.55.131:50292  ESTABLISHED -                   keepalive (3505,90/0/0)
              tcp        0      0 ::ffff:10.150.16.136:21212  ::ffff:10.150.55.131:61915  ESTABLISHED -                   keepalive (2752,87/0/0)
              tcp        0      0 ::ffff:10.150.16.136:21212  ::ffff:10.150.55.131:61925  ESTABLISHED -                   keepalive (2748,99/0/0)
              tcp        0      0 ::ffff:10.150.16.136:21212  ::ffff:10.150.55.131:61889  ESTABLISHED -                   keepalive (2750,44/0/0)
              tcp        0      0 ::ffff:10.150.16.136:21212  ::ffff:10.150.12.211:38105  ESTABLISHED -                   keepalive (1011,35/0/0)
              tcp        0      0 ::ffff:10.150.16.136:21212  ::ffff:10.150.55.131:62301  ESTABLISHED -                   keepalive (3136,67/0/0)
              tcp        0      0 ::ffff:10.150.16.136:21212  ::ffff:10.150.55.131:61670  ESTABLISHED -                   keepalive (2477,27/0/0)
              tcp        0      0 ::ffff:10.150.16.136:21212  ::ffff:10.150.55.131:61695  ESTABLISHED -                   keepalive (2479,55/0/0)
              tcp        0      0 ::ffff:10.150.16.136:21212  ::ffff:10.150.55.131:62321  ESTABLISHED -                   keepalive (3137,75/0/0)
              tcp        0      0 ::ffff:10.150.16.136:21212  ::ffff:10.150.55.131:61660  ESTABLISHED -                   keepalive (2479,09/0/0)
              tcp        0      0 ::ffff:10.150.16.136:21212  ::ffff:10.150.55.131:61930  ESTABLISHED -                   keepalive (2753,01/0/0)
              tcp        0      0 ::ffff:10.150.16.136:21212  ::ffff:10.150.55.131:62316  ESTABLISHED -                   keepalive (3137,73/0/0)
              tcp        0      0 ::ffff:10.150.16.136:21212  ::ffff:10.150.55.131:61935  ESTABLISHED -                   keepalive (2753,07/0/0)

              Comment


              • #8
                albert,
                It appears the right port, 21212, is listening. I have some more followup questions:
                Could you try 'sqlcmd --servers=<hostname> --port=21211' from a remote and local host?
                Is this issue affecting all the nodes in the cluster? For each affected host, can you jstack the sqlcmd process, jstack <sqlcmd pid>?
                Peter Zhao

                Comment


                • #9
                  EDIT: False Positive
                  I tried with an older version (by mistake) of sqlcmd and I got a prompt and could run queries.


                  This is the dump using the 5.4 sqlcmd
                  Code:
                  Full thread dump OpenJDK 64-Bit Server VM (24.75-b04 mixed mode):
                  
                  "Estimated Time Updater" daemon prio=10 tid=0x00007f951426d800 nid=0xeb1b runnable [0x00007f950918f000]
                     java.lang.Thread.State: RUNNABLE
                          at java.util.zip.ZipFile.getEntry(Native Method)
                          at java.util.zip.ZipFile.getInputStream(ZipFile.java:351)
                          - locked <0x00000000f64ffa18> (a java.util.jar.JarFile)
                          at java.util.jar.JarFile.getInputStream(JarFile.java:412)
                          - locked <0x00000000f64ffa18> (a java.util.jar.JarFile)
                          at sun.misc.URLClassPath$JarLoader$2.getInputStream(URLClassPath.java:775)
                          at sun.misc.Resource.cachedInputStream(Resource.java:77)
                          - locked <0x00000000f5627fb0> (a sun.misc.URLClassPath$JarLoader$2)
                          at sun.misc.Resource.getByteBuffer(Resource.java:160)
                          at java.net.URLClassLoader.defineClass(URLClassLoader.java:436)
                          at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
                          at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
                          at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
                          at java.security.AccessController.doPrivileged(Native Method)
                          at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
                          at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
                          - locked <0x00000000f5626de8> (a java.lang.Object)
                          at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
                          at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
                          at org.voltcore.logging.VoltLogger.createRunnableLoggingTask(VoltLogger.java:154)
                          at org.voltcore.logging.VoltLogger.execute(VoltLogger.java:137)
                          at org.voltcore.logging.VoltLogger.info(VoltLogger.java:263)
                          at org.voltcore.utils.EstTimeUpdater$1.run(EstTimeUpdater.java:42)
                  
                  "VoltDB Client Reaper Thread - 0" daemon prio=10 tid=0x00007f951423b800 nid=0xeb17 runnable [0x00007f9509223000]
                     java.lang.Thread.State: RUNNABLE
                          at sun.misc.Unsafe.getObjectVolatile(Native Method)
                          at java.util.concurrent.ConcurrentHashMap.entryAt(ConcurrentHashMap.java:305)
                          at java.util.concurrent.ConcurrentHashMap$HashIterator.advance(ConcurrentHashMap.java:1337)
                          at java.util.concurrent.ConcurrentHashMap$HashIterator.<init>(ConcurrentHashMap.java:1327)
                          at java.util.concurrent.ConcurrentHashMap$EntryIterator.<init>(ConcurrentHashMap.java:1416)
                          at java.util.concurrent.ConcurrentHashMap$EntrySet.iterator(ConcurrentHashMap.java:1467)
                          at org.voltdb.client.Distributer$CallExpiration.run(Distributer.java:291)
                          at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
                          at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
                          at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
                          at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
                          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
                          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
                          at org.voltcore.utils.CoreUtils$7$1.run(CoreUtils.java:735)
                          at java.lang.Thread.run(Thread.java:745)
                  
                  "Volt Client Network - 1" daemon prio=10 tid=0x00007f951422b000 nid=0xeb16 runnable [0x00007f9509324000]
                     java.lang.Thread.State: RUNNABLE
                          at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
                          at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
                          at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
                          at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
                          - locked <0x00000000f651c418> (a io.netty_voltpatches.NinjaKeySet)
                          - locked <0x00000000f651c4e0> (a java.util.Collections$UnmodifiableSet)
                          - locked <0x00000000f651c370> (a sun.nio.ch.EPollSelectorImpl)
                          at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
                          at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
                          at org.voltcore.network.VoltNetwork.run(VoltNetwork.java:303)
                          at java.lang.Thread.run(Thread.java:745)
                  
                  "Service Thread" daemon prio=10 tid=0x00007f95140ae800 nid=0xeb13 runnable [0x0000000000000000]
                     java.lang.Thread.State: RUNNABLE
                  
                  "C2 CompilerThread1" daemon prio=10 tid=0x00007f95140ac000 nid=0xeb12 waiting on condition [0x0000000000000000]
                     java.lang.Thread.State: RUNNABLE
                  
                  "C2 CompilerThread0" daemon prio=10 tid=0x00007f95140a9000 nid=0xeb11 waiting on condition [0x0000000000000000]
                     java.lang.Thread.State: RUNNABLE
                  
                  "Signal Dispatcher" daemon prio=10 tid=0x00007f95140a7000 nid=0xeb10 waiting on condition [0x0000000000000000]
                     java.lang.Thread.State: RUNNABLE
                  
                  "Finalizer" daemon prio=10 tid=0x00007f951407c000 nid=0xeb0f in Object.wait() [0x00007f950a061000]
                     java.lang.Thread.State: WAITING (on object monitor)
                          at java.lang.Object.wait(Native Method)
                          - waiting on <0x00000000f6400f50> (a java.lang.ref.ReferenceQueue$Lock)
                          at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
                          - locked <0x00000000f6400f50> (a java.lang.ref.ReferenceQueue$Lock)
                          at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
                          at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
                  
                  "Reference Handler" daemon prio=10 tid=0x00007f951407a000 nid=0xeb0e in Object.wait() [0x00007f950a162000]
                     java.lang.Thread.State: WAITING (on object monitor)
                          at java.lang.Object.wait(Native Method)
                          - waiting on <0x00000000f64006e0> (a java.lang.ref.Reference$Lock)
                          at java.lang.Object.wait(Object.java:503)
                          at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
                          - locked <0x00000000f64006e0> (a java.lang.ref.Reference$Lock)
                  
                  "main" prio=10 tid=0x00007f951400a000 nid=0xeb08 waiting on condition [0x00007f95194d2000]
                     java.lang.Thread.State: WAITING (parking)
                          at sun.misc.Unsafe.park(Native Method)
                          - parking to wait for  <0x00000000f55d07e8> (a java.util.concurrent.Semaphore$NonfairSync)
                          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
                          at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
                          at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
                          at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
                          at java.util.concurrent.Semaphore.acquire(Semaphore.java:317)
                          at org.voltdb.client.SyncCallback.waitForResponse(SyncCallback.java:74)
                          at org.voltdb.client.ClientImpl.callProcedure(ClientImpl.java:267)
                          at org.voltdb.client.ClientImpl.callProcedureWithTimeout(ClientImpl.java:226)
                          at org.voltdb.client.ClientImpl.callProcedure(ClientImpl.java:206)
                          at org.voltdb.utils.SQLCommand.loadStoredProcedures(SQLCommand.java:1043)
                          at org.voltdb.utils.SQLCommand.main(SQLCommand.java:1281)
                  
                  "VM Thread" prio=10 tid=0x00007f9514075800 nid=0xeb0d runnable
                  
                  "GC task thread#0 (ParallelGC)" prio=10 tid=0x00007f9514020000 nid=0xeb09 runnable
                  
                  "GC task thread#1 (ParallelGC)" prio=10 tid=0x00007f9514022000 nid=0xeb0a runnable
                  
                  "GC task thread#2 (ParallelGC)" prio=10 tid=0x00007f9514023800 nid=0xeb0b runnable
                  
                  "GC task thread#3 (ParallelGC)" prio=10 tid=0x00007f9514025800 nid=0xeb0c runnable
                  
                  "VM Periodic Task Thread" prio=10 tid=0x00007f95140b9000 nid=0xeb14 waiting on condition
                  
                  JNI global references: 178
                  
                  Heap
                   PSYoungGen      total 17920K, used 4273K [0x00000000f5500000, 0x00000000f7800000, 0x0000000100000000)
                    eden space 15360K, 11% used [0x00000000f5500000,0x00000000f56aec20,0x00000000f6400000)
                    from space 2560K, 99% used [0x00000000f6400000,0x00000000f667db70,0x00000000f6680000)
                    to   space 2560K, 0% used [0x00000000f7580000,0x00000000f7580000,0x00000000f7800000)
                   ParOldGen       total 40448K, used 1281K [0x00000000e0000000, 0x00000000e2780000, 0x00000000f5500000)
                    object space 40448K, 3% used [0x00000000e0000000,0x00000000e0140488,0x00000000e2780000)
                   PSPermGen       total 21504K, used 10361K [0x00000000d5a00000, 0x00000000d6f00000, 0x00000000e0000000)
                    object space 21504K, 48% used [0x00000000d5a00000,0x00000000d641e620,0x00000000d6f00000)
                  Last edited by albert; 02-12-2016, 09:58 AM.

                  Comment


                  • #10
                    albert,
                    Thanks for your assistance in troubleshooting.
                    Peter Zhao

                    Comment


                    • #11
                      Sorry, I think I wasn't clear. When you asked me to try "sqlcmd --servers...." I tried and used the "sqlcmd" that was on my $PATH worked, but that was and old version and could'nt do what I wanted to. The one I need (to load classes) is the the 5.4 that still doesn't respond.

                      The stackdump that I posted is from the 5.4 version.

                      Comment


                      • #12
                        albert,
                        Let's continue by trying the following:
                        1)Try unsetting sqlcmd from the path and calling the <absolute path of sqlcmd> with --servers=<hostname> --port=21212
                        2)Try using the IP instead of the hostname in #1

                        After talking to an engineer about the jstack, we have a few additional questions:
                        1)Could you retry this jstack output a few times and provide it's output?
                        2)What is the error, if any, presented and the command you ran?
                        3)Could you show us the logs during the time you run this command?
                        Peter Zhao
                        Last edited by pzhao; 02-15-2016, 01:13 PM.

                        Comment


                        • #13
                          Code:
                          Full thread dump OpenJDK 64-Bit Server VM (24.75-b04 mixed mode):
                          
                          "Attach Listener" daemon prio=10 tid=0x00007f1f44001000 nid=0x7011 waiting on condition [0x0000000000000000]
                             java.lang.Thread.State: RUNNABLE
                          
                          "Async Logger" daemon prio=10 tid=0x00007f1f1c001800 nid=0x6ffa waiting on condition [0x00007f1f5d44d000]
                             java.lang.Thread.State: WAITING (parking)
                                  at sun.misc.Unsafe.park(Native Method)
                                  - parking to wait for  <0x00000000f5acdff8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
                                  at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
                                  at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
                                  at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
                                  at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
                                  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
                                  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
                                  at org.voltcore.utils.CoreUtils$6$1.run(CoreUtils.java:644)
                                  at java.lang.Thread.run(Thread.java:745)
                          
                          "Estimated Time Updater" daemon prio=10 tid=0x00007f1f74300800 nid=0x6fda waiting on condition [0x00007f1f5d40c000]
                             java.lang.Thread.State: TIMED_WAITING (sleeping)
                                  at java.lang.Thread.sleep(Native Method)
                                  at org.voltcore.utils.EstTimeUpdater$1.run(EstTimeUpdater.java:37)
                          
                          "VoltDB Client Reaper Thread - 0" daemon prio=10 tid=0x00007f1f742c1000 nid=0x6fd6 waiting on condition [0x00007f1f5d492000]
                             java.lang.Thread.State: TIMED_WAITING (parking)
                                  at sun.misc.Unsafe.park(Native Method)
                                  - parking to wait for  <0x00000000f579b750> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
                                  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
                                  at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
                                  at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
                                  at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
                                  at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
                                  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
                                  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
                                  at org.voltcore.utils.CoreUtils$6$1.run(CoreUtils.java:644)
                                  at java.lang.Thread.run(Thread.java:745)
                          
                          "Volt Client Network - 1" daemon prio=10 tid=0x00007f1f742bf800 nid=0x6fd5 runnable [0x00007f1f5d593000]
                             java.lang.Thread.State: RUNNABLE
                                  at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
                                  at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
                                  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
                                  at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
                                  - locked <0x00000000f5b719b0> (a io.netty_voltpatches.NinjaKeySet)
                                  - locked <0x00000000f5b69ee0> (a java.util.Collections$UnmodifiableSet)
                                  - locked <0x00000000f5b69ab0> (a sun.nio.ch.EPollSelectorImpl)
                                  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
                                  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
                                  at org.voltcore.network.VoltNetwork.run(VoltNetwork.java:303)
                                  at java.lang.Thread.run(Thread.java:745)
                          
                          "Thread-0" daemon prio=10 tid=0x00007f1f7423b800 nid=0x6fd4 runnable [0x00007f1f5d736000]
                             java.lang.Thread.State: RUNNABLE
                                  at java.net.PlainSocketImpl.socketConnect(Native Method)
                                  at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
                                  - locked <0x00000000f6091160> (a java.net.SocksSocketImpl)
                                  at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
                                  at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
                                  at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
                                  at java.net.Socket.connect(Socket.java:579)
                                  at java.net.Socket.connect(Socket.java:528)
                                  at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
                                  at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
                                  at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
                                  - locked <0x00000000f60907e8> (a sun.net.www.http.HttpClient)
                                  at sun.net.www.http.HttpClient.<init>(HttpClient.java:211)
                                  at sun.net.www.http.HttpClient.New(HttpClient.java:308)
                                  at sun.net.www.http.HttpClient.New(HttpClient.java:326)
                                  at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:997)
                                  at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:933)
                                  at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:851)
                                  at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1301)
                                  - locked <0x00000000f60885f8> (a sun.net.www.protocol.http.HttpURLConnection)
                                  at org.voltdb.utils.SQLCommand.openURL(SQLCommand.java:1582)
                                  at org.voltdb.utils.SQLCommand.access$300(SQLCommand.java:65)
                                  at org.voltdb.utils.SQLCommand$2.run(SQLCommand.java:1562)
                                  at java.lang.Thread.run(Thread.java:745)
                          
                          "Service Thread" daemon prio=10 tid=0x00007f1f740ae800 nid=0x6fd2 runnable [0x0000000000000000]
                             java.lang.Thread.State: RUNNABLE
                          
                          "C2 CompilerThread1" daemon prio=10 tid=0x00007f1f740ac000 nid=0x6fd1 waiting on condition [0x0000000000000000]
                             java.lang.Thread.State: RUNNABLE
                          
                          "C2 CompilerThread0" daemon prio=10 tid=0x00007f1f740a9000 nid=0x6fd0 waiting on condition [0x0000000000000000]
                             java.lang.Thread.State: RUNNABLE
                          
                          "Signal Dispatcher" daemon prio=10 tid=0x00007f1f740a7000 nid=0x6fcf runnable [0x0000000000000000]
                             java.lang.Thread.State: RUNNABLE
                          
                          "Finalizer" daemon prio=10 tid=0x00007f1f7407c000 nid=0x6fce in Object.wait() [0x00007f1f5e2a5000]
                             java.lang.Thread.State: WAITING (on object monitor)
                                  at java.lang.Object.wait(Native Method)
                                  - waiting on <0x00000000f5504800> (a java.lang.ref.ReferenceQueue$Lock)
                                  at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
                                  - locked <0x00000000f5504800> (a java.lang.ref.ReferenceQueue$Lock)
                                  at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
                                  at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
                          
                          "Reference Handler" daemon prio=10 tid=0x00007f1f7407a000 nid=0x6fcd in Object.wait() [0x00007f1f5e3a6000]
                             java.lang.Thread.State: WAITING (on object monitor)
                                  at java.lang.Object.wait(Native Method)
                                  - waiting on <0x00000000f5504418> (a java.lang.ref.Reference$Lock)
                                  at java.lang.Object.wait(Object.java:503)
                                  at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
                                  - locked <0x00000000f5504418> (a java.lang.ref.Reference$Lock)
                          
                          "main" prio=10 tid=0x00007f1f7400a000 nid=0x6fc7 waiting on condition [0x00007f1f79741000]
                             java.lang.Thread.State: WAITING (parking)
                                  at sun.misc.Unsafe.park(Native Method)
                                  - parking to wait for  <0x00000000f6033310> (a java.util.concurrent.Semaphore$NonfairSync)
                                  at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
                                  at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
                                  at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
                                  at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
                                  at java.util.concurrent.Semaphore.acquire(Semaphore.java:317)
                                  at org.voltdb.client.SyncCallback.waitForResponse(SyncCallback.java:74)
                                  at org.voltdb.client.ClientImpl.callProcedure(ClientImpl.java:265)
                                  at org.voltdb.client.ClientImpl.callProcedureWithTimeout(ClientImpl.java:224)
                                  at org.voltdb.client.ClientImpl.callProcedure(ClientImpl.java:204)
                                  at org.voltdb.utils.SQLCommand.loadStoredProcedures(SQLCommand.java:1229)
                                  at org.voltdb.utils.SQLCommand.main(SQLCommand.java:1466)
                          
                          "VM Thread" prio=10 tid=0x00007f1f74075800 nid=0x6fcc runnable
                          
                          "GC task thread#0 (ParallelGC)" prio=10 tid=0x00007f1f74020000 nid=0x6fc8 runnable
                          
                          "GC task thread#1 (ParallelGC)" prio=10 tid=0x00007f1f74022000 nid=0x6fc9 runnable
                          
                          "GC task thread#2 (ParallelGC)" prio=10 tid=0x00007f1f74023800 nid=0x6fca runnable
                          
                          "GC task thread#3 (ParallelGC)" prio=10 tid=0x00007f1f74025800 nid=0x6fcb runnable
                          
                          "VM Periodic Task Thread" prio=10 tid=0x00007f1f740b9000 nid=0x6fd3 waiting on condition
                          
                          JNI global references: 189
                          I'm looking at the log and doesn't show anything. Only previous entries of some queries by someone else.

                          This is the result from "tail -10 .../volt.log" before the execution of sqlcmd, and after is the same (same last timestamp).
                          Code:
                          2016-02-15 13:55:32,980   WARN  [Ad Hoc Planner - 0] COMPILER: [GetNumeroEspecialSMSActivo.class]: Procedure GetNumeroEspecialSMSActivo has a statement with a non-deterministic result - statement: "SELECT numero_especial, codigo_prestadora, glosa, codigo_tipo_mensaje, tiene_servicio FROM numeros_esp_sms_his WHERE numero_especial = SUBSTR(cast(? as varchar), 1, LENGTH (numero_especial)) AND estado = ? AND fecha_inicio_vigencia <= ? AND fecha_fin_vigencia >= ? ORDER BY LENGTH (numero_especial) DESC LIMIT 1;" , reason: the query result does not guarantee a consistent ordering
                          2016-02-15 13:55:32,985   WARN  [Ad Hoc Planner - 0] COMPILER: [CreditoDatos.class]: Procedure CreditoDatos has a statement with a non-deterministic result - statement: "select b.codigo_bolsa_maxicarga, b.descripcion, b.tipo_credito, b.comportamiento, b.duracion_tiempo, b.unidad_tiempo from ordenes_bolsas_maxicarga a, bolsas_maxicarga b where tipo_operacion=? and a.codigo_bolsa =b.codigo_bolsa_maxicarga;", reason: the query result does not guarantee a consistent ordering
                          2016-02-15 13:55:32,985   WARN  [Ad Hoc Planner - 0] COMPILER: [SelectNumeroEspSmsXCatRbt.class]: Procedure SelectNumeroEspSmsXCatRbt has a statement with a non-deterministic result - statement: "select * from numeros_esp_sms_x_cat_rbt where codigo_proveedor = ? and codigo_categoria = ?;", reason: the query result does not guarantee a consistent ordering
                          2016-02-15 13:55:32,986   WARN  [Ad Hoc Planner - 0] COMPILER: [WHDeleteSelection.class]: Procedure WHDeleteSelection has a statement with a non-deterministic result - statement: "select clave_primaria, clave_particion, col_varchar, col_int, col_float  from wh_tabla_1 where col_int = ? and col_varchar = ?;", reason: the query result does not guarantee a consistent ordering
                          2016-02-15 13:55:32,987   WARN  [Ad Hoc Planner - 0] COMPILER: [GetGrupoFacturacionVigencia.class]: Procedure GetGrupoFacturacionVigencia has a statement with a non-deterministic result - statement: "select * from grupos_facturacion_x_suscr where codigo_suscriptor = ? and ( (? between fecha_vigencia_inicio and COALESCE(fecha_vigencia_fin,CURRENT_TIMESTAMP ) ) or ( ? between fecha_vigencia_inicio and COALESCE(fecha_vigencia_fin,CURRENT_TIMESTAMP) ) or ( fecha_vigencia_inicio between ? and ? ) or ( COALESCE(fecha_vigencia_fin,CURRENT_TIMESTAMP) between ? and ? ) )order by fecha_vigencia_inicio desc;", reason: the query result does not guarantee a consistent ordering
                          2016-02-15 13:55:32,988   WARN  [Ad Hoc Planner - 0] COMPILER: [ServicioMxcLoad.class]: Procedure ServicioMxcLoad has a statement with a non-deterministic result - statement: "SELECT * FROM servicios_recargas_x_suscr sl, servicios_recargas sr WHERE sr.codigo_servicio_recarga = sl.codigo_servicio_recarga AND sl.codigo_suscriptor = ? AND sl.codigo_servicio_recarga = ? AND sl.fecha_termino > ?;", reason: the query result does not guarantee a consistent ordering
                          2016-02-15 13:55:32,989   WARN  [Ad Hoc Planner - 0] COMPILER: [ObtenerCodigoSuscriptorHistorico.class]: Procedure ObtenerCodigoSuscriptorHistorico has a statement with a non-deterministic result - statement: "select * from cuentas_x_suscr_his  where cuenta like ? and concepto like ? order by cuenta;", reason: the query result does not guarantee a consistent ordering
                          2016-02-15 13:55:32,991   WARN  [Ad Hoc Planner - 0] COMPILER: [GetTransactionLocks.class]: Procedure GetTransactionLocks has a statement with a non-deterministic result - statement: "select * from locks where id_transaccion = ?;", reason: the query result does not guarantee a consistent ordering
                          2016-02-15 13:55:36,122   INFO  [Iv2ExecutionSite: 0:2] HOST: Globally updating the current application catalog and deployment (new hashes 2AFC8DA294, DF3B47F6F7).
                          2016-02-15 13:55:37,010   INFO  [Iv2ExecutionSite: 0:2] IMPORT: No importers specified skipping Streaming Import initialization.

                          Comment


                          • #14
                            albert,
                            Based on the new jstack you've provided, sqlcmd is not version 5.4 as you've stated earlier. With a lack of knowledge regarding how your environment is setup, I'd suggest removing older VoltDB kits to avoid confusion over which version of VoltDB you're running and reset the PATH environment variable.
                            Peter Zhao

                            Comment


                            • #15
                              That was my bad, this is with the 5.4 version :

                              /voltdb00/voltdb-ent-5.4/bin/sqlcmd --servers=<IP-ADDRESS> --port=21211
                              Code:
                              2016-02-15 17:40:28
                              Full thread dump OpenJDK 64-Bit Server VM (24.75-b04 mixed mode):
                              
                              "Attach Listener" daemon prio=10 tid=0x00007f1e20001000 nid=0x8573 waiting on condition [0x0000000000000000]
                                 java.lang.Thread.State: RUNNABLE
                              
                              "Async Logger" daemon prio=10 tid=0x00007f1df8001800 nid=0x855b waiting on condition [0x00007f1e49699000]
                                 java.lang.Thread.State: WAITING (parking)
                                      at sun.misc.Unsafe.park(Native Method)
                                      - parking to wait for  <0x00000000f64c97e0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
                                      at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
                                      at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
                                      at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
                                      at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
                                      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
                                      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
                                      at java.lang.Thread.run(Thread.java:745)
                              
                              "Estimated Time Updater" daemon prio=10 tid=0x00007f1e54276800 nid=0x8559 waiting on condition [0x00007f1e49658000]
                                 java.lang.Thread.State: TIMED_WAITING (sleeping)
                                      at java.lang.Thread.sleep(Native Method)
                                      at org.voltcore.utils.EstTimeUpdater$1.run(EstTimeUpdater.java:37)
                              
                              "VoltDB Client Reaper Thread - 0" daemon prio=10 tid=0x00007f1e5423d800 nid=0x8555 waiting on condition [0x00007f1e496ea000]
                                 java.lang.Thread.State: TIMED_WAITING (parking)
                                      at sun.misc.Unsafe.park(Native Method)
                                      - parking to wait for  <0x00000000f6400a90> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
                                      at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
                                      at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
                                      at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
                                      at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
                                      at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
                                      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
                                      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
                                      at org.voltcore.utils.CoreUtils$7$1.run(CoreUtils.java:735)
                                      at java.lang.Thread.run(Thread.java:745)
                              
                              "Volt Client Network - 1" daemon prio=10 tid=0x00007f1e5423c800 nid=0x8554 runnable [0x00007f1e497eb000]
                                 java.lang.Thread.State: RUNNABLE
                                      at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
                                      at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
                                      at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
                                      at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
                                      - locked <0x00000000f650b7d0> (a io.netty_voltpatches.NinjaKeySet)
                                      - locked <0x00000000f650cae0> (a java.util.Collections$UnmodifiableSet)
                                      - locked <0x00000000f650b718> (a sun.nio.ch.EPollSelectorImpl)
                                      at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
                                      at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
                                      at org.voltcore.network.VoltNetwork.run(VoltNetwork.java:303)
                                      at java.lang.Thread.run(Thread.java:745)
                              
                              "Thread-0" daemon prio=10 tid=0x00007f1e541ec000 nid=0x8553 runnable [0x00007f1e4994a000]
                                 java.lang.Thread.State: RUNNABLE
                                      at java.net.PlainSocketImpl.socketConnect(Native Method)
                                      at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
                                      - locked <0x00000000f653eef0> (a java.net.SocksSocketImpl)
                                      at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
                                      at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
                                      at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
                                      at java.net.Socket.connect(Socket.java:579)
                                      at java.net.Socket.connect(Socket.java:528)
                                      at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
                                      at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
                                      at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
                                      - locked <0x00000000f653f068> (a sun.net.www.http.HttpClient)
                                      at sun.net.www.http.HttpClient.<init>(HttpClient.java:211)
                                      at sun.net.www.http.HttpClient.New(HttpClient.java:308)
                                      at sun.net.www.http.HttpClient.New(HttpClient.java:326)
                                      at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:997)
                                      at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:933)
                                      at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:851)
                                      at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1301)
                                      - locked <0x00000000f653f100> (a sun.net.www.protocol.http.HttpURLConnection)
                                      at org.voltdb.utils.SQLCommand.openURL(SQLCommand.java:1363)
                                      at org.voltdb.utils.SQLCommand.access$200(SQLCommand.java:67)
                                      at org.voltdb.utils.SQLCommand$2.run(SQLCommand.java:1343)
                                      at java.lang.Thread.run(Thread.java:745)
                              
                              "Service Thread" daemon prio=10 tid=0x00007f1e540ae800 nid=0x8551 runnable [0x0000000000000000]
                                 java.lang.Thread.State: RUNNABLE
                              
                              "C2 CompilerThread1" daemon prio=10 tid=0x00007f1e540ac000 nid=0x8550 waiting on condition [0x0000000000000000]
                                 java.lang.Thread.State: RUNNABLE
                              
                              "C2 CompilerThread0" daemon prio=10 tid=0x00007f1e540a9000 nid=0x854f waiting on condition [0x0000000000000000]
                                 java.lang.Thread.State: RUNNABLE
                              
                              "Signal Dispatcher" daemon prio=10 tid=0x00007f1e540a7000 nid=0x854e runnable [0x0000000000000000]
                                 java.lang.Thread.State: RUNNABLE
                              
                              "Finalizer" daemon prio=10 tid=0x00007f1e5407c000 nid=0x854d in Object.wait() [0x00007f1e4a4c8000]
                                 java.lang.Thread.State: WAITING (on object monitor)
                                      at java.lang.Object.wait(Native Method)
                                      - waiting on <0x00000000f6408f50> (a java.lang.ref.ReferenceQueue$Lock)
                                      at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
                                      - locked <0x00000000f6408f50> (a java.lang.ref.ReferenceQueue$Lock)
                                      at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
                                      at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
                              
                              "Reference Handler" daemon prio=10 tid=0x00007f1e5407a000 nid=0x854c in Object.wait() [0x00007f1e4a5c9000]
                                 java.lang.Thread.State: WAITING (on object monitor)
                                      at java.lang.Object.wait(Native Method)
                                      - waiting on <0x00000000f64086e0> (a java.lang.ref.Reference$Lock)
                                      at java.lang.Object.wait(Object.java:503)
                                      at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
                                      - locked <0x00000000f64086e0> (a java.lang.ref.Reference$Lock)
                              
                              "main" prio=10 tid=0x00007f1e5400a000 nid=0x8546 waiting on condition [0x00007f1e5999c000]
                                 java.lang.Thread.State: WAITING (parking)
                                      at sun.misc.Unsafe.park(Native Method)
                                      - parking to wait for  <0x00000000f55d08c0> (a java.util.concurrent.Semaphore$NonfairSync)
                                      at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
                                      at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
                                      at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
                                      at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
                                      at java.util.concurrent.Semaphore.acquire(Semaphore.java:317)
                                      at org.voltdb.client.SyncCallback.waitForResponse(SyncCallback.java:74)
                                      at org.voltdb.client.ClientImpl.callProcedure(ClientImpl.java:267)
                                      at org.voltdb.client.ClientImpl.callProcedureWithTimeout(ClientImpl.java:226)
                                      at org.voltdb.client.ClientImpl.callProcedure(ClientImpl.java:206)
                                      at org.voltdb.utils.SQLCommand.loadStoredProcedures(SQLCommand.java:1043)
                                      at org.voltdb.utils.SQLCommand.main(SQLCommand.java:1281)
                              
                              "VM Thread" prio=10 tid=0x00007f1e54075800 nid=0x854b runnable
                              
                              "GC task thread#0 (ParallelGC)" prio=10 tid=0x00007f1e54020000 nid=0x8547 runnable
                              
                              "GC task thread#1 (ParallelGC)" prio=10 tid=0x00007f1e54022000 nid=0x8548 runnable
                              
                              "GC task thread#2 (ParallelGC)" prio=10 tid=0x00007f1e54023800 nid=0x8549 runnable
                              
                              "GC task thread#3 (ParallelGC)" prio=10 tid=0x00007f1e54025800 nid=0x854a runnable
                              
                              "VM Periodic Task Thread" prio=10 tid=0x00007f1e540b9000 nid=0x8552 waiting on condition
                              
                              JNI global references: 190

                              Comment

                              Working...
                              X