Announcement

Collapse
No announcement yet.

VoltDB tpc-c-like Benchmark Comparison-Benchmark Description

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • #16
    build.xml

    Originally posted by tcallaghan View Post
    Tuan,

    Can you paste your build.xml file into a forum reply plus include your client and server machine configurations (OS, RAM, CPU, etc.)?

    Thanks,
    Tim
    <?xml version="1.0" ?>
    <project default="default" name="VoltDB">
    <!-- make environment var foo available as env.foo -->
    <property environment="env"/>
    <path id='project.classpath'>
    <pathelement location='${env.HOME}/voltbin/voltdbfat.jar'/>
    <pathelement path="${java.class.path}"/>
    </path>
    <target name="tpcc" description="Call the standard benchmark target configured to use test cluster.">
    <antcall target="benchmark" inheritAll='true'>
    <!-- Number of servers in your cluster -->
    <param name="hostcount" value="1"/>
    <!-- Number of VoltDB partitions per server -->
    <param name="sitesperhost" value="6"/>
    <!-- Number of warehouses for the benchmark, set equal to the number of unique partitions, as in ((hostcount * sitesperhost) / (kfactor + 1)) -->
    <param name="warehouses" value="6"/>
    <!-- Set less than number of warehouses, used to load warehouses in parallel (12 or less) -->
    <param name="loadthreads" value="3" />
    <!-- k-safety value for the benchmark -->
    <param name="kfactor" value="0"/>
    <!-- number of client machines for benchmark -->
    <param name="clientcount" value="1"/>
    <!-- number of client processes per client machine (usually left at 1) -->
    <param name="processesperclient" value="1"/>
    <!-- duration, in milliseconds, for benchmark run (60,000 = 1 minute) -->
    <param name="duration" value="60000" />
    <!-- list of all server hostnames for benchmark -->
    <param name="host1" value="localhost"/>
    <!-- list of all client hostnames for benchmark -->
    <param name="clienthost1" value="localhost"/>
    <!-- JVM heap size for servers -->
    <param name="volt.server.memory" value="512"/>
    <!-- JVM heap size for clients -->
    <param name="volt.client.memory" value="512"/>
    </antcall>
    </target>
    <target name='benchmark'
    description="Compile VoltDB and run a benchmark">
    <java fork="true" failonerror="true" classname="org.voltdb.benchmark.BenchmarkControlle r" >
    <classpath refid='project.classpath' />
    <jvmarg value="-Xmx512m" />
    <assertions><disable/></assertions>
    <arg value="CLIENT=${client}" />
    <arg value="BACKEND=${backend}" />
    <arg value="HOSTCOUNT=${hostcount}" />
    <arg value="SITESPERHOST=${sitesperhost}"/>
    <arg value="KFACTOR=${kfactor}"/>
    <arg value="CLIENTCOUNT=${clientcount}" />
    <arg value="PROCESSESPERCLIENT=${processesperclient}" />
    <arg value="INTERVAL=${interval}" />
    <arg value="DURATION=${duration}" />
    <arg value="REMOTEPATH=${remotepath}" />
    <arg value="REMOTEUSER=${remoteuser}"/>
    <arg value="HOST=${host1}" />
    <arg value="CLIENTHOST=${clienthost1}" />
    <arg value="LISTENFORDEBUGGER=${debug}" />
    <arg value="USEPROFILE=${useprofile}" />
    <arg value="CHECKTRANSACTION=${checktransaction}" />
    <arg value="CLIENTHEAP=${volt.client.memory}" />
    <arg value="SERVERHEAP=${volt.server.memory}" />
    <arg value="LOCAL=${local}" />
    <!-- Actual TXN rate sent to cluster will be:
    TXNRATE * CLIENTCOUNT * PROCESSESPERCLIENT -->
    <arg value="TXNRATE=${txnrate}" />
    <!-- Actual number of connections opened to cluster will be:
    NUMCONNECTIONS * CLIENTCOUNT * PROCESSESPERCLIENT * HOSTCOUNT -->
    <arg value="NUMCONNECTIONS=${numconnections}" />
    <!-- tpcc parameters -->
    <arg value="warehouses=${warehouses}" />
    <arg value="scalefactor=${sf}" />
    <arg value="skewfactor=${skew}" />
    <arg value="loadthreads=${loadthreads}" />
    <!-- automatic online snapshot parameters -->
    <arg value="SNAPSHOTPATH=${snapshotPath}" />
    <arg value="SNAPSHOTFREQUENCY=${snapshotFrequency}" />
    <arg value="SNAPSHOTPREFIX=${snapshotPrefix}" />
    <arg value="SNAPSHOTRETAIN=${snapshotRetain}" />
    </java>
    </target>
    </project>

    Comment


    • #17
      Os, ram, cpu

      $ uname -a
      Linux hanoi 2.6.32-23-generic #37-Ubuntu SMP Fri Jun 11 08:03:28 UTC 2010 x86_64 GNU/Linux
      $ free -m
      total used free shared buffers cached
      Mem: 3961 1732 2228 0 9 143
      -/+ buffers/cache: 1579 2381
      Swap: 8095 903 7191

      $ head -10 /proc/cpuinfo
      processor : 0
      vendor_id : GenuineIntel
      cpu family : 6
      model : 23
      model name : Intel(R) Core(TM)2 Quad CPU Q9650 @ 3.00GHz
      stepping : 10
      cpu MHz : 2004.000
      cache size : 6144 KB
      physical id : 0
      siblings : 4

      Comment


      • #18
        Display types of transactions

        Hi,

        I would be great if the output also provides the number of transactions for each transaction type.
        Thanks,

        Tuan

        Comment


        • #19
          re: number of transactions for each transaction type

          Originally posted by tuancao View Post
          Hi,

          I would be great if the output also provides the number of transactions for each transaction type.

          Thanks,
          Tuan
          Tuan,
          If you look at the output from the benchmark you'll see the following:

          [java] At time 60000 out of 60000 (100%):
          [java] In the past 10000 ms:
          [java] Completed 572811 txns at a rate of 57281.10 txns/s
          [java] Since the benchmark began:
          [java] Completed 3413447 txns at a rate of 56890.78 txns/s
          [java]
          [java] ============================== BENCHMARK RESULTS ==============================
          [java] Time: 60000 ms
          [java] Total transactions: 3413447
          [java] Transactions per second: 56890.78
          [java] Delivery: 136273 total 2271.22 txn/s 136273.00 txn/m
          [java] New Order: 1536975 total 25616.25 txn/s 1536975.00 txn/m
          [java] Order Status: 136623 total 2277.05 txn/s 136623.00 txn/m
          [java] Payment: 1466581 total 24443.02 txn/s 1466581.00 txn/m
          [java] Reset Warehouse: 0 total 0.00 txn/s 0.00 txn/m
          [java] Stock Level: 136995 total 2283.25 txn/s 136995.00 txn/m
          [java] Breakdown by client:
          [java] 10.10.20.155:0: 3413447 total 56890.78 txn/s 3413447.00 txn/m
          [java] ================================================== =============================

          So in this example I'm doing a total of 56,890.78 transactions per second. Below that it shows the total, transactions per second, and transactions per minute of each transaction type.

          Is that what you are looking for?
          -Tim

          Comment


          • #20
            benchmark

            Tim,

            Thank you very much. Can you upload your build.xml? I tried to run the benchmark on my machine but the result wasn't so good.

            Tuan

            Comment


            • #21
              re: benchmark

              Tuan,

              I reviewed your information.

              Given that you are running the benchmark controller, client, and server on a quad-core desktop with 4GB RAM I think your results are appropriate. Try running the benchmark controller and client from a separate machine to get better results.

              -Tim

              Comment


              • #22
                Hi Tim, Could you please give

                Hi Tim,


                Could you please give more details instructions about how to run benchmark controller and client on different machines? I followed the instructions posted above to run TPCC and it works fine when localMood = true, but when I change localMood=false, and let host run on a remote machine, the controller is waiting for server to be ready forever. The last info I can see is:
                [java] localhostINFO: Maximum clock/network skew is 0 milliseconds (according to leader)


                Could you help me figure this out? Thanks a lot!

                Comment


                • #23
                  Running TPC-C

                  We have stopped developing/supporting/using the benchmark controller framework we used for these initial benchmarks. We now run our internal benchmarking using some python scripting and the REST interface that is part of VEM.

                  You can find the TPC-C implementation (driver, schema, project files..) here: https://github.com/VoltDB/voltdb/tree/master/tests/test_apps/tpcc



                  Ryan.

                  Comment


                  • #24
                    Thank you very much, Ryan! I

                    Originally posted by rbetts View Post
                    We have stopped developing/supporting/using the benchmark controller framework we used for these initial benchmarks. We now run our internal benchmarking using some python scripting and the REST interface that is part of VEM.

                    You can find the TPC-C implementation (driver, schema, project files..) here: https://github.com/VoltDB/voltdb/tree/master/tests/test_apps/tpcc

                    Ryan.
                    Thank you very much, Ryan! I will check it out!

                    Comment


                    • #25
                      How to Download the code

                      could u tell me how to get the code.
                      i don't have username/passwd, the account of community didn't work for the svn login

                      Comment


                      • #26
                        The code is available on

                        Originally posted by billmanson View Post
                        could u tell me how to get the code.
                        i don't have username/passwd, the account of community didn't work for the svn login
                        The code is available on github: https://github.com/VoltDB/voltdb/tre...test_apps/tpcc
                        Ryan.

                        Comment


                        • #27
                          Hy,
                          I'm new to everything :) and I have to do TPC-C benchmark on VoltDB and MySQL. So, I have downloaded VMWare version of VoltDB.
                          I saw on this link (https://github.com/VoltDB/voltdb/tre...test_apps/tpcc) a lot of code :) and I would like to know what I have to do to run TPC-C benchmark on my machine.
                          Is there any site that explain TPC-C benchmark for beginners?

                          Thanks

                          Comment


                          • #28
                            Hi Bheath,
                            For VoltDB transactions per second number, can I get them from the Cluster Transactions (tps) graph provided in VoltDB Management Console, or is there any other method to get the numbers? I have attached the snapshot of the graph.12.jpg

                            Comment


                            • #29
                              Originally posted by tcallaghan View Post
                              Tuan,
                              If you look at the output from the benchmark you'll see the following:

                              [java] At time 60000 out of 60000 (100%):
                              [java] In the past 10000 ms:
                              [java] Completed 572811 txns at a rate of 57281.10 txns/s
                              [java] Since the benchmark began:
                              [java] Completed 3413447 txns at a rate of 56890.78 txns/s
                              [java]
                              [java] ============================== BENCHMARK RESULTS ==============================
                              [java] Time: 60000 ms
                              [java] Total transactions: 3413447
                              [java] Transactions per second: 56890.78
                              [java] Delivery: 136273 total 2271.22 txn/s 136273.00 txn/m
                              [java] New Order: 1536975 total 25616.25 txn/s 1536975.00 txn/m
                              [java] Order Status: 136623 total 2277.05 txn/s 136623.00 txn/m
                              [java] Payment: 1466581 total 24443.02 txn/s 1466581.00 txn/m
                              [java] Reset Warehouse: 0 total 0.00 txn/s 0.00 txn/m
                              [java] Stock Level: 136995 total 2283.25 txn/s 136995.00 txn/m
                              [java] Breakdown by client:
                              [java] 10.10.20.155:0: 3413447 total 56890.78 txn/s 3413447.00 txn/m
                              [java] ================================================== =============================

                              So in this example I'm doing a total of 56,890.78 transactions per second. Below that it shows the total, transactions per second, and transactions per minute of each transaction type.

                              Is that what you are looking for?
                              -Tim
                              Hi Tim,
                              How did you measure the txn/sec. Did you measure it from VoltDB Management Console or you do it manually? Also how is this breakup of txn/sec for each type of transaction obtained? If you do it through some java code, can you plz give me the link?

                              Comment


                              • #30
                                Hi sanket,

                                The benchmark is a java application that measures the txn/sec and other metrics using the ClientStats and ClientStatsContext classes that are part of the java client. The Javadoc for these is here: https://docs.voltdb.com/javadoc/java-client-api/

                                For examples of using ClientStats and ClientStatsContext, you can look at any of our apps in the examples folder included in the kit.

                                Another example I can point out is a simple benchmark template (https://github.com/benjaminballard/a.../src/benchmark) that has a helper class BenchmarkStats that handles the boilerplate of collecting and reporting these stats. The Benchmark class has an example of how this is used:

                                BenchmarkStats stats = new BenchmarkStats(client);
                                stats.startBenchmark();
                                ... //do some work
                                stats.endBenchmark();

                                For other clients, and in general, it is easier to capture txn/sec by calling "exec @Statistics PROCEDUREPROFILE 0;" periodically and calculating the differences. This is the same source of data that is used by the VMC graphs. We recently (v6.8) added a script under the tools folder called "watch_performance.py" that periodically calls @Statistics and outputs the tps and other metrics for each procedure executed within the interval. This can be redirected to a log file to capture periodic performance metrics over the course of some long-running workload.

                                -Ben

                                Comment

                                Working...
                                X