Announcement

Collapse
No announcement yet.

VoltDB tpc-c-like Benchmark Comparison-Benchmark Description

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • VoltDB tpc-c-like Benchmark Comparison-Benchmark Description

    Benchmark Description

    The benchmark described here is an OLTP system performance test. It simulates an order-entry environment for a business with multiple warehouses. The transactions include entering and delivering orders, recording payments, checking the status of orders, and monitoring the level of stock at the warehouses.
    Hardware

    Node Hardware

    All nodes used for this test had the following identical configuration.
    • Dell R610 1-Unit Chassis
    • 2x 2.66Ghz Quad-Core Xeon 5550.
    • 12x 4GB (48GB) DDR3-1333 Registered ECC DIMMs
    • PERC6/i SAS Controller with 256mb Battery-Backed Cache (set to "Write Back")
    • 3x 72gb 15K RPM 2.5in Enterprise SAS 6Gbps Drives
      • 2x in RAID0 for table/index data
      • 1x in standalone for log data

    • Broadcom Gigabit NIC

    Node Software

    • Centos Linux 5.4
    • Sun Java JDK/JRE 1.6u19
    • RDBMS1
    • MySQL 5.5m3
    • VoltDB 1.0.01

    Switch

    • Dell PowerConnect 6248 48 port Gigabit Ethernet Switch

    RDBMS1 Configuration

    • 4k block size
    • 8GB database cache
    • Table/Index data on 2x RAID0
    • Log Data on 1x standalone
    • results were recorded by checking timestamps of rows in the “orders” table (excluding orders from initial warehouse loading)
    • first 10 seconds of the run were ignored
    • client tested through LAN and directly on server switch

    MySQL Configuration

    • sync_binlog = 1
    • innodb_buffer_pool_size = 8589934592
    • innodb_log_file_size = 2040109465
    • innodb_flush_method = O_DIRECT
    • innodb_thread_concurrency = 0
    • innodb_doublewrite = 1
    • innodb_flush_lot_at_trx_commit = 1
    • Table/Index data on 2x RAID0
    • Log Data (binary and InnoDB) on 1x standalone
    • results were recorded by checking number of new orders created over two minutes
    • client tested through LAN and directly on server switch

    RDBMS1 and MySQL

    • Wait times were set to 0.
    • Transaction type percentages:
      • new order 43.48%
      • payment 43.48%
      • delivery 4.35%
      • stock level 4.35%
      • order status 4.35%


    VoltDB Benchmark

    The VoltDB benchmark differs from the official tpc-c benchmark in two significant ways. Operationally, the VoltDB benchmark does not include any wait times, which we feel are no longer relevant. It also does not include fulfillment of orders submitted to one warehouse, with items from another warehouse (approximately 10% of the new order transactions in the official benchmark). Each benchmark was run with 12 warehouses (partitions) per node.

    The code for the VoltDB benchmark is available here:http://svnmirror.voltdb.com/eng/trun...enchmark/tpcc/

    Results

    Each benchmark was run several times to ensure system tuning/parameters.
    RDBMS1 - Total Transactions/second number shown is derived from peak new orders created for a 60 second interval during the run.

    3 1 483
    3 3 1,134
    3 5 1,155
    10 5 911
    10 10 697
    MySQL - Total Transactions/second number shown derived from total new orders over 2 minute run.

    1 3 232 Base configuration
    5 5 208 Base configuration
    3 3 229 Base configuration
    3 3 232 innodb_doublewrite=0 (potential for partial page writes)
    3 3 373 above plus innodb_flush_log_at_trx_commit=0 (write to log 1x per second, flush to disk 1x per second)
    3 3 816 above plus sync_binlog=0 (number of binary log writes before flushing to disk, 0 = OS control)
    VoltDB - Total Transactions/second number shown is averaged over a three minute run.

    12 1 53,000
    36 2 150,000
    60 3 250,000
    72 3 300,000
    144 5 560,000

  • #2
    Benchmark omission

    Would you explain why it was necessary to omit fulfillment of orders submitted to one warehouse with items from another warehouse from the benchmark?

    Regards,

    Iggy Fernandez

    Comment


    • #3
      Not technically necessary...

      Originally posted by iggy_fernandez View Post
      Would you explain why it was necessary to omit fulfillment of orders submitted to one warehouse with items from another warehouse from the benchmark?

      Regards,

      Iggy Fernandez
      It's a change to the TPC-C spec that benefits VoltDB, which is why we mentioned it prominently in our description of what we did. It's not technically necessary. Since our code partitions on warehouse id, these transactions could be done with a multi-partition transaction. However, since our multi-parition transactions are slower, we've limited our TPC-C-like benchmark to single-partition transactions for the time being.

      Similarly, we used to run our TPC-C like benchmark without rolling back any transactions, even though the spec called for a small number of rollbacks. Once we supported robust rollback, we removed this caveat and now run the benchmark with rollbacks. In the future, as we further optimize our multi-partition transaction support, we would love to better meet the spec in this area.

      I would like to point out that VoltDB is still performing an average of about 26 SQL statements per transaction (if memory serves), including joins, sorting and aggregation. This works out to well over a ten million SQL statements per second on our twelve node cluster.


      (Note, this response was also posted to the same question in a blog post comment: http://blog.voltdb.com/key-value-benchmarking/

      Comment


      • #4
        The Payment transaction

        Originally posted by jhugg View Post
        It's a change to the TPC-C spec that benefits VoltDB, which is why we mentioned it prominently in our description of what we did. It's not technically necessary. ..

        (Note, this response was also posted to the same question in a blog post comment: http://blog.voltdb.com/key-value-benchmarking/
        The Payment transaction is also multi-partition. It appears that it was broken into two independently committed transactions for the purposes of the benchmark: the first transaction handles the updates to the WAREHOUSE and HISTORY tables (partitionInfo = "WAREHOUSE.W_ID: 0") and the second transaction handles the updates to the CUSTOMER table (partitionInfo = "CUSTOMER.C_W_ID: 3").
        Can you confirm that the Payment transaction was split into two independently committed transactions for the purpose of the benchmark? The relevant code segment is as follows:
        http://svnmirror.voltdb.com/eng/trun...PCCClient.java
        public void callPaymentById(short w_id, byte d_id, double h_amount,
        short c_w_id, byte c_d_id, int c_id, TimestampType now) throws IOException {
        if (m_blockOnBackpressure) {
        if (m_scaleParams.warehouses > 1) {
        final VerifyBasicCallback cb = new VerifyBasicCallback();
        while(!m_voltClient.callProcedure( cb,
        Constants.PAYMENT_BY_ID_W,
        w_id, d_id, h_amount, c_w_id, c_d_id, c_id, now)) {
        try {
        m_voltClient.backpressureBarrier();
        } catch (InterruptedException e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
        }
        }
        final VerifyBasicCallback cb2 = new VerifyBasicCallback(TPCCSimulation.Transaction.PAY MENT,
        Constants.PAYMENT_BY_ID_C);
        while (!m_voltClient.callProcedure( cb2,
        Constants.PAYMENT_BY_ID_C,
        w_id, d_id, h_amount, c_w_id, c_d_id, c_id, now)) {
        try {
        m_voltClient.backpressureBarrier();
        } catch (InterruptedException e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
        }
        }
        }
        else {
        final VerifyBasicCallback cb = new VerifyBasicCallback(TPCCSimulation.Transaction.PAY MENT,
        Constants.PAYMENT_BY_ID);
        while (!m_voltClient.callProcedure( cb,
        Constants.PAYMENT_BY_ID,
        w_id, d_id, h_amount, c_w_id, c_d_id, c_id, now)) {
        try {
        m_voltClient.backpressureBarrier();
        } catch (InterruptedException e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
        }
        }
        }
        } else {
        if (m_scaleParams.warehouses > 1) {
        m_voltClient.callProcedure(new VerifyBasicCallback(),
        Constants.PAYMENT_BY_ID_W,
        w_id, d_id, h_amount, c_w_id, c_d_id, c_id, now);
        m_queuedMessage = m_voltClient.callProcedure(new VerifyBasicCallback(TPCCSimulation.Transaction.PAY MENT,
        Constants.PAYMENT_BY_ID_C),
        Constants.PAYMENT_BY_ID_C,
        w_id, d_id, h_amount, c_w_id, c_d_id, c_id, now);
        }
        else {
        m_queuedMessage = m_voltClient.callProcedure(new VerifyBasicCallback(TPCCSimulation.Transaction.PAY MENT,
        Constants.PAYMENT_BY_ID),
        Constants.PAYMENT_BY_ID,
        w_id, d_id, h_amount, c_w_id, c_d_id, c_id, now);
        }
        }
        }
        Regards,
        Iggy Fernandez

        Comment


        • #5
          K-Safety

          What value was used for K-Safety in the benchmark?


          Regards,


          Iggy Fernandez

          Comment


          • #6
            No replication was used.

            Originally posted by iggy_fernandez View Post
            What value was used for K-Safety in the benchmark?

            Regards,

            Iggy Fernandez
            If you run with 2-copies (k=1), then it runs half as fast on the same number of machines. It still scales at the same rate as you add machines. Feel free to run it yourself. Let me know if you need help.

            Comment


            • #7
              Yes

              Originally posted by iggy_fernandez View Post
              The Payment transaction is also multi-partition. It appears that it was broken into two independently committed transactions for the purposes of the benchmark: the first transaction handles the updates to the WAREHOUSE and HISTORY tables ...
              Iggy Fernandez
              I can confirm this.

              Comment


              • #8
                Thanks for the clarifications

                Originally posted by jhugg View Post
                I can confirm this.
                Thanks for the clarifications.


                Regards,


                Iggy Fernandez

                Comment


                • #9
                  Running benchmark code

                  Hi,


                  Can you tell me a bit how I can build and run the benchmark? (The first step for me to modify the benchmark code)


                  Thanks,
                  Tuan

                  Comment


                  • #10
                    re: Running benchmark code

                    Originally posted by tuancao View Post
                    Hi,

                    Can you tell me a bit how I can build and run the benchmark? (The first step for me to modify the benchmark code)

                    Thanks,
                    Tuan
                    Hi Tuan,
                    I see that the README only describes tpc-c and has no instructions. We will get you some instructions on Monday.


                    Can you share how you plan to modify the benchmark?


                    Regards,
                    Bobbi

                    Comment


                    • #11
                      re: Running benchmark code

                      Originally posted by tuancao View Post
                      Hi,

                      Can you tell me a bit how I can build and run the benchmark? (The first step for me to modify the benchmark code)

                      Thanks,
                      Tuan
                      Tuan,
                      Here is a list of steps to run the benchmark, let me know if you need additional clarification. We plan on making this part of our more formal documentation set down the road.

                      1. Setup password-less SSH

                      - On each machine you plan to run the benchmark on, you need to add your account's public key to the file \~/.ssh/authorized_keys and/or \~/.ssh/authorized_keys2. The name of the file you need to add depends on the ssh version you're using, but to be safe, you can add the key to both. You may also need to change permissions on these files using "chmod 600 authorized_keys".
                      - Check that you can ssh to the machines without a password.
                      - If the machines use a shared home directory, you probably only need to add/change these files in one place.


                      2. Build/install VoltDB for benchmarking

                      - To execute our benchmarking infrastructure, you need to build VoltDB from source using the "voltbin" ant target.
                      - Get a current copy of VoltDB source using "svn checkout http://svnmirror.voltdb.com/eng/trunk/"
                      - In the root directory of VoltDB source ("cd trunk" if you just checked out source), execute "ant voltbin" to create "obj/release/voltbin" with the required components.
                      - In your home folder on all machines you plan to run the benchmark on (server or clients) as well as the machine you are initiating the benchmark from, create a folder named "voltbin" and copy all the files from obj/release/voltbin


                      3. Modify the "tpcc" target in build.xml

                      - Comments are inline, save this as a file named build.xml

                      <?xml version="1.0" ?>
                      <project default="default" name="VoltDB">

                      <!-- make environment var foo available as env.foo -->
                      <property environment="env"/>

                      <path id='project.classpath'>
                      <pathelement location='${env.HOME}/voltbin/voltdbfat.jar'/>
                      <pathelement path="${java.class.path}"/>
                      </path>

                      <target name="tpcc" description="Call the standard benchmark target configured to use test cluster.">
                      <antcall target="benchmark" inheritAll='true'>
                      <!-- Number of servers in your cluster -->
                      <param name="hostcount" value="1"/>
                      <!-- Number of VoltDB partitions per server -->
                      <param name="sitesperhost" value="12"/>
                      <!-- Number of warehouses for the benchmark, set equal to the number of unique partitions, as in ((hostcount * sitesperhost) / (kfactor + 1)) -->
                      <param name="warehouses" value="12"/>
                      <!-- Set less than number of warehouses, used to load warehouses in parallel (12 or less) -->
                      <param name="loadthreads" value="6" />
                      <!-- k-safety value for the benchmark -->
                      <param name="kfactor" value="0"/>
                      <!-- number of client machines for benchmark -->
                      <param name="clientcount" value="1"/>
                      <!-- number of client processes per client machine (usually left at 1) -->
                      <param name="processesperclient" value="1"/>
                      <!-- duration, in milliseconds, for benchmark run (60,000 = 1 minute) -->
                      <param name="duration" value="60000" />
                      <!-- list of all server hostnames for benchmark -->
                      <param name="host1" value="volt3l"/>
                      <param name="host2" value="volt3b"/>
                      <param name="host3" value="volt3c"/>
                      <param name="host4" value="volt3d"/>
                      <param name="host5" value="volt3e"/>
                      <param name="host6" value="volt3f"/>
                      <param name="host7" value="volt3g"/>
                      <param name="host8" value="volt3h"/>
                      <param name="host9" value="volt3i"/>
                      <param name="host10" value="volt3j"/>
                      <param name="host11" value="volt3k"/>
                      <param name="host12" value="volt3l"/>
                      <!-- list of all client hostnames for benchmark -->
                      <param name="clienthost1" value="localhost"/>
                      <param name="clienthost2" value="volt4b"/>
                      <param name="clienthost3" value="volt4c"/>
                      <!-- JVM heap size for servers -->
                      <param name="volt.server.memory" value="2048"/>
                      <!-- JVM heap size for clients -->
                      <param name="volt.client.memory" value="1024"/>
                      </antcall>
                      </target>

                      <target name='benchmark'
                      description="Compile VoltDB and run a benchmark">
                      <java fork="true" failonerror="true" classname="org.voltdb.benchmark.BenchmarkControlle r" >
                      <classpath refid='project.classpath' />
                      <jvmarg value="-Xmx512m" />
                      <assertions><disable/></assertions>
                      <arg value="CLIENT=${client}" />
                      <arg value="BACKEND=${backend}" />
                      <arg value="HOSTCOUNT=${hostcount}" />
                      <arg value="SITESPERHOST=${sitesperhost}"/>
                      <arg value="KFACTOR=${kfactor}"/>
                      <arg value="CLIENTCOUNT=${clientcount}" />
                      <arg value="PROCESSESPERCLIENT=${processesperclient}" />
                      <arg value="INTERVAL=${interval}" />
                      <arg value="DURATION=${duration}" />
                      <arg value="REMOTEPATH=${remotepath}" />
                      <arg value="REMOTEUSER=${remoteuser}"/>
                      <arg value="HOST=${host1}" />
                      <arg value="HOST=${host2}" />
                      <arg value="HOST=${host3}" />
                      <arg value="HOST=${host4}" />
                      <arg value="HOST=${host5}" />
                      <arg value="HOST=${host6}" />
                      <arg value="HOST=${host7}" />
                      <arg value="HOST=${host8}" />
                      <arg value="HOST=${host9}" />
                      <arg value="HOST=${host10}" />
                      <arg value="HOST=${host11}" />
                      <arg value="HOST=${host12}" />
                      <arg value="CLIENTHOST=${clienthost1}" />
                      <arg value="CLIENTHOST=${clienthost2}" />
                      <arg value="CLIENTHOST=${clienthost3}" />
                      <arg value="LISTENFORDEBUGGER=${debug}" />
                      <arg value="USEPROFILE=${useprofile}" />
                      <arg value="CHECKTRANSACTION=${checktransaction}" />
                      <arg value="CLIENTHEAP=${volt.client.memory}" />
                      <arg value="SERVERHEAP=${volt.server.memory}" />
                      <arg value="LOCAL=${local}" />
                      <!-- Actual TXN rate sent to cluster will be:
                      TXNRATE * CLIENTCOUNT * PROCESSESPERCLIENT -->
                      <arg value="TXNRATE=${txnrate}" />
                      <!-- Actual number of connections opened to cluster will be:
                      NUMCONNECTIONS * CLIENTCOUNT * PROCESSESPERCLIENT * HOSTCOUNT -->
                      <arg value="NUMCONNECTIONS=${numconnections}" />

                      <!-- tpcc parameters -->
                      <arg value="warehouses=${warehouses}" />
                      <arg value="scalefactor=${sf}" />
                      <arg value="skewfactor=${skew}" />
                      <arg value="loadthreads=${loadthreads}" />

                      <!-- automatic online snapshot parameters -->
                      <arg value="SNAPSHOTPATH=${snapshotPath}" />
                      <arg value="SNAPSHOTFREQUENCY=${snapshotFrequency}" />
                      <arg value="SNAPSHOTPREFIX=${snapshotPrefix}" />
                      <arg value="SNAPSHOTRETAIN=${snapshotRetain}" />
                      </java>
                      </target>

                      </project>

                      4. Run the benchmark

                      - On the machine initiating the benchmark in the directory containing build.xml, run "ant tpcc".

                      Comment


                      • #12
                        memory requirement

                        Originally posted by tcallaghan View Post
                        Tuan,
                        Here is a list of steps to run the benchmark, let me know if you need additional clarification. We plan on making this part of our more formal documentation set down the road.

                        1. Setup password-less SSH
                        ...

                        - On the machine initiating the benchmark in the directory containing build.xml, run "ant tpcc".
                        Tim,
                        Thank you very much. The instruction is very helpful. I have just tried to run it on my local machine. ( used only host1 and set it to localhost, removed all the information about remote nodes, ran with 6 warehouses,...). It seemed to use all of my memory, there were a lot of swapping. Then it gave the following error:

                        [java] (23094069)localhost: 233681 [PeriodicWork] WARN NETWORK - Closing connection to org.voltdb.network.VoltPort@66e64686:localhost/127.0.0.1:58171 at 1279252122248 because it refuses to read responses
                        [java] In the past 10000 ms:
                        [java] Completed 0 txns at a rate of 0.00 txns/s
                        [java] Since the benchmark began:
                        [java] Completed 0 txns at a rate of 0.00 txns/s
                        By the way, I have copied my build.xml as below:

                        <<?xml version="1.0" ?>

                        <\code>

                        Also, do you know how much memory is required to store all the TPCC database in memory? If I remember correctly from one of the hstore papers, they stated that 100MB per warehouse. So if we run with 12 warehouses, then we need only about 1.2GB of memory. My machine has 4GB of memory. I don't know why all 4GB was fully used.

                        Thanks,
                        Tuan

                        Comment


                        • #13
                          re: memory requirement

                          Originally posted by tuancao View Post
                          Tim,
                          Thank you very much. The instruction is very helpful. I have just tried to run it on my local machine. ( used only host1 and set it to localhost, removed all the information about remote nodes, ran with 6 warehouses,...). It seemed to use all of my memory, there were a lot of swapping. ...
                          Tuan
                          Tuan,

                          You can make a few changes to the build.xml file to make more memory available to the VoltDB server for data/index. The following 2 lines are important:

                          <!-- JVM heap size for servers -->
                          <param name="volt.server.memory" value="2048"/>
                          <!-- JVM heap size for clients -->
                          <param name="volt.client.memory" value="1024"/>

                          "volt.server.memory" is the JVM heap size for your VoltDB server, and is currently using 2GB (the value is in MB). Since you are running a simple benchmark you can probably reduce this to 512 and rerun without issue. You didn't mention where you were running your client from but you can safely reduce "volt.client.memory" to 512 as well for the benchmark.

                          Let me know if that helps.

                          -Tim

                          Comment


                          • #14
                            memory requirement

                            Originally posted by tcallaghan View Post
                            Tuan,

                            You can make a few changes to the build.xml file to make more memory available to the VoltDB server for data/index. The following 2 lines are important:

                            <!-- JVM heap size for servers -->
                            <param name="volt.server.memory" value="2048"/>
                            <!-- JVM heap size for clients -->
                            <param name="volt.client.memory" value="1024"/>

                            -Tim
                            Hi Tim,
                            Thanks for the response.
                            I reduced volt.server.memory and volt.client.memory both to 512. This is the result:
                            [java] ============================== BENCHMARK RESULTS ==============================
                            [java] Time: 60000 ms
                            [java] Total transactions: 317
                            [java] Transactions per second: 5.28
                            [java] Delivery: 16 total 0.27 txn/s 16.00 txn/m
                            [java] New Order: 142 total 2.37 txn/s 142.00 txn/m
                            [java] Order Status: 18 total 0.30 txn/s 18.00 txn/m
                            [java] Payment: 135 total 2.25 txn/s 135.00 txn/m
                            [java] Reset Warehouse: 0 total 0.00 txn/s 0.00 txn/m
                            [java] Stock Level: 6 total 0.10 txn/s 6.00 txn/m
                            [java] Breakdown by client:
                            [java] localhost:0: 317 total 5.28 txn/s 317.00 txn/m
                            [java] ================================================== =============================
                            I noticed there are still a lot of swapping happened.

                            One thing I would like to know is the amount of memory to store the TPCC database. Do you have any idea how big it is?

                            Thanks,
                            Tuan

                            Comment


                            • #15
                              re: memory requirement

                              Originally posted by tuancao View Post
                              Hi Tim,
                              Thanks for the response.
                              I reduced volt.server.memory and volt.client.memory both to 512. This is the result:
                              [java] ============================== BENCHMARK RESULTS
                              Tuan
                              Tuan,

                              Can you paste your build.xml file into a forum reply plus include your client and server machine configurations (OS, RAM, CPU, etc.)?


                              Thanks,

                              Tim

                              Comment

                              Working...
                              X