Page 1 of 3 123 LastLast
Results 1 to 10 of 27

Thread: VoltDB tpc-c-like Benchmark Comparison-Benchmark Description

  1. #1
    New Member
    Join Date
    Nov 2009
    Posts
    14

    VoltDB tpc-c-like Benchmark Comparison-Benchmark Description

    Benchmark Description

    The benchmark described here is an OLTP system performance test. It simulates an order-entry environment for a business with multiple warehouses. The transactions include entering and delivering orders, recording payments, checking the status of orders, and monitoring the level of stock at the warehouses.
    Hardware

    Node Hardware

    All nodes used for this test had the following identical configuration.

    • Dell R610 1-Unit Chassis
    • 2x 2.66Ghz Quad-Core Xeon 5550.
    • 12x 4GB (48GB) DDR3-1333 Registered ECC DIMMs
    • PERC6/i SAS Controller with 256mb Battery-Backed Cache (set to "Write Back")
    • 3x 72gb 15K RPM 2.5in Enterprise SAS 6Gbps Drives
      • 2x in RAID0 for table/index data
      • 1x in standalone for log data

    • Broadcom Gigabit NIC

    Node Software

    • Centos Linux 5.4
    • Sun Java JDK/JRE 1.6u19
    • RDBMS1
    • MySQL 5.5m3
    • VoltDB 1.0.01

    Switch

    • Dell PowerConnect 6248 48 port Gigabit Ethernet Switch

    RDBMS1 Configuration

    • 4k block size
    • 8GB database cache
    • Table/Index data on 2x RAID0
    • Log Data on 1x standalone
    • results were recorded by checking timestamps of rows in the “orders” table (excluding orders from initial warehouse loading)
    • first 10 seconds of the run were ignored
    • client tested through LAN and directly on server switch

    MySQL Configuration

    • sync_binlog = 1
    • innodb_buffer_pool_size = 8589934592
    • innodb_log_file_size = 2040109465
    • innodb_flush_method = O_DIRECT
    • innodb_thread_concurrency = 0
    • innodb_doublewrite = 1
    • innodb_flush_lot_at_trx_commit = 1
    • Table/Index data on 2x RAID0
    • Log Data (binary and InnoDB) on 1x standalone
    • results were recorded by checking number of new orders created over two minutes
    • client tested through LAN and directly on server switch

    RDBMS1 and MySQL

    • Wait times were set to 0.
    • Transaction type percentages:
      • new order 43.48%
      • payment 43.48%
      • delivery 4.35%
      • stock level 4.35%
      • order status 4.35%


    VoltDB Benchmark

    The VoltDB benchmark differs from the official tpc-c benchmark in two significant ways. Operationally, the VoltDB benchmark does not include any wait times, which we feel are no longer relevant. It also does not include fulfillment of orders submitted to one warehouse, with items from another warehouse (approximately 10% of the new order transactions in the official benchmark). Each benchmark was run with 12 warehouses (partitions) per node.

    The code for the VoltDB benchmark is available here:http://svnmirror.voltdb.com/eng/trun...enchmark/tpcc/

    Results

    Each benchmark was run several times to ensure system tuning/parameters.
    RDBMS1 - Total Transactions/second number shown is derived from peak new orders created for a 60 second interval during the run.

    Nodes Warehouses Clients Total Transactions / Second
    1 3 1 483
    1 3 3 1,134
    1 3 5 1,155
    1 10 5 911
    1 10 10 697

    MySQL - Total Transactions/second number shown derived from total new orders over 2 minute run.

    Nodes Warehouses Clients Total Transactions / Second MySQL Configuration Changes
    1 1 3 232 Base configuration
    1 5 5 208 Base configuration
    1 3 3 229 Base configuration
    1 3 3 232 innodb_doublewrite=0 (potential for partial page writes)
    1 3 3 373 above plus innodb_flush_log_at_trx_commit=0 (write to log 1x per second, flush to disk 1x per second)
    1 3 3 816 above plus sync_binlog=0 (number of binary log writes before flushing to disk, 0 = OS control)

    VoltDB - Total Transactions/second number shown is averaged over a three minute run.

    Nodes Warehouses Clients Total Transactions / Second
    1 12 1 53,000
    3 36 2 150,000
    5 60 3 250,000
    6 72 3 300,000
    12 144 5 560,000

  2. #2

    Benchmark omission

    Would you explain why it was necessary to omit fulfillment of orders submitted to one warehouse with items from another warehouse from the benchmark?

    Regards,

    Iggy Fernandez

  3. #3
    VoltDB Team
    Join Date
    Nov 2009
    Posts
    150

    Not technically necessary...

    Quote Originally Posted by iggy_fernandez View Post
    Would you explain why it was necessary to omit fulfillment of orders submitted to one warehouse with items from another warehouse from the benchmark?

    Regards,

    Iggy Fernandez
    It's a change to the TPC-C spec that benefits VoltDB, which is why we mentioned it prominently in our description of what we did. It's not technically necessary. Since our code partitions on warehouse id, these transactions could be done with a multi-partition transaction. However, since our multi-parition transactions are slower, we've limited our TPC-C-like benchmark to single-partition transactions for the time being.

    Similarly, we used to run our TPC-C like benchmark without rolling back any transactions, even though the spec called for a small number of rollbacks. Once we supported robust rollback, we removed this caveat and now run the benchmark with rollbacks. In the future, as we further optimize our multi-partition transaction support, we would love to better meet the spec in this area.

    I would like to point out that VoltDB is still performing an average of about 26 SQL statements per transaction (if memory serves), including joins, sorting and aggregation. This works out to well over a ten million SQL statements per second on our twelve node cluster.


    (Note, this response was also posted to the same question in a blog post comment: http://blog.voltdb.com/key-value-benchmarking/

  4. #4

    The Payment transaction

    Quote Originally Posted by jhugg View Post
    It's a change to the TPC-C spec that benefits VoltDB, which is why we mentioned it prominently in our description of what we did. It's not technically necessary. ..

    (Note, this response was also posted to the same question in a blog post comment: http://blog.voltdb.com/key-value-benchmarking/
    The Payment transaction is also multi-partition. It appears that it was broken into two independently committed transactions for the purposes of the benchmark: the first transaction handles the updates to the WAREHOUSE and HISTORY tables (partitionInfo = "WAREHOUSE.W_ID: 0") and the second transaction handles the updates to the CUSTOMER table (partitionInfo = "CUSTOMER.C_W_ID: 3").
    Can you confirm that the Payment transaction was split into two independently committed transactions for the purpose of the benchmark? The relevant code segment is as follows:
    http://svnmirror.voltdb.com/eng/trun...PCCClient.java
    public void callPaymentById(short w_id, byte d_id, double h_amount,
    short c_w_id, byte c_d_id, int c_id, TimestampType now) throws IOException {
    if (m_blockOnBackpressure) {
    if (m_scaleParams.warehouses > 1) {
    final VerifyBasicCallback cb = new VerifyBasicCallback();
    while(!m_voltClient.callProcedure( cb,
    Constants.PAYMENT_BY_ID_W,
    w_id, d_id, h_amount, c_w_id, c_d_id, c_id, now)) {
    try {
    m_voltClient.backpressureBarrier();
    } catch (InterruptedException e) {
    // TODO Auto-generated catch block
    e.printStackTrace();
    }
    }
    final VerifyBasicCallback cb2 = new VerifyBasicCallback(TPCCSimulation.Transaction.PAY MENT,
    Constants.PAYMENT_BY_ID_C);
    while (!m_voltClient.callProcedure( cb2,
    Constants.PAYMENT_BY_ID_C,
    w_id, d_id, h_amount, c_w_id, c_d_id, c_id, now)) {
    try {
    m_voltClient.backpressureBarrier();
    } catch (InterruptedException e) {
    // TODO Auto-generated catch block
    e.printStackTrace();
    }
    }
    }
    else {
    final VerifyBasicCallback cb = new VerifyBasicCallback(TPCCSimulation.Transaction.PAY MENT,
    Constants.PAYMENT_BY_ID);
    while (!m_voltClient.callProcedure( cb,
    Constants.PAYMENT_BY_ID,
    w_id, d_id, h_amount, c_w_id, c_d_id, c_id, now)) {
    try {
    m_voltClient.backpressureBarrier();
    } catch (InterruptedException e) {
    // TODO Auto-generated catch block
    e.printStackTrace();
    }
    }
    }
    } else {
    if (m_scaleParams.warehouses > 1) {
    m_voltClient.callProcedure(new VerifyBasicCallback(),
    Constants.PAYMENT_BY_ID_W,
    w_id, d_id, h_amount, c_w_id, c_d_id, c_id, now);
    m_queuedMessage = m_voltClient.callProcedure(new VerifyBasicCallback(TPCCSimulation.Transaction.PAY MENT,
    Constants.PAYMENT_BY_ID_C),
    Constants.PAYMENT_BY_ID_C,
    w_id, d_id, h_amount, c_w_id, c_d_id, c_id, now);
    }
    else {
    m_queuedMessage = m_voltClient.callProcedure(new VerifyBasicCallback(TPCCSimulation.Transaction.PAY MENT,
    Constants.PAYMENT_BY_ID),
    Constants.PAYMENT_BY_ID,
    w_id, d_id, h_amount, c_w_id, c_d_id, c_id, now);
    }
    }
    }
    Regards,
    Iggy Fernandez

  5. #5

    K-Safety

    What value was used for K-Safety in the benchmark?


    Regards,


    Iggy Fernandez

  6. #6
    VoltDB Team
    Join Date
    Nov 2009
    Posts
    150

    No replication was used.

    Quote Originally Posted by iggy_fernandez View Post
    What value was used for K-Safety in the benchmark?

    Regards,

    Iggy Fernandez
    If you run with 2-copies (k=1), then it runs half as fast on the same number of machines. It still scales at the same rate as you add machines. Feel free to run it yourself. Let me know if you need help.

  7. #7
    VoltDB Team
    Join Date
    Nov 2009
    Posts
    150

    Yes

    Quote Originally Posted by iggy_fernandez View Post
    The Payment transaction is also multi-partition. It appears that it was broken into two independently committed transactions for the purposes of the benchmark: the first transaction handles the updates to the WAREHOUSE and HISTORY tables ...
    Iggy Fernandez
    I can confirm this.

  8. #8

    Thanks for the clarifications

    Quote Originally Posted by jhugg View Post
    I can confirm this.
    Thanks for the clarifications.


    Regards,


    Iggy Fernandez

  9. #9

    Running benchmark code

    Hi,


    Can you tell me a bit how I can build and run the benchmark? (The first step for me to modify the benchmark code)


    Thanks,
    Tuan

  10. #10
    New Member
    Join Date
    Nov 2009
    Posts
    14

    re: Running benchmark code

    Quote Originally Posted by tuancao View Post
    Hi,

    Can you tell me a bit how I can build and run the benchmark? (The first step for me to modify the benchmark code)

    Thanks,
    Tuan
    Hi Tuan,
    I see that the README only describes tpc-c and has no instructions. We will get you some instructions on Monday.


    Can you share how you plan to modify the benchmark?


    Regards,
    Bobbi

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •