Forum: Other

Post: VoltDB tpc-c-like Benchmark Comparison-Benchmark Description

VoltDB tpc-c-like Benchmark Comparison-Benchmark Description
bheath
May 26, 2010
Benchmark Description
The benchmark described here is an OLTP system performance test. It simulates an order-entry environment for a business with multiple warehouses. The transactions include entering and delivering orders, recording payments, checking the status of orders, and monitoring the level of stock at the warehouses.
Hardware
Node Hardware
All nodes used for this test had the following identical configuration.


  • Dell R610 1-Unit Chassis
  • 2x 2.66Ghz Quad-Core Xeon 5550.
  • 12x 4GB (48GB) DDR3-1333 Registered ECC DIMMs
  • PERC6/i SAS Controller with 256mb Battery-Backed Cache (set to "Write Back")
  • 3x 72gb 15K RPM 2.5in Enterprise SAS 6Gbps Drives

    • 2x in RAID0 for table/index data
    • 1x in standalone for log data


  • Broadcom Gigabit NIC

Node Software


  • Centos Linux 5.4
  • Sun Java JDK/JRE 1.6u19
  • RDBMS1
  • MySQL 5.5m3
  • VoltDB 1.0.01

Switch


  • Dell PowerConnect 6248 48 port Gigabit Ethernet Switch

RDBMS1 Configuration


  • 4k block size
  • 8GB database cache
  • Table/Index data on 2x RAID0
  • Log Data on 1x standalone
  • results were recorded by checking timestamps of rows in the “orders” table (excluding orders from initial warehouse loading)
  • first 10 seconds of the run were ignored
  • client tested through LAN and directly on server switch

MySQL Configuration


  • sync_binlog = 1
  • innodb_buffer_pool_size = 8589934592
  • innodb_log_file_size = 2040109465
  • innodb_flush_method = O_DIRECT
  • innodb_thread_concurrency = 0
  • innodb_doublewrite = 1
  • innodb_flush_lot_at_trx_commit = 1
  • Table/Index data on 2x RAID0
  • Log Data (binary and InnoDB) on 1x standalone
  • results were recorded by checking number of new orders created over two minutes
  • client tested through LAN and directly on server switch

RDBMS1 and MySQL


  • Wait times were set to 0.
  • Transaction type percentages:

    • new order 43.48%
    • payment 43.48%
    • delivery 4.35%
    • stock level 4.35%
    • order status 4.35%



VoltDB Benchmark
The VoltDB benchmark differs from the official tpc-c benchmark in two significant ways. Operationally, the VoltDB benchmark does not include any wait times, which we feel are no longer relevant. It also does not include fulfillment of orders submitted to one warehouse, with items from another warehouse (approximately 10% of the new order transactions in the official benchmark). Each benchmark was run with 12 warehouses (partitions) per node.

The code for the VoltDB benchmark is available here:http://svnmirror.voltdb.com/eng/trunk/tests/frontend/org/voltdb/benchmark/tpcc/

Results
Each benchmark was run several times to ensure system tuning/parameters.
RDBMS1 - Total Transactions/second number shown is derived from peak new orders created for a 60 second interval during the run.





































Nodes Warehouses Clients Total Transactions / Second
1 3 1 483
1 3 3 1,134
1 3 5 1,155
1 10 5 911
1 10 10 697


MySQL - Total Transactions/second number shown derived from total new orders over 2 minute run.


















































Nodes Warehouses Clients Total Transactions / Second MySQL Configuration Changes
1 1 3 232 Base configuration
1 5 5 208 Base configuration
1 3 3 229 Base configuration
1 3 3 232 innodb_doublewrite=0 (potential for partial page writes)
1 3 3 373 above plus innodb_flush_log_at_trx_commit=0 (write to log 1x per second, flush to disk 1x per second)
1 3 3 816 above plus sync_binlog=0 (number of binary log writes before flushing to disk, 0 = OS control)


VoltDB - Total Transactions/second number shown is averaged over a three minute run.





































Nodes Warehouses Clients Total Transactions / Second
1 12 1 53,000
3 36
2 150,000
5 60
3 250,000
6 72
3 300,000
12 144
5 560,000
Benchmark omission
iggy_fernandez
May 28, 2010
Would you explain why it was necessary to omit fulfillment of orders submitted to one warehouse with items from another warehouse from the benchmark?

Regards,

Iggy Fernandez
Not technically necessary...
jhugg
May 31, 2010
Would you explain why it was necessary to omit fulfillment of orders submitted to one warehouse with items from another warehouse from the benchmark?

Regards,

Iggy Fernandez


It's a change to the TPC-C spec that benefits VoltDB, which is why we mentioned it prominently in our description of what we did. It's not technically necessary. Since our code partitions on warehouse id, these transactions could be done with a multi-partition transaction. However, since our multi-parition transactions are slower, we've limited our TPC-C-like benchmark to single-partition transactions for the time being.

Similarly, we used to run our TPC-C like benchmark without rolling back any transactions, even though the spec called for a small number of rollbacks. Once we supported robust rollback, we removed this caveat and now run the benchmark with rollbacks. In the future, as we further optimize our multi-partition transaction support, we would love to better meet the spec in this area.

I would like to point out that VoltDB is still performing an average of about 26 SQL statements per transaction (if memory serves), including joins, sorting and aggregation. This works out to well over a ten million SQL statements per second on our twelve node cluster.


(Note, this response was also posted to the same question in a blog post comment: http://blog.voltdb.com/key-value-benchmarking/
The Payment transaction
iggy_fernandez
Jun 5, 2010
It's a change to the TPC-C spec that benefits VoltDB, which is why we mentioned it prominently in our description of what we did. It's not technically necessary. ..

(Note, this response was also posted to the same question in a blog post comment: http://blog.voltdb.com/key-value-benchmarking/


The Payment transaction is also multi-partition. It appears that it was broken into two independently committed transactions for the purposes of the benchmark: the first transaction handles the updates to the WAREHOUSE and HISTORY tables (partitionInfo = "WAREHOUSE.W_ID: 0") and the second transaction handles the updates to the CUSTOMER table (partitionInfo = "CUSTOMER.C_W_ID: 3").
Can you confirm that the Payment transaction was split into two independently committed transactions for the purpose of the benchmark? The relevant code segment is as follows:
http://svnmirror.voltdb.com/eng/trunk/tests/frontend/org/voltdb/benchmark/tpcc/TPCCClient.java
public void callPaymentById(short w_id, byte d_id, double h_amount,
short c_w_id, byte c_d_id, int c_id, TimestampType now) throws IOException {
if (m_blockOnBackpressure) {
if (m_scaleParams.warehouses > 1) {
final VerifyBasicCallback cb = new VerifyBasicCallback();
while(!m_voltClient.callProcedure( cb,
Constants.PAYMENT_BY_ID_W,
w_id, d_id, h_amount, c_w_id, c_d_id, c_id, now)) {
try {
m_voltClient.backpressureBarrier();
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
final VerifyBasicCallback cb2 = new VerifyBasicCallback(TPCCSimulation.Transaction.PAYMENT,
Constants.PAYMENT_BY_ID_C);
while (!m_voltClient.callProcedure( cb2,
Constants.PAYMENT_BY_ID_C,
w_id, d_id, h_amount, c_w_id, c_d_id, c_id, now)) {
try {
m_voltClient.backpressureBarrier();
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
else {
final VerifyBasicCallback cb = new VerifyBasicCallback(TPCCSimulation.Transaction.PAYMENT,
Constants.PAYMENT_BY_ID);
while (!m_voltClient.callProcedure( cb,
Constants.PAYMENT_BY_ID,
w_id, d_id, h_amount, c_w_id, c_d_id, c_id, now)) {
try {
m_voltClient.backpressureBarrier();
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
} else {
if (m_scaleParams.warehouses > 1) {
m_voltClient.callProcedure(new VerifyBasicCallback(),
Constants.PAYMENT_BY_ID_W,
w_id, d_id, h_amount, c_w_id, c_d_id, c_id, now);
m_queuedMessage = m_voltClient.callProcedure(new VerifyBasicCallback(TPCCSimulation.Transaction.PAYMENT,
Constants.PAYMENT_BY_ID_C),
Constants.PAYMENT_BY_ID_C,
w_id, d_id, h_amount, c_w_id, c_d_id, c_id, now);
}
else {
m_queuedMessage = m_voltClient.callProcedure(new VerifyBasicCallback(TPCCSimulation.Transaction.PAYMENT,
Constants.PAYMENT_BY_ID),
Constants.PAYMENT_BY_ID,
w_id, d_id, h_amount, c_w_id, c_d_id, c_id, now);
}
}
}
Regards,
Iggy Fernandez
Yes
jhugg
Jun 5, 2010
The Payment transaction is also multi-partition. It appears that it was broken into two independently committed transactions for the purposes of the benchmark: the first transaction handles the updates to the WAREHOUSE and HISTORY tables ...
Iggy Fernandez


I can confirm this.
Thanks for the clarifications
iggy_fernandez
Jun 5, 2010
I can confirm this.


Thanks for the clarifications.


Regards,


Iggy Fernandez
K-Safety
iggy_fernandez
Jun 5, 2010
What value was used for K-Safety in the benchmark?


Regards,


Iggy Fernandez
No replication was used.
jhugg
Jun 5, 2010
What value was used for K-Safety in the benchmark?

Regards,

Iggy Fernandez


If you run with 2-copies (k=1), then it runs half as fast on the same number of machines. It still scales at the same rate as you add machines. Feel free to run it yourself. Let me know if you need help.
Running benchmark code
tuancao
Jun 26, 2010
Hi,


Can you tell me a bit how I can build and run the benchmark? (The first step for me to modify the benchmark code)


Thanks,
Tuan
re: Running benchmark code
bheath
Jun 26, 2010
Hi,

Can you tell me a bit how I can build and run the benchmark? (The first step for me to modify the benchmark code)

Thanks,
Tuan


Hi Tuan,
I see that the README only describes tpc-c and has no instructions. We will get you some instructions on Monday.


Can you share how you plan to modify the benchmark?


Regards,
Bobbi
re: Running benchmark code
tcallaghan
Jun 28, 2010
Hi,

Can you tell me a bit how I can build and run the benchmark? (The first step for me to modify the benchmark code)

Thanks,
Tuan

Tuan,
Here is a list of steps to run the benchmark, let me know if you need additional clarification. We plan on making this part of our more formal documentation set down the road.

1. Setup password-less SSH

- On each machine you plan to run the benchmark on, you need to add your account's public key to the file \~/.ssh/authorized_keys and/or \~/.ssh/authorized_keys2. The name of the file you need to add depends on the ssh version you're using, but to be safe, you can add the key to both. You may also need to change permissions on these files using "chmod 600 authorized_keys".
- Check that you can ssh to the machines without a password.
- If the machines use a shared home directory, you probably only need to add/change these files in one place.


2. Build/install VoltDB for benchmarking

- To execute our benchmarking infrastructure, you need to build VoltDB from source using the "voltbin" ant target.
- Get a current copy of VoltDB source using "svn checkout http://svnmirror.voltdb.com/eng/trunk/"
- In the root directory of VoltDB source ("cd trunk" if you just checked out source), execute "ant voltbin" to create "obj/release/voltbin" with the required components.
- In your home folder on all machines you plan to run the benchmark on (server or clients) as well as the machine you are initiating the benchmark from, create a folder named "voltbin" and copy all the files from obj/release/voltbin


3. Modify the "tpcc" target in build.xml

- Comments are inline, save this as a file named build.xml

<?xml version="1.0" ?>
<project default="default" name="VoltDB">

<!-- make environment var foo available as env.foo -->
<property environment="env"/>

<path id='project.classpath'>
<pathelement location='${env.HOME}/voltbin/voltdbfat.jar'/>
<pathelement path="${java.class.path}"/>
</path>

<target name="tpcc" description="Call the standard benchmark target configured to use test cluster.">
<antcall target="benchmark" inheritAll='true'>
<!-- Number of servers in your cluster -->
<param name="hostcount" value="1"/>
<!-- Number of VoltDB partitions per server -->
<param name="sitesperhost" value="12"/>
<!-- Number of warehouses for the benchmark, set equal to the number of unique partitions, as in ((hostcount * sitesperhost) / (kfactor + 1)) -->
<param name="warehouses" value="12"/>
<!-- Set less than number of warehouses, used to load warehouses in parallel (12 or less) -->
<param name="loadthreads" value="6" />
<!-- k-safety value for the benchmark -->
<param name="kfactor" value="0"/>
<!-- number of client machines for benchmark -->
<param name="clientcount" value="1"/>
<!-- number of client processes per client machine (usually left at 1) -->
<param name="processesperclient" value="1"/>
<!-- duration, in milliseconds, for benchmark run (60,000 = 1 minute) -->
<param name="duration" value="60000" />
<!-- list of all server hostnames for benchmark -->
<param name="host1" value="volt3l"/>
<param name="host2" value="volt3b"/>
<param name="host3" value="volt3c"/>
<param name="host4" value="volt3d"/>
<param name="host5" value="volt3e"/>
<param name="host6" value="volt3f"/>
<param name="host7" value="volt3g"/>
<param name="host8" value="volt3h"/>
<param name="host9" value="volt3i"/>
<param name="host10" value="volt3j"/>
<param name="host11" value="volt3k"/>
<param name="host12" value="volt3l"/>
<!-- list of all client hostnames for benchmark -->
<param name="clienthost1" value="localhost"/>
<param name="clienthost2" value="volt4b"/>
<param name="clienthost3" value="volt4c"/>
<!-- JVM heap size for servers -->
<param name="volt.server.memory" value="2048"/>
<!-- JVM heap size for clients -->
<param name="volt.client.memory" value="1024"/>
</antcall>
</target>

<target name='benchmark'
description="Compile VoltDB and run a benchmark">
<java fork="true" failonerror="true" classname="org.voltdb.benchmark.BenchmarkController" >
<classpath refid='project.classpath' />
<jvmarg value="-Xmx512m" />
<assertions><disable/></assertions>
<arg value="CLIENT=${client}" />
<arg value="BACKEND=${backend}" />
<arg value="HOSTCOUNT=${hostcount}" />
<arg value="SITESPERHOST=${sitesperhost}"/>
<arg value="KFACTOR=${kfactor}"/>
<arg value="CLIENTCOUNT=${clientcount}" />
<arg value="PROCESSESPERCLIENT=${processesperclient}" />
<arg value="INTERVAL=${interval}" />
<arg value="DURATION=${duration}" />
<arg value="REMOTEPATH=${remotepath}" />
<arg value="REMOTEUSER=${remoteuser}"/>
<arg value="HOST=${host1}" />
<arg value="HOST=${host2}" />
<arg value="HOST=${host3}" />
<arg value="HOST=${host4}" />
<arg value="HOST=${host5}" />
<arg value="HOST=${host6}" />
<arg value="HOST=${host7}" />
<arg value="HOST=${host8}" />
<arg value="HOST=${host9}" />
<arg value="HOST=${host10}" />
<arg value="HOST=${host11}" />
<arg value="HOST=${host12}" />
<arg value="CLIENTHOST=${clienthost1}" />
<arg value="CLIENTHOST=${clienthost2}" />
<arg value="CLIENTHOST=${clienthost3}" />
<arg value="LISTENFORDEBUGGER=${debug}" />
<arg value="USEPROFILE=${useprofile}" />
<arg value="CHECKTRANSACTION=${checktransaction}" />
<arg value="CLIENTHEAP=${volt.client.memory}" />
<arg value="SERVERHEAP=${volt.server.memory}" />
<arg value="LOCAL=${local}" />
<!-- Actual TXN rate sent to cluster will be:
TXNRATE * CLIENTCOUNT * PROCESSESPERCLIENT -->
<arg value="TXNRATE=${txnrate}" />
<!-- Actual number of connections opened to cluster will be:
NUMCONNECTIONS * CLIENTCOUNT * PROCESSESPERCLIENT * HOSTCOUNT -->
<arg value="NUMCONNECTIONS=${numconnections}" />

<!-- tpcc parameters -->
<arg value="warehouses=${warehouses}" />
<arg value="scalefactor=${sf}" />
<arg value="skewfactor=${skew}" />
<arg value="loadthreads=${loadthreads}" />

<!-- automatic online snapshot parameters -->
<arg value="SNAPSHOTPATH=${snapshotPath}" />
<arg value="SNAPSHOTFREQUENCY=${snapshotFrequency}" />
<arg value="SNAPSHOTPREFIX=${snapshotPrefix}" />
<arg value="SNAPSHOTRETAIN=${snapshotRetain}" />
</java>
</target>

</project>

4. Run the benchmark

- On the machine initiating the benchmark in the directory containing build.xml, run "ant tpcc".
memory requirement
tuancao
Jul 15, 2010
Tuan,
Here is a list of steps to run the benchmark, let me know if you need additional clarification. We plan on making this part of our more formal documentation set down the road.

1. Setup password-less SSH
...

- On the machine initiating the benchmark in the directory containing build.xml, run "ant tpcc".


Tim,
Thank you very much. The instruction is very helpful. I have just tried to run it on my local machine. ( used only host1 and set it to localhost, removed all the information about remote nodes, ran with 6 warehouses,...). It seemed to use all of my memory, there were a lot of swapping. Then it gave the following error:

[java] (23094069)localhost: 233681 [PeriodicWork] WARN NETWORK - Closing connection to org.voltdb.network.VoltPort@66e64686:localhost/127.0.0.1:58171 at 1279252122248 because it refuses to read responses
[java] In the past 10000 ms:
[java] Completed 0 txns at a rate of 0.00 txns/s
[java] Since the benchmark began:
[java] Completed 0 txns at a rate of 0.00 txns/s
By the way, I have copied my build.xml as below:

<<?xml version="1.0" ?>

<\code>

Also, do you know how much memory is required to store all the TPCC database in memory? If I remember correctly from one of the hstore papers, they stated that 100MB per warehouse. So if we run with 12 warehouses, then we need only about 1.2GB of memory. My machine has 4GB of memory. I don't know why all 4GB was fully used.

Thanks,
Tuan
re: memory requirement
tcallaghan
Jul 16, 2010
Tim,
Thank you very much. The instruction is very helpful. I have just tried to run it on my local machine. ( used only host1 and set it to localhost, removed all the information about remote nodes, ran with 6 warehouses,...). It seemed to use all of my memory, there were a lot of swapping. ...
Tuan


Tuan,

You can make a few changes to the build.xml file to make more memory available to the VoltDB server for data/index. The following 2 lines are important:

<!-- JVM heap size for servers -->
<param name="volt.server.memory" value="2048"/>
<!-- JVM heap size for clients -->
<param name="volt.client.memory" value="1024"/>

"volt.server.memory" is the JVM heap size for your VoltDB server, and is currently using 2GB (the value is in MB). Since you are running a simple benchmark you can probably reduce this to 512 and rerun without issue. You didn't mention where you were running your client from but you can safely reduce "volt.client.memory" to 512 as well for the benchmark.

Let me know if that helps.

-Tim
memory requirement
tuancao
Jul 16, 2010
Tuan,

You can make a few changes to the build.xml file to make more memory available to the VoltDB server for data/index. The following 2 lines are important:

<!-- JVM heap size for servers -->
<param name="volt.server.memory" value="2048"/>
<!-- JVM heap size for clients -->
<param name="volt.client.memory" value="1024"/>

-Tim


Hi Tim,
Thanks for the response.
I reduced volt.server.memory and volt.client.memory both to 512. This is the result:
[java] ============================== BENCHMARK RESULTS ==============================
[java] Time: 60000 ms
[java] Total transactions: 317
[java] Transactions per second: 5.28
[java] Delivery: 16 total 0.27 txn/s 16.00 txn/m
[java] New Order: 142 total 2.37 txn/s 142.00 txn/m
[java] Order Status: 18 total 0.30 txn/s 18.00 txn/m
[java] Payment: 135 total 2.25 txn/s 135.00 txn/m
[java] Reset Warehouse: 0 total 0.00 txn/s 0.00 txn/m
[java] Stock Level: 6 total 0.10 txn/s 6.00 txn/m
[java] Breakdown by client:
[java] localhost:0: 317 total 5.28 txn/s 317.00 txn/m
[java] ===============================================================================
I noticed there are still a lot of swapping happened.

One thing I would like to know is the amount of memory to store the TPCC database. Do you have any idea how big it is?

Thanks,
Tuan
re: memory requirement
tcallaghan
Jul 16, 2010
Hi Tim,
Thanks for the response.
I reduced volt.server.memory and volt.client.memory both to 512. This is the result:
[java] ============================== BENCHMARK RESULTS
Tuan


Tuan,

Can you paste your build.xml file into a forum reply plus include your client and server machine configurations (OS, RAM, CPU, etc.)?


Thanks,

Tim
build.xml
tuancao
Jul 16, 2010
Tuan,

Can you paste your build.xml file into a forum reply plus include your client and server machine configurations (OS, RAM, CPU, etc.)?

Thanks,
Tim

<?xml version="1.0" ?>
<project default="default" name="VoltDB">
<!-- make environment var foo available as env.foo -->
<property environment="env"/>
<path id='project.classpath'>
<pathelement location='${env.HOME}/voltbin/voltdbfat.jar'/>
<pathelement path="${java.class.path}"/>
</path>
<target name="tpcc" description="Call the standard benchmark target configured to use test cluster.">
<antcall target="benchmark" inheritAll='true'>
<!-- Number of servers in your cluster -->
<param name="hostcount" value="1"/>
<!-- Number of VoltDB partitions per server -->
<param name="sitesperhost" value="6"/>
<!-- Number of warehouses for the benchmark, set equal to the number of unique partitions, as in ((hostcount * sitesperhost) / (kfactor + 1)) -->
<param name="warehouses" value="6"/>
<!-- Set less than number of warehouses, used to load warehouses in parallel (12 or less) -->
<param name="loadthreads" value="3" />
<!-- k-safety value for the benchmark -->
<param name="kfactor" value="0"/>
<!-- number of client machines for benchmark -->
<param name="clientcount" value="1"/>
<!-- number of client processes per client machine (usually left at 1) -->
<param name="processesperclient" value="1"/>
<!-- duration, in milliseconds, for benchmark run (60,000 = 1 minute) -->
<param name="duration" value="60000" />
<!-- list of all server hostnames for benchmark -->
<param name="host1" value="localhost"/>
<!-- list of all client hostnames for benchmark -->
<param name="clienthost1" value="localhost"/>
<!-- JVM heap size for servers -->
<param name="volt.server.memory" value="512"/>
<!-- JVM heap size for clients -->
<param name="volt.client.memory" value="512"/>
</antcall>
</target>
<target name='benchmark'
description="Compile VoltDB and run a benchmark">
<java fork="true" failonerror="true" classname="org.voltdb.benchmark.BenchmarkController" >
<classpath refid='project.classpath' />
<jvmarg value="-Xmx512m" />
<assertions><disable/></assertions>
<arg value="CLIENT=${client}" />
<arg value="BACKEND=${backend}" />
<arg value="HOSTCOUNT=${hostcount}" />
<arg value="SITESPERHOST=${sitesperhost}"/>
<arg value="KFACTOR=${kfactor}"/>
<arg value="CLIENTCOUNT=${clientcount}" />
<arg value="PROCESSESPERCLIENT=${processesperclient}" />
<arg value="INTERVAL=${interval}" />
<arg value="DURATION=${duration}" />
<arg value="REMOTEPATH=${remotepath}" />
<arg value="REMOTEUSER=${remoteuser}"/>
<arg value="HOST=${host1}" />
<arg value="CLIENTHOST=${clienthost1}" />
<arg value="LISTENFORDEBUGGER=${debug}" />
<arg value="USEPROFILE=${useprofile}" />
<arg value="CHECKTRANSACTION=${checktransaction}" />
<arg value="CLIENTHEAP=${volt.client.memory}" />
<arg value="SERVERHEAP=${volt.server.memory}" />
<arg value="LOCAL=${local}" />
<!-- Actual TXN rate sent to cluster will be:
TXNRATE * CLIENTCOUNT * PROCESSESPERCLIENT -->
<arg value="TXNRATE=${txnrate}" />
<!-- Actual number of connections opened to cluster will be:
NUMCONNECTIONS * CLIENTCOUNT * PROCESSESPERCLIENT * HOSTCOUNT -->
<arg value="NUMCONNECTIONS=${numconnections}" />
<!-- tpcc parameters -->
<arg value="warehouses=${warehouses}" />
<arg value="scalefactor=${sf}" />
<arg value="skewfactor=${skew}" />
<arg value="loadthreads=${loadthreads}" />
<!-- automatic online snapshot parameters -->
<arg value="SNAPSHOTPATH=${snapshotPath}" />
<arg value="SNAPSHOTFREQUENCY=${snapshotFrequency}" />
<arg value="SNAPSHOTPREFIX=${snapshotPrefix}" />
<arg value="SNAPSHOTRETAIN=${snapshotRetain}" />
</java>
</target>
</project>
Os, ram, cpu
tuancao
Jul 16, 2010
$ uname -a
Linux hanoi 2.6.32-23-generic #37-Ubuntu SMP Fri Jun 11 08:03:28 UTC 2010 x86_64 GNU/Linux
$ free -m
total used free shared buffers cached
Mem: 3961 1732 2228 0 9 143
-/+ buffers/cache: 1579 2381
Swap: 8095 903 7191

$ head -10 /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 23
model name : Intel(R) Core(TM)2 Quad CPU Q9650 @ 3.00GHz
stepping : 10
cpu MHz : 2004.000
cache size : 6144 KB
physical id : 0
siblings : 4
Display types of transactions
tuancao
Jul 18, 2010
Hi,

I would be great if the output also provides the number of transactions for each transaction type.
Thanks,

Tuan
re: number of transactions for each transaction type
tcallaghan
Jul 19, 2010
Hi,

I would be great if the output also provides the number of transactions for each transaction type.

Thanks,
Tuan


Tuan,
If you look at the output from the benchmark you'll see the following:

[java] At time 60000 out of 60000 (100%):
[java] In the past 10000 ms:
[java] Completed 572811 txns at a rate of 57281.10 txns/s
[java] Since the benchmark began:
[java] Completed 3413447 txns at a rate of 56890.78 txns/s
[java]
[java] ============================== BENCHMARK RESULTS ==============================
[java] Time: 60000 ms
[java] Total transactions: 3413447
[java] Transactions per second: 56890.78
[java] Delivery: 136273 total 2271.22 txn/s 136273.00 txn/m
[java] New Order: 1536975 total 25616.25 txn/s 1536975.00 txn/m
[java] Order Status: 136623 total 2277.05 txn/s 136623.00 txn/m
[java] Payment: 1466581 total 24443.02 txn/s 1466581.00 txn/m
[java] Reset Warehouse: 0 total 0.00 txn/s 0.00 txn/m
[java] Stock Level: 136995 total 2283.25 txn/s 136995.00 txn/m
[java] Breakdown by client:
[java] 10.10.20.155:0: 3413447 total 56890.78 txn/s 3413447.00 txn/m
[java] ===============================================================================

So in this example I'm doing a total of 56,890.78 transactions per second. Below that it shows the total, transactions per second, and transactions per minute of each transaction type.

Is that what you are looking for?
-Tim
benchmark
tuancao
Jul 19, 2010
Tim,

Thank you very much. Can you upload your build.xml? I tried to run the benchmark on my machine but the result wasn't so good.

Tuan
re: benchmark
tcallaghan
Jul 19, 2010
Tuan,

I reviewed your information.

Given that you are running the benchmark controller, client, and server on a quad-core desktop with 4GB RAM I think your results are appropriate. Try running the benchmark controller and client from a separate machine to get better results.

-Tim
Hi Tim, Could you please give
weiwei
Feb 21, 2012
Hi Tim,


Could you please give more details instructions about how to run benchmark controller and client on different machines? I followed the instructions posted above to run TPCC and it works fine when localMood = true, but when I change localMood=false, and let host run on a remote machine, the controller is waiting for server to be ready forever. The last info I can see is:
[java] localhostINFO: Maximum clock/network skew is 0 milliseconds (according to leader)


Could you help me figure this out? Thanks a lot!
Running TPC-C
rbetts
Feb 23, 2012
We have stopped developing/supporting/using the benchmark controller framework we used for these initial benchmarks. We now run our internal benchmarking using some python scripting and the REST interface that is part of VEM.

You can find the TPC-C implementation (driver, schema, project files..) here: https://github.com/VoltDB/voltdb/tree/master/tests/test_apps/tpcc



Ryan.
Thank you very much, Ryan! I
weiwei
Feb 23, 2012
We have stopped developing/supporting/using the benchmark controller framework we used for these initial benchmarks. We now run our internal benchmarking using some python scripting and the REST interface that is part of VEM.

You can find the TPC-C implementation (driver, schema, project files..) here: https://github.com/VoltDB/voltdb/tree/master/tests/test_apps/tpcc

Ryan.


Thank you very much, Ryan! I will check it out!
How to Download the code
billmanson
Aug 12, 2012
could u tell me how to get the code.
i don't have username/passwd, the account of community didn't work for the svn login
The code is available on
rbetts
Aug 13, 2012
could u tell me how to get the code.
i don't have username/passwd, the account of community didn't work for the svn login


The code is available on github: https://github.com/VoltDB/voltdb/tree/master/tests/test_apps/tpcc
Ryan.
Nikpraskaton
May 14, 2013
Hy,
I'm new to everything :) and I have to do TPC-C benchmark on VoltDB and MySQL. So, I have downloaded VMWare version of VoltDB.
I saw on this link (https://github.com/VoltDB/voltdb/tree/master/tests/test_apps/tpcc) a lot of code :) and I would like to know what I have to do to run TPC-C benchmark on my machine.
Is there any site that explain TPC-C benchmark for beginners?

Thanks
sanket
Nov 29, 2016
Hi Bheath,
For VoltDB transactions per second number, can I get them from the Cluster Transactions (tps) graph provided in VoltDB Management Console, or is there any other method to get the numbers? I have attached the snapshot of the graph.111
sanket
Nov 29, 2016
Tuan,
If you look at the output from the benchmark you'll see the following:

[java] At time 60000 out of 60000 (100%):
[java] In the past 10000 ms:
[java] Completed 572811 txns at a rate of 57281.10 txns/s
[java] Since the benchmark began:
[java] Completed 3413447 txns at a rate of 56890.78 txns/s
[java]
[java] ============================== BENCHMARK RESULTS ==============================
[java] Time: 60000 ms
[java] Total transactions: 3413447
[java] Transactions per second: 56890.78
[java] Delivery: 136273 total 2271.22 txn/s 136273.00 txn/m
[java] New Order: 1536975 total 25616.25 txn/s 1536975.00 txn/m
[java] Order Status: 136623 total 2277.05 txn/s 136623.00 txn/m
[java] Payment: 1466581 total 24443.02 txn/s 1466581.00 txn/m
[java] Reset Warehouse: 0 total 0.00 txn/s 0.00 txn/m
[java] Stock Level: 136995 total 2283.25 txn/s 136995.00 txn/m
[java] Breakdown by client:
[java] 10.10.20.155:0: 3413447 total 56890.78 txn/s 3413447.00 txn/m
[java] ===============================================================================

So in this example I'm doing a total of 56,890.78 transactions per second. Below that it shows the total, transactions per second, and transactions per minute of each transaction type.

Is that what you are looking for?
-Tim


Hi Tim,
How did you measure the txn/sec. Did you measure it from VoltDB Management Console or you do it manually? Also how is this breakup of txn/sec for each type of transaction obtained? If you do it through some java code, can you plz give me the link?
bballard
Nov 29, 2016
Hi sanket,

The benchmark is a java application that measures the txn/sec and other metrics using the ClientStats and ClientStatsContext classes that are part of the java client. The Javadoc for these is here: https://docs.voltdb.com/javadoc/java-client-api/

For examples of using ClientStats and ClientStatsContext, you can look at any of our apps in the examples folder included in the kit.

Another example I can point out is a simple benchmark template (https://github.com/benjaminballard/app-template/tree/master/client/src/benchmark) that has a helper class BenchmarkStats that handles the boilerplate of collecting and reporting these stats. The Benchmark class has an example of how this is used:

BenchmarkStats stats = new BenchmarkStats(client);
stats.startBenchmark();
... //do some work
stats.endBenchmark();

For other clients, and in general, it is easier to capture txn/sec by calling "exec @Statistics PROCEDUREPROFILE 0;" periodically and calculating the differences. This is the same source of data that is used by the VMC graphs. We recently (v6.8) added a script under the tools folder called "watch_performance.py" that periodically calls @Statistics and outputs the tps and other metrics for each procedure executed within the interval. This can be redirected to a log file to capture periodic performance metrics over the course of some long-running workload.

-Ben