⬅ Forum: Other

Would you explain why it was necessary to omit fulfillment of orders submitted to one warehouse with items from another warehouse from the benchmark?

Regards,

Iggy Fernandez

Would you explain why it was necessary to omit fulfillment of orders submitted to one warehouse with items from another warehouse from the benchmark?

Regards,

Iggy Fernandez

It's a change to the TPC-C spec that benefits VoltDB, which is why we mentioned it prominently in our description of what we did. It's not technically necessary. Since our code partitions on warehouse id, these transactions could be done with a multi-partition transaction. However, since our multi-parition transactions are slower, we've limited our TPC-C-like benchmark to single-partition transactions for the time being.

Similarly, we used to run our TPC-C like benchmark without rolling back any transactions, even though the spec called for a small number of rollbacks. Once we supported robust rollback, we removed this caveat and now run the benchmark with rollbacks. In the future, as we further optimize our multi-partition transaction support, we would love to better meet the spec in this area.

I would like to point out that VoltDB is still performing an average of about 26 SQL statements per transaction (if memory serves), including joins, sorting and aggregation. This works out to well over a ten million SQL statements per second on our twelve node cluster.


(Note, this response was also posted to the same question in a blog post comment: http://blog.voltdb.com/key-value-benchmarking/

It's a change to the TPC-C spec that benefits VoltDB, which is why we mentioned it prominently in our description of what we did. It's not technically necessary. ..

(Note, this response was also posted to the same question in a blog post comment: http://blog.voltdb.com/key-value-benchmarking/

The Payment transaction is also multi-partition. It appears that it was broken into two independently committed transactions for the purposes of the benchmark: the first transaction handles the updates to the WAREHOUSE and HISTORY tables (partitionInfo = "WAREHOUSE.W_ID: 0") and the second transaction handles the updates to the CUSTOMER table (partitionInfo = "CUSTOMER.C_W_ID: 3").
Can you confirm that the Payment transaction was split into two independently committed transactions for the purpose of the benchmark? The relevant code segment is as follows:
http://svnmirror.voltdb.com/eng/trunk/tests/frontend/org/voltdb/benchmark/tpcc/TPCCClient.java
public void callPaymentById(short w_id, byte d_id, double h_amount,
short c_w_id, byte c_d_id, int c_id, TimestampType now) throws IOException {
if (m_blockOnBackpressure) {
if (m_scaleParams.warehouses > 1) {
final VerifyBasicCallback cb = new VerifyBasicCallback();
while(!m_voltClient.callProcedure( cb,
Constants.PAYMENT_BY_ID_W,
w_id, d_id, h_amount, c_w_id, c_d_id, c_id, now)) {
try {
m_voltClient.backpressureBarrier();
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
final VerifyBasicCallback cb2 = new VerifyBasicCallback(TPCCSimulation.Transaction.PAYMENT,
Constants.PAYMENT_BY_ID_C);
while (!m_voltClient.callProcedure( cb2,
Constants.PAYMENT_BY_ID_C,
w_id, d_id, h_amount, c_w_id, c_d_id, c_id, now)) {
try {
m_voltClient.backpressureBarrier();
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
else {
final VerifyBasicCallback cb = new VerifyBasicCallback(TPCCSimulation.Transaction.PAYMENT,
Constants.PAYMENT_BY_ID);
while (!m_voltClient.callProcedure( cb,
Constants.PAYMENT_BY_ID,
w_id, d_id, h_amount, c_w_id, c_d_id, c_id, now)) {
try {
m_voltClient.backpressureBarrier();
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
} else {
if (m_scaleParams.warehouses > 1) {
m_voltClient.callProcedure(new VerifyBasicCallback(),
Constants.PAYMENT_BY_ID_W,
w_id, d_id, h_amount, c_w_id, c_d_id, c_id, now);
m_queuedMessage = m_voltClient.callProcedure(new VerifyBasicCallback(TPCCSimulation.Transaction.PAYMENT,
Constants.PAYMENT_BY_ID_C),
Constants.PAYMENT_BY_ID_C,
w_id, d_id, h_amount, c_w_id, c_d_id, c_id, now);
}
else {
m_queuedMessage = m_voltClient.callProcedure(new VerifyBasicCallback(TPCCSimulation.Transaction.PAYMENT,
Constants.PAYMENT_BY_ID),
Constants.PAYMENT_BY_ID,
w_id, d_id, h_amount, c_w_id, c_d_id, c_id, now);
}
}
}
Regards,
Iggy Fernandez

The Payment transaction is also multi-partition. It appears that it was broken into two independently committed transactions for the purposes of the benchmark: the first transaction handles the updates to the WAREHOUSE and HISTORY tables ...
Iggy Fernandez

I can confirm this.

I can confirm this.

Thanks for the clarifications.


Regards,


Iggy Fernandez

What value was used for K-Safety in the benchmark?


Regards,


Iggy Fernandez

What value was used for K-Safety in the benchmark?

Regards,

Iggy Fernandez

If you run with 2-copies (k=1), then it runs half as fast on the same number of machines. It still scales at the same rate as you add machines. Feel free to run it yourself. Let me know if you need help.

Hi,


Can you tell me a bit how I can build and run the benchmark? (The first step for me to modify the benchmark code)


Thanks,
Tuan

Hi,

Can you tell me a bit how I can build and run the benchmark? (The first step for me to modify the benchmark code)

Thanks,
Tuan

Hi Tuan,
I see that the README only describes tpc-c and has no instructions. We will get you some instructions on Monday.


Can you share how you plan to modify the benchmark?


Regards,
Bobbi

Hi,

Can you tell me a bit how I can build and run the benchmark? (The first step for me to modify the benchmark code)

Thanks,
Tuan

Tuan,
Here is a list of steps to run the benchmark, let me know if you need additional clarification. We plan on making this part of our more formal documentation set down the road.

1. Setup password-less SSH
- On each machine you plan to run the benchmark on, you need to add your account's public key to the file \~/.ssh/authorized_keys and/or \~/.ssh/authorized_keys2. The name of the file you need to add depends on the ssh version you're using, but to be safe, you can add the key to both. You may also need to change permissions on these files using "chmod 600 authorized_keys".
- Check that you can ssh to the machines without a password.
- If the machines use a shared home directory, you probably only need to add/change these files in one place.


2. Build/install VoltDB for benchmarking
- To execute our benchmarking infrastructure, you need to build VoltDB from source using the "voltbin" ant target.
- Get a current copy of VoltDB source using "svn checkout http://svnmirror.voltdb.com/eng/trunk/"
- In the root directory of VoltDB source ("cd trunk" if you just checked out source), execute "ant voltbin" to create "obj/release/voltbin" with the required components.
- In your home folder on all machines you plan to run the benchmark on (server or clients) as well as the machine you are initiating the benchmark from, create a folder named "voltbin" and copy all the files from obj/release/voltbin


3. Modify the "tpcc" target in build.xml
- Comments are inline, save this as a file named build.xml

<?xml version="1.0" ?>
<project default="default" name="VoltDB">

<!-- make environment var foo available as env.foo -->
<property environment="env"/>

<path id='project.classpath'>
<pathelement location='${env.HOME}/voltbin/voltdbfat.jar'/>
<pathelement path="${java.class.path}"/>
</path>

<target name="tpcc" description="Call the standard benchmark target configured to use test cluster.">
<antcall target="benchmark" inheritAll='true'>
<!-- Number of servers in your cluster -->
<param name="hostcount" value="1"/>
<!-- Number of VoltDB partitions per server -->
<param name="sitesperhost" value="12"/>
<!-- Number of warehouses for the benchmark, set equal to the number of unique partitions, as in ((hostcount * sitesperhost) / (kfactor + 1)) -->
<param name="warehouses" value="12"/>
<!-- Set less than number of warehouses, used to load warehouses in parallel (12 or less) -->
<param name="loadthreads" value="6" />
<!-- k-safety value for the benchmark -->
<param name="kfactor" value="0"/>
<!-- number of client machines for benchmark -->
<param name="clientcount" value="1"/>
<!-- number of client processes per client machine (usually left at 1) -->
<param name="processesperclient" value="1"/>
<!-- duration, in milliseconds, for benchmark run (60,000 = 1 minute) -->
<param name="duration" value="60000" />
<!-- list of all server hostnames for benchmark -->
<param name="host1" value="volt3l"/>
<param name="host2" value="volt3b"/>
<param name="host3" value="volt3c"/>
<param name="host4" value="volt3d"/>
<param name="host5" value="volt3e"/>
<param name="host6" value="volt3f"/>
<param name="host7" value="volt3g"/>
<param name="host8" value="volt3h"/>
<param name="host9" value="volt3i"/>
<param name="host10" value="volt3j"/>
<param name="host11" value="volt3k"/>
<param name="host12" value="volt3l"/>
<!-- list of all client hostnames for benchmark -->
<param name="clienthost1" value="localhost"/>
<param name="clienthost2" value="volt4b"/>
<param name="clienthost3" value="volt4c"/>
<!-- JVM heap size for servers -->
<param name="volt.server.memory" value="2048"/>
<!-- JVM heap size for clients -->
<param name="volt.client.memory" value="1024"/>
</antcall>
</target>

<target name='benchmark'
description="Compile VoltDB and run a benchmark">
<java fork="true" failonerror="true" classname="org.voltdb.benchmark.BenchmarkController" >
<classpath refid='project.classpath' />
<jvmarg value="-Xmx512m" />
<assertions><disable/></assertions>
<arg value="CLIENT=${client}" />
<arg value="BACKEND=${backend}" />
<arg value="HOSTCOUNT=${hostcount}" />
<arg value="SITESPERHOST=${sitesperhost}"/>
<arg value="KFACTOR=${kfactor}"/>
<arg value="CLIENTCOUNT=${clientcount}" />
<arg value="PROCESSESPERCLIENT=${processesperclient}" />
<arg value="INTERVAL=${interval}" />
<arg value="DURATION=${duration}" />
<arg value="REMOTEPATH=${remotepath}" />
<arg value="REMOTEUSER=${remoteuser}"/>
<arg value="HOST=${host1}" />
<arg value="HOST=${host2}" />
<arg value="HOST=${host3}" />
<arg value="HOST=${host4}" />
<arg value="HOST=${host5}" />
<arg value="HOST=${host6}" />
<arg value="HOST=${host7}" />
<arg value="HOST=${host8}" />
<arg value="HOST=${host9}" />
<arg value="HOST=${host10}" />
<arg value="HOST=${host11}" />
<arg value="HOST=${host12}" />
<arg value="CLIENTHOST=${clienthost1}" />
<arg value="CLIENTHOST=${clienthost2}" />
<arg value="CLIENTHOST=${clienthost3}" />
<arg value="LISTENFORDEBUGGER=${debug}" />
<arg value="USEPROFILE=${useprofile}" />
<arg value="CHECKTRANSACTION=${checktransaction}" />
<arg value="CLIENTHEAP=${volt.client.memory}" />
<arg value="SERVERHEAP=${volt.server.memory}" />
<arg value="LOCAL=${local}" />
<!-- Actual TXN rate sent to cluster will be:
TXNRATE * CLIENTCOUNT * PROCESSESPERCLIENT -->
<arg value="TXNRATE=${txnrate}" />
<!-- Actual number of connections opened to cluster will be:
NUMCONNECTIONS * CLIENTCOUNT * PROCESSESPERCLIENT * HOSTCOUNT -->
<arg value="NUMCONNECTIONS=${numconnections}" />

<!-- tpcc parameters -->
<arg value="warehouses=${warehouses}" />
<arg value="scalefactor=${sf}" />
<arg value="skewfactor=${skew}" />
<arg value="loadthreads=${loadthreads}" />

<!-- automatic online snapshot parameters -->
<arg value="SNAPSHOTPATH=${snapshotPath}" />
<arg value="SNAPSHOTFREQUENCY=${snapshotFrequency}" />
<arg value="SNAPSHOTPREFIX=${snapshotPrefix}" />
<arg value="SNAPSHOTRETAIN=${snapshotRetain}" />
</java>
</target>

</project>

4. Run the benchmark
- On the machine initiating the benchmark in the directory containing build.xml, run "ant tpcc".

Tuan,
Here is a list of steps to run the benchmark, let me know if you need additional clarification. We plan on making this part of our more formal documentation set down the road.

1. Setup password-less SSH...

- On the machine initiating the benchmark in the directory containing build.xml, run "ant tpcc".

Tim,
Thank you very much. The instruction is very helpful. I have just tried to run it on my local machine. ( used only host1 and set it to localhost, removed all the information about remote nodes, ran with 6 warehouses,...). It seemed to use all of my memory, there were a lot of swapping. Then it gave the following error:

[java] (23094069)localhost: 233681 [PeriodicWork] WARN NETWORK - Closing connection to org.voltdb.network.VoltPort@66e64686:localhost/127.0.0.1:58171 at 1279252122248 because it refuses to read responses
[java] In the past 10000 ms:
[java] Completed 0 txns at a rate of 0.00 txns/s
[java] Since the benchmark began:
[java] Completed 0 txns at a rate of 0.00 txns/s
By the way, I have copied my build.xml as below:

<<?xml version="1.0" ?>

<\code>

Also, do you know how much memory is required to store all the TPCC database in memory? If I remember correctly from one of the hstore papers, they stated that 100MB per warehouse. So if we run with 12 warehouses, then we need only about 1.2GB of memory. My machine has 4GB of memory. I don't know why all 4GB was fully used.

Thanks,
Tuan

Tim,
Thank you very much. The instruction is very helpful. I have just tried to run it on my local machine. ( used only host1 and set it to localhost, removed all the information about remote nodes, ran with 6 warehouses,...). It seemed to use all of my memory, there were a lot of swapping. ...
Tuan

Tuan,

You can make a few changes to the build.xml file to make more memory available to the VoltDB server for data/index. The following 2 lines are important:

<!-- JVM heap size for servers -->
<param name="volt.server.memory" value="2048"/>
<!-- JVM heap size for clients -->
<param name="volt.client.memory" value="1024"/>

"volt.server.memory" is the JVM heap size for your VoltDB server, and is currently using 2GB (the value is in MB). Since you are running a simple benchmark you can probably reduce this to 512 and rerun without issue. You didn't mention where you were running your client from but you can safely reduce "volt.client.memory" to 512 as well for the benchmark.

Let me know if that helps.

-Tim

Tuan,

You can make a few changes to the build.xml file to make more memory available to the VoltDB server for data/index. The following 2 lines are important:

<!-- JVM heap size for servers -->
<param name="volt.server.memory" value="2048"/>
<!-- JVM heap size for clients -->
<param name="volt.client.memory" value="1024"/>

-Tim

Hi Tim,
Thanks for the response.
I reduced volt.server.memory and volt.client.memory both to 512. This is the result:
[java] ============================== BENCHMARK RESULTS ==============================
[java] Time: 60000 ms
[java] Total transactions: 317
[java] Transactions per second: 5.28
[java] Delivery: 16 total 0.27 txn/s 16.00 txn/m
[java] New Order: 142 total 2.37 txn/s 142.00 txn/m
[java] Order Status: 18 total 0.30 txn/s 18.00 txn/m
[java] Payment: 135 total 2.25 txn/s 135.00 txn/m
[java] Reset Warehouse: 0 total 0.00 txn/s 0.00 txn/m
[java] Stock Level: 6 total 0.10 txn/s 6.00 txn/m
[java] Breakdown by client:
[java] localhost:0: 317 total 5.28 txn/s 317.00 txn/m
[java] ===============================================================================
I noticed there are still a lot of swapping happened.

One thing I would like to know is the amount of memory to store the TPCC database. Do you have any idea how big it is?

Thanks,
Tuan

Hi Tim,
Thanks for the response.
I reduced volt.server.memory and volt.client.memory both to 512. This is the result:
[java] ============================== BENCHMARK RESULTS
Tuan

Tuan,

Can you paste your build.xml file into a forum reply plus include your client and server machine configurations (OS, RAM, CPU, etc.)?


Thanks,

Tim

Tuan,

Can you paste your build.xml file into a forum reply plus include your client and server machine configurations (OS, RAM, CPU, etc.)?

Thanks,
Tim

<?xml version="1.0" ?>
<project default="default" name="VoltDB">
<!-- make environment var foo available as env.foo -->
<property environment="env"/>
<path id='project.classpath'>
<pathelement location='${env.HOME}/voltbin/voltdbfat.jar'/>
<pathelement path="${java.class.path}"/>
</path>
<target name="tpcc" description="Call the standard benchmark target configured to use test cluster.">
<antcall target="benchmark" inheritAll='true'>
<!-- Number of servers in your cluster -->
<param name="hostcount" value="1"/>
<!-- Number of VoltDB partitions per server -->
<param name="sitesperhost" value="6"/>
<!-- Number of warehouses for the benchmark, set equal to the number of unique partitions, as in ((hostcount * sitesperhost) / (kfactor + 1)) -->
<param name="warehouses" value="6"/>
<!-- Set less than number of warehouses, used to load warehouses in parallel (12 or less) -->
<param name="loadthreads" value="3" />
<!-- k-safety value for the benchmark -->
<param name="kfactor" value="0"/>
<!-- number of client machines for benchmark -->
<param name="clientcount" value="1"/>
<!-- number of client processes per client machine (usually left at 1) -->
<param name="processesperclient" value="1"/>
<!-- duration, in milliseconds, for benchmark run (60,000 = 1 minute) -->
<param name="duration" value="60000" />
<!-- list of all server hostnames for benchmark -->
<param name="host1" value="localhost"/>
<!-- list of all client hostnames for benchmark -->
<param name="clienthost1" value="localhost"/>
<!-- JVM heap size for servers -->
<param name="volt.server.memory" value="512"/>
<!-- JVM heap size for clients -->
<param name="volt.client.memory" value="512"/>
</antcall>
</target>
<target name='benchmark'
description="Compile VoltDB and run a benchmark">
<java fork="true" failonerror="true" classname="org.voltdb.benchmark.BenchmarkController" >
<classpath refid='project.classpath' />
<jvmarg value="-Xmx512m" />
<assertions><disable/></assertions>
<arg value="CLIENT=${client}" />
<arg value="BACKEND=${backend}" />
<arg value="HOSTCOUNT=${hostcount}" />
<arg value="SITESPERHOST=${sitesperhost}"/>
<arg value="KFACTOR=${kfactor}"/>
<arg value="CLIENTCOUNT=${clientcount}" />
<arg value="PROCESSESPERCLIENT=${processesperclient}" />
<arg value="INTERVAL=${interval}" />
<arg value="DURATION=${duration}" />
<arg value="REMOTEPATH=${remotepath}" />
<arg value="REMOTEUSER=${remoteuser}"/>
<arg value="HOST=${host1}" />
<arg value="CLIENTHOST=${clienthost1}" />
<arg value="LISTENFORDEBUGGER=${debug}" />
<arg value="USEPROFILE=${useprofile}" />
<arg value="CHECKTRANSACTION=${checktransaction}" />
<arg value="CLIENTHEAP=${volt.client.memory}" />
<arg value="SERVERHEAP=${volt.server.memory}" />
<arg value="LOCAL=${local}" />
<!-- Actual TXN rate sent to cluster will be:
TXNRATE * CLIENTCOUNT * PROCESSESPERCLIENT -->
<arg value="TXNRATE=${txnrate}" />
<!-- Actual number of connections opened to cluster will be:
NUMCONNECTIONS * CLIENTCOUNT * PROCESSESPERCLIENT * HOSTCOUNT -->
<arg value="NUMCONNECTIONS=${numconnections}" />
<!-- tpcc parameters -->
<arg value="warehouses=${warehouses}" />
<arg value="scalefactor=${sf}" />
<arg value="skewfactor=${skew}" />
<arg value="loadthreads=${loadthreads}" />
<!-- automatic online snapshot parameters -->
<arg value="SNAPSHOTPATH=${snapshotPath}" />
<arg value="SNAPSHOTFREQUENCY=${snapshotFrequency}" />
<arg value="SNAPSHOTPREFIX=${snapshotPrefix}" />
<arg value="SNAPSHOTRETAIN=${snapshotRetain}" />
</java>
</target>
</project>

$ uname -a
Linux hanoi 2.6.32-23-generic #37-Ubuntu SMP Fri Jun 11 08:03:28 UTC 2010 x86_64 GNU/Linux
$ free -m
total used free shared buffers cached
Mem: 3961 1732 2228 0 9 143
-/+ buffers/cache: 1579 2381
Swap: 8095 903 7191

$ head -10 /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 23
model name : Intel(R) Core(TM)2 Quad CPU Q9650 @ 3.00GHz
stepping : 10
cpu MHz : 2004.000
cache size : 6144 KB
physical id : 0
siblings : 4

Hi,

I would be great if the output also provides the number of transactions for each transaction type.
Thanks,

Tuan

Hi,

I would be great if the output also provides the number of transactions for each transaction type.

Thanks,
Tuan

Tuan,
If you look at the output from the benchmark you'll see the following:

[java] At time 60000 out of 60000 (100%):
[java] In the past 10000 ms:
[java] Completed 572811 txns at a rate of 57281.10 txns/s
[java] Since the benchmark began:
[java] Completed 3413447 txns at a rate of 56890.78 txns/s
[java]
[java] ============================== BENCHMARK RESULTS ==============================
[java] Time: 60000 ms
[java] Total transactions: 3413447
[java] Transactions per second: 56890.78
[java] Delivery: 136273 total 2271.22 txn/s 136273.00 txn/m
[java] New Order: 1536975 total 25616.25 txn/s 1536975.00 txn/m
[java] Order Status: 136623 total 2277.05 txn/s 136623.00 txn/m
[java] Payment: 1466581 total 24443.02 txn/s 1466581.00 txn/m
[java] Reset Warehouse: 0 total 0.00 txn/s 0.00 txn/m
[java] Stock Level: 136995 total 2283.25 txn/s 136995.00 txn/m
[java] Breakdown by client:
[java] 10.10.20.155:0: 3413447 total 56890.78 txn/s 3413447.00 txn/m
[java] ===============================================================================

So in this example I'm doing a total of 56,890.78 transactions per second. Below that it shows the total, transactions per second, and transactions per minute of each transaction type.

Is that what you are looking for?
-Tim

Tim,

Thank you very much. Can you upload your build.xml? I tried to run the benchmark on my machine but the result wasn't so good.

Tuan

Tuan,

I reviewed your information.

Given that you are running the benchmark controller, client, and server on a quad-core desktop with 4GB RAM I think your results are appropriate. Try running the benchmark controller and client from a separate machine to get better results.

-Tim

Hi Tim,


Could you please give more details instructions about how to run benchmark controller and client on different machines? I followed the instructions posted above to run TPCC and it works fine when localMood = true, but when I change localMood=false, and let host run on a remote machine, the controller is waiting for server to be ready forever. The last info I can see is:
[java] localhostINFO: Maximum clock/network skew is 0 milliseconds (according to leader)


Could you help me figure this out? Thanks a lot!

We have stopped developing/supporting/using the benchmark controller framework we used for these initial benchmarks. We now run our internal benchmarking using some python scripting and the REST interface that is part of VEM.

You can find the TPC-C implementation (driver, schema, project files..) here: https://github.com/VoltDB/voltdb/tree/master/tests/test_apps/tpcc



Ryan.

We have stopped developing/supporting/using the benchmark controller framework we used for these initial benchmarks. We now run our internal benchmarking using some python scripting and the REST interface that is part of VEM.

You can find the TPC-C implementation (driver, schema, project files..) here: https://github.com/VoltDB/voltdb/tree/master/tests/test_apps/tpcc

Ryan.

Thank you very much, Ryan! I will check it out!

could u tell me how to get the code.
i don't have username/passwd, the account of community didn't work for the svn login

could u tell me how to get the code.
i don't have username/passwd, the account of community didn't work for the svn login

The code is available on github: https://github.com/VoltDB/voltdb/tree/master/tests/test_apps/tpcc
Ryan.

Hy,
I'm new to everything :) and I have to do TPC-C benchmark on VoltDB and MySQL. So, I have downloaded VMWare version of VoltDB.
I saw on this link (https://github.com/VoltDB/voltdb/tree/master/tests/test_apps/tpcc) a lot of code :) and I would like to know what I have to do to run TPC-C benchmark on my machine.
Is there any site that explain TPC-C benchmark for beginners?

Thanks

Hi Bheath,
For VoltDB transactions per second number, can I get them from the Cluster Transactions (tps) graph provided in VoltDB Management Console, or is there any other method to get the numbers? I have attached the snapshot of the graph.111

Tuan,
If you look at the output from the benchmark you'll see the following:

[java] At time 60000 out of 60000 (100%):
[java] In the past 10000 ms:
[java] Completed 572811 txns at a rate of 57281.10 txns/s
[java] Since the benchmark began:
[java] Completed 3413447 txns at a rate of 56890.78 txns/s
[java]
[java] ============================== BENCHMARK RESULTS ==============================
[java] Time: 60000 ms
[java] Total transactions: 3413447
[java] Transactions per second: 56890.78
[java] Delivery: 136273 total 2271.22 txn/s 136273.00 txn/m
[java] New Order: 1536975 total 25616.25 txn/s 1536975.00 txn/m
[java] Order Status: 136623 total 2277.05 txn/s 136623.00 txn/m
[java] Payment: 1466581 total 24443.02 txn/s 1466581.00 txn/m
[java] Reset Warehouse: 0 total 0.00 txn/s 0.00 txn/m
[java] Stock Level: 136995 total 2283.25 txn/s 136995.00 txn/m
[java] Breakdown by client:
[java] 10.10.20.155:0: 3413447 total 56890.78 txn/s 3413447.00 txn/m
[java] ===============================================================================

So in this example I'm doing a total of 56,890.78 transactions per second. Below that it shows the total, transactions per second, and transactions per minute of each transaction type.

Is that what you are looking for?
-Tim

Hi Tim,
How did you measure the txn/sec. Did you measure it from VoltDB Management Console or you do it manually? Also how is this breakup of txn/sec for each type of transaction obtained? If you do it through some java code, can you plz give me the link?

Hi sanket,

The benchmark is a java application that measures the txn/sec and other metrics using the ClientStats and ClientStatsContext classes that are part of the java client. The Javadoc for these is here: https://docs.voltdb.com/javadoc/java-client-api/

For examples of using ClientStats and ClientStatsContext, you can look at any of our apps in the examples folder included in the kit.

Another example I can point out is a simple benchmark template (https://github.com/benjaminballard/app-template/tree/master/client/src/benchmark) that has a helper class BenchmarkStats that handles the boilerplate of collecting and reporting these stats. The Benchmark class has an example of how this is used:

BenchmarkStats stats = new BenchmarkStats(client);
stats.startBenchmark();
... //do some work
stats.endBenchmark();

For other clients, and in general, it is easier to capture txn/sec by calling "exec @Statistics PROCEDUREPROFILE 0;" periodically and calculating the differences. This is the same source of data that is used by the VMC graphs. We recently (v6.8) added a script under the tools folder called "watch_performance.py" that periodically calls @Statistics and outputs the tps and other metrics for each procedure executed within the interval. This can be redirected to a log file to capture periodic performance metrics over the course of some long-running workload.

-Ben