Forum: Other

Post: How do I do a "correct" tpcc benchmark?

How do I do a "correct" tpcc benchmark?
msmummy
Feb 10, 2014
Hello guys, I am trying to do a TPC-C benchmark on voltdb. However, I found in the tpcc source code that the remote option in the new order is turned off. So all new order transactions will include only one partition. I am skeptical whether it fits the TPCC specification... Anyway, I tried to turn it on, but the benchmark wouldn't execute normally as I hope... It seems there is a bug there..

The source code I am using is from github, the most recent version (4.1)..

Can anybody give me any suggestions?? thanks ....
msmummy
Feb 10, 2014
By the way, the source won't compile on my desktop (Ubuntu 13.10), because implicit conversion from size_t to int in the indexkey.h

I had to remove the "-Werror" in the build.py
msmummy
Feb 11, 2014
and also, when i read the source code of payment transaction, I found if the payment is across partition, the benchmark just execute it as two independent sub-transactions... I believe this violates serializability....
jhugg
Feb 11, 2014
Your observations are correct. Our TPC-C implementation is in no way official and takes a few shortcuts. We don't do NEW_ORDERs across warehouses and we split payments as described. There's some more info and discussion here: https://forum.voltdb.com/showthread.php?8-VoltDB-tpc-c-like-Benchmark-Comparison-Benchmark-Description

Our provided TPC-C implementation is not something we talk up much because of these compromises, but we do use it internally to measure performance of a non-trivial workload. It does do millions of SQL statements per second on pretty modest hardware.

There are plenty of reasons why it would be nearly impossible to get on the TPC-C charts with VoltDB (mostly the wait-time requirements), but someday we may try to get to a more DBT2-style workload. We have some cross-partition transaction improvements in the pipeline and I suspect these efforts might make sense around then. In the meantime, feel free to give it a shot.

As for Ubuntu 13.10, we're aware of the issue and hope to have it fixed shortly.
msmummy
Feb 12, 2014
Thank you reply, It solves my problem...

but we ran into new problems, even with this tpcc benchmark that all transaction run a on single partition, when we are trying to test it on our cluster, many of the clients won't finish...

we have four server machines (32 core each) and 4 client machines, each machine run 16 sites, each client machine run 32 clients. In our tests, many clients won't proceed normally. After it prints the "start benchmark", it goes to sleep and never wake up.

We test it like 20 times, and only got a result (still some sleeping) for 1~2 times.

When we reduce the client numbers to like 8, the problem is gone.

Can you give me some suggestions?
jhugg
Feb 12, 2014
Since the VoltDB clients are asynchronous, running 32 on a machine is unlikely to provide much benefit over 8 (or fewer). What kind of hardware are the client machines?
msmummy
Feb 12, 2014
the client machines are the same, with 32 cores each.

ok, I see... So I guess we won't need too many clients. I was worried we couldn't fill every site... I thought the warehouse id is fixed once a client starts up. I checked again and find it is generated every time for a new transaction, so it is safe...

thank you very much..
challpagal
Jan 31, 2015
we have four server machines (32 core each) and 4 client machines, each machine run 16 sites, each client machine run 32 clients. In our tests, many clients won't proceed normally. After it prints the "start benchmark", it goes to sleep and never wake up.???
pzhao
Jan 31, 2015
Challagal,

My name is Peter from VoltDB. I'd love to help you out with your setup.

Suggestions/questions:
What version of voltdb is being run?
Try simpler setup to just prove that,voltdb is up and running and client app behaves as expected.
Which benchmark are you using? A voltdb benchmark or some thing your team wrote?
If you don't mind sharing your deployment and custom source code if applicable. You can share here or send it to support@voltdb.com, whichever is more comfortable for you.
You mentioned you were running a benchmark, are you running all clients at the very beginning? To find the highest tps , you should turn on one client at a time and check monitoring service, vmc/studio, that tps is increasing.