Forum: Building VoltDB Applications

Post: Overhead of partitioning

Overhead of partitioning
sanket
Nov 28, 2016
Hi,
When I was going through the documentation of partitioning, I came across the overhead involved in managing partitions. What kinds of overheads are there? Currently I am working with 5 partitions and my cpu is 24 core without hyper-threading and I am getting the best TPS as compared to other number of partitions. Can you please guide me on this issue?
bballard
Nov 28, 2016
You may want to read "How VoltDB does Transactions": https://www.voltdb.com/how-voltdb-does-transactions

Also, in the Planning Guide, there is a section on "Determining Sites Per Host": https://docs.voltdb.com/PlanningGuide/BenchmarkSites.php

Typically you should get higher performance as you increase the sitesperhost up to somewhere between 50-75% of the number of cores. However, on machines with a small number of cores, I usually find you can go higher than this. For example, with 4 cores you might still use sitesperhost=8. This all assumes a workload using mostly partitioned stored procedures.

If you are truly peaking at sitesperhost=5, then perhaps you are working with very large payloads so that the I/O is more significant than CPU, or perhaps your workload is skewed or not partitioned, or uses a lot of AdHoc SQL statements?
sanket
Nov 29, 2016
Yes, actually I am working with 10 GB of TPC-C database, but all the queries I am using are partitioned stored procedures. My workload is also partitioned but I am not sure about 'skewed workload'.