Forum: Building VoltDB Applications

Post: No performance gain when scale out

No performance gain when scale out
ccshih
Aug 22, 2012
Hi,
I am running a simple ticketing system on VMs. Each VM equips two cores and 16GB RAM.
I ran the same load test program with different cluster settings, and got the following results:
hostcount = 1, sitesperhost = 1 ==> 7000-9000 TPS
hostcount = 1, sitesperhost = 2 ==> 12000-14000 TPS
hostcount = 2, sitesperhost = 2 ==> 9500-10000 TPS
The ticketing system has only one table, which has three columns: event_no, date, and number of available tickets. Each request from clients asks a ticket specified by event_no and date.
Any hints to this problem?
Thanks.
Hi ccshih, Are you using a
aweisberg
Aug 22, 2012
Hi ccshih,
Are you using a synchronous client? How many client threads? Is the 2-node number k=0 or k=1?
The big change when going from one to two nodes is that latency increases and you need more concurrent transactions to get the same performance. Requests are being forwarded %50 of the time instead of always arriving at the correct node so that adds an extra hop which is more expensive in a virtualized environment. The global transaction ordering system is also disabled when running with one node and that allows you to get good performance at lower levels of concurrency because the system is actually driven by load and falls back to periodic heartbeats in the absence of sufficient load.
Measuring scale out going from 1-2 or 1-3 nodes is usually not a good starting point because single node systems have artificially boosted numbers. You should find this when testing other systems as well although the drop off varies. Testing say 3, 6, and 9 is a better starting point. That said the drop off is steeper then it should be and the scaling is not as good as it should be although it does get usefully faster as you add nodes all the way up 30 nodes (as far as we tested).
This is something we are working on now, there is a patch to have the the Java client library route requests to the correct node, but it is tied to the rewrite of transaction initiation that also improves the 1-n node drop off and improves latency when running synchronous clients with low concurrency.
-Ariel
Dear Arial, Thanks for your
ccshih
Aug 27, 2012
Hi ccshih,
Are you using a synchronous client? How many client threads? Is the 2-node number k=0 or k=1?
The big change when going from one to two nodes is that latency increases and you need more concurrent transactions to get the same performance. Requests are being forwarded %50 of the time instead of always arriving at the correct node so that adds an extra hop which is more expensive in a virtualized environment. The global transaction ordering system is also disabled when running with one node and that allows you to get good performance at lower levels of concurrency because the system is actually driven by load and falls back to periodic heartbeats in the absence of sufficient load.
Measuring scale out going from 1-2 or 1-3 nodes is usually not a good starting point because single node systems have artificially boosted numbers. You should find this when testing other systems as well although the drop off varies. Testing say 3, 6, and 9 is a better starting point. That said the drop off is steeper then it should be and the scaling is not as good as it should be although it does get usefully faster as you add nodes all the way up 30 nodes (as far as we tested).
This is something we are working on now, there is a patch to have the the Java client library route requests to the correct node, but it is tied to the rewrite of transaction initiation that also improves the 1-n node drop off and improves latency when running synchronous clients with low concurrency.
-Ariel


Dear Arial,
Thanks for your insight. Your guess is right. I was using a synchronous client, and it was the bottleneck of the system that insufficient load was provided. After switching to async client, I can find substantial performance gain.
--ccshih