Mar 11, 2013
We don't put much stock in their analysis or testing. Another title for the paper might be, "A performance comparison of several naively-configured systems running naively-configured YCSB".
The numbers they get are much lower than we (VoltDB Engineering) would expect per-node, and they also don't scale as we expect. Yes, if you take a generic application designed for a different system and make the smallest effort possible to port it to VoltDB, performance is likely going to be sub-optimal. This seems like what the authors did, and we would argue that it makes for a pretty useless paper. YCSB, the benchmark in question, was written for NoSQL systems like Cassandra. Many of the performance results in the paper correlate strongly with how good a fit YCSB is for the system in question in an out-of-the-box configuration. The Redis community seems to have a similar criticism of their methods.
We believe the authors didn't read our performance tuning guide, which is freely available on our website, and probably only skimmed parts of our user guide. They certainly didn't reach out to us to ask if they were running VoltDB optimally. We believe with some small tweaks it's likely they would see very different performance characteristics, not just from VoltDB, but from many systems they tested. One could argue (as several have) that you shouldn't have to read a performance guide to get great performance out of a system. I get this, and we're constantly trying to make VoltDB perform better on a wider set of workloads, configurations and settings. Still, all of the systems in this paper require a basic understanding of how to tune an application to get great performance. Furthermore, if you're going to publish a survey benchmarking paper implying peak performance, you have some responsibility to show you've really gotten peak performance from each system. Survey papers like this are invariably more useful if they back up their findings with code. They describe their methods, but without code, we're left to fill in a lot of blanks.
In conclusion, we encourage users to try out VoltDB and see for themselves how fast it is. When you do, we'd love to hear from you too. Two-way communication with our users is how we make the users happier and the product ever-better.
Mar 11, 2013
Thanks, I agree with you.