Key-Value Benchmark Code and Details
May 29, 2010
Benchmark Driver CodeGit Checkout
Git Source Web Browsing (Github.com)
Note: The VoltDB binaries included with this distribution are based on a slightly pre-1.0 build (1 week early). They are the exact binaries used to get our published results and I don't expect they should perform differently than the final build. They are 64-bit only and run on OSX 10.6+ and probably most Linux distributions. I'll add a 1.0.01 build soon as an option.
Node HardwareAll nodes used for this test had the following identical configuration.
- Dell R610 1-Unit Chassis
- 2x 2.66Ghz Quad-Core Xeon 5550.
- 12x 4gb (48gb) DDR-1333 Registered ECC DIMMs
- PERC6/i SAS Controller with 256mb Battery-Backed Cache
- 3x 72gb 15K RPM 2.5in Enterprise SAS 6Gbps Drives
- Broadcom Gigabit NIC (4 ports)
- Centos Linux 5.5
- Sun Java JDK/JRE 1.6u19
- Cassandra 0.6.1
- VoltDB 1.0.01 (one week pre-release)
- Dell PowerConnect 6248 48 port Gigabit Ethernet Switch
KV Test Standard Configurations
- Single node with no availability assurance. As fast as possible on one node.
- Three node cluster with no availability assurance (replication). As fast as possible on three nodes.
- Three node cluster with replication (2 copies of all data) for availability and durability.
ImplementationVoltDBServer: VoltDB was set to 8 sites per node. Intra-node and extra-node traffic used the same single gigabit ethernet link per machine. Schema was two simple tables. The table for Benchmark 1 contained a key and value pair, partitioned and indexed on the key. The table for benchmarks 2 and 3 contained a single key and 50 integer columns, partitioned and indexed on the key. The only different between "fast as possible" and replicated runs was the k-factor was set to 1 instead of 0.
Client: For Benchmark 1, we created "Load", "Get" and "Put" stored procedures, each with a single SQL statement. For Benchmarks 2 and 3, we created another "Load" stored procedure, as well a two more stored procedures to perform the operations to be measured in each benchmark. VoltDB was able to perform each operation with one network round trip to the server. The main stored procedure for Benchmark 3 performed an average of 16 SQL operations per call. Each client node created 8 threads per server node. These threads used the asynchronous VoltDB client library and performed as many operations as possible in the time allotted. The number 8 was determined from some hill-climbing pre-benchmarks to be optimal. For measurements of 3 node clusters, two VoltDB client nodes were concurrently used. The throughput of their concurrent runs was added together for the final result.
CassandraServer: The 3 disks for the server were configured to stripe 2 in RAID 0, leaving two volumes available to the OS. The data and OS was put on the RAIDed volume and the commit log was put on the non-RAID volume. The server was set up with two simple Column Families, one for Benchmark 1 and another for benchmark 2. Both Column Families used byte-wise comparison. The number of concurrent reads was increased to 16. The JVM maximum heap size was increased to 24GB. For the 3-node benchmarks, a private IP subnet on a second port was used for inter-cluster communication. For the replicated benchmarks, the Replication Factor was increased from 1 to 2. The defaults were used for all other settings.
Client: Java was used as the client language. The Thrift API was used to connect to the servers. In all benchmarks, two client nodes were used to alleviate network bandwidth issues (the VoltDB benchmark used one client node for one-node clusters). The throughput of their concurrent runs was added together for the final result. Each client node created 64 threads per server node. These threads did blocking I/O and performed as many operations as possible in the time allotted. The number 64 was determined from some hill-climbing pre-benchmarks to be optimal. The actual use of the Cassandra interface was optimized to minimize client-server messaging, using batch operations where possible. For the "fast as possible" runs, consistency was set to ONE for reads and ANY for writes. For the replicated test, both reads and writes were set to QUORUM.
MySQL BenchmarkingThis distribution includes a MySQL backend to benchmark against. The prepared statement-based implementations seems to work ok. It also has some code we wrote to use Memcached or MySQL stored procedures. We didn't have much luck running faster with either of these enhancements, but the code is included anyway. Try it out or use it as a starting point for additional enhancements.