Forum: Managing VoltDB

Post: Abourt poor performance

Abourt poor performance
dl_zhenglp
Apr 16, 2012
When i use 3 nodes cluster to test data inserting, i found poor performance.
If only use single node inserting(not cluster), about 50,000rows/s,but when i use 3 nodes cluster inserting, only 200rows/s。
Who can answer me why?
-------------------------------------
1. My configration here below:
【deployment.xml】(3 Node as the same)
cluster hostcount="3" sitesperhost="2" kfactor="1"
【build.xml】(3 Node as the same)
arg value="10.4.57.250" (leader ip)
Use the config as above, when i can start the 3 nodes cluster no problem.
------------------------------
2. My Client
【Client.java】
final ClientConfig clientConfig = new ClientConfig("operator", "mech");
final org.voltdb.client.Client voltclient = org.voltdb.client.ClientFactory.createClient(clientConfig);
try {
voltclient.createConnection("10.4.57.248");
voltclient.createConnection("10.4.57.249");
voltclient.createConnection("10.4.57.250");
} catch (IOException e) {
e.printStackTrace();
System.exit(-1);
}
final Client client = new Client(voltclient);
int i = 0;
while (i <= 10000000) {
client.doInsert1(i + "", "1", "1", "1","1", "0", "C", "1","19031008", null, null, null, null, "190001", "300001", 1);
i++;
}
I found latency
dl_zhenglp
Apr 16, 2012
when i use studio.web tool Performance Monitor &#65292; 3 nodes cluster inserting test, i found the latency&#65288;ms&#65289; > 3. But when single node inserting test, latency&#65288;ms&#65289; < 0.15.

I think this is a problem. Who can help me?
Attach file
dl_zhenglp
Apr 16, 2012
Attach file

attach20

attach21
Asynchronous client
nshi
Apr 16, 2012
Hi,

The performance degradation was because your synchronous client could not generate enough load for the multi-node cluster. Please refer to this comment on how to write an asynchronous client,
http://forum.voltdb.com/showthread.php?373-Performance-crushed-when-second-node-introduced&highlight=Performance+crushed+node+introduced.

Ning
there is still no single node speed
dl_zhenglp
Apr 18, 2012
Hi,

The performance degradation was because your synchronous client could not generate enough load for the multi-node cluster. Please refer to this comment on how to write an asynchronous client,




Thank you very much, Indeed four times times faster. but there is still no single node speed,Only up to 800rows per second.
I do not think it should be so slow, can you help with the analysis of reasons?
Asynchronous processing is as follows:

static class SignInCallback implements ProcedureCallback {
@Override
public void clientCallback(ClientResponse response) throws Exception {
final byte status = response.getStatus();
if (status != ClientResponse.SUCCESS) {
System.err.println("Failed to execute!!!");
System.err.println(response.getStatusString());
System.err.println(response.getException());
System.exit(-1);
} else {
// do nothing on success
}
}
}

String serverList = "voltdb1, voltdb2, voltdb3";

ClientConfig config = new ClientConfig("developer", "tech");
final org.voltdb.client.Client voltclient = ClientFactory
.createClient(config);
String[] voltServers = serverList.split(",");

for (String thisServer : voltServers) {
try {
thisServer = thisServer.trim();
System.out.printf("Connecting to server: '%s'
", thisServer);

voltclient.createConnection(thisServer);
} catch (IOException e) {
e.printStackTrace();
System.exit(-1);
}
}

SignInCallback callBack = new SignInCallback();

// Insert data
System.out.print("Inserting data ... ");
for (int i = 0; i < 100000; i++) {
try {
boolean bl = voltclient.callProcedure(callBack, "Insert1", i + "", "1",
"1", "1", "1", "1", "1", "1", "1", null, null, null,
null, "1", "1", 1);
} catch (Exception e) {
e.printStackTrace();
}
}
Console Output:
client:
[java] Client started...
[java] Connecting to server: 'voltdb1'
[java] Connecting to server: 'voltdb2'
[java] Connecting to server: 'voltdb3'
[java] Inserting data ... 2012-04-18 17:20:18,936 [Volt Network] DEBUG NETWORK - Closing channel org.voltdb.network.VoltPort@77827284:voltdb1/10.4.57.250:21212
[java] 2012-04-18 17:20:18,937 [Volt Network] DEBUG NETWORK - Closing channel org.voltdb.network.VoltPort@2cb0ce8f:voltdb2/10.4.57.249:21212
[java] 2012-04-18 17:20:18,937 [Volt Network] DEBUG NETWORK - Closing channel org.voltdb.network.VoltPort@42a9c09e:voltdb3/10.4.57.248:21212
[java] Inserting data done...
BUILD SUCCESSFUL
Total time: 2 minutes 1 second
NTP and hardware setup
nshi
Apr 18, 2012
Thank you very much, Indeed four times times faster. but there is still no single node speed,Only up to 800rows per second.
I do not think it should be so slow, can you help with the analysis of reasons?
Asynchronous processing is as follows:



Hi,

Is NTP setup properly on all the machines? You can check the clock skew by looking for the line that starts with "Maximum clock/network skew is" in the log files generated by each VoltDB server. It's best to have the clock skew under 10 milliseconds.

For how to setup NTP properly, you can follow this, see: PerfGuide

[TD="class: xl66, width: 467"]



Are the machines physical machines or virtual machines? What kind of network do you have between them?

Ning
NTP and hardware setup
dl_zhenglp
Apr 19, 2012
Hi,

Is NTP setup properly on all the machines? You can check the clock skew by looking for the line that starts with "Maximum clock/network skew is" in the log files generated by each VoltDB server. It's best to have the clock skew under 10 milliseconds.


Thank you for your help...

But still very slow...

I use three physical machines, not virtual machines.

My ntp configration is(Same as Example 5.1. Custom NTP Configuration File)&#65306;


server myntpsvr burst iburst minpoll 4 maxpoll 4

peer voltsvr1 burst iburst minpoll 4 maxpoll 4
peer voltsvr2 burst iburst minpoll 4 maxpoll 4
peer voltsvr3 burst iburst minpoll 4 maxpoll 4

server 127.127.0.1
fudge 127.127.0.1 stratum 10


[root@voltdb1 ~]# ntpq -p
remote refid st t when poll reach delay offset jitter
==============================================================================
*202.118.6.8 218.75.4.130 3 u 13 16 377 6.881 0.108 0.607
voltdb1 .INIT. 16 u - 16 0 0.000 0.000 0.000
voltdb2 202.118.6.8 4 u - 16 377 0.457 -0.294 0.129
voltdb3 202.118.6.8 4 u - 16 377 0.423 -3.707 0.056

My voltdb1.log:

[java] 2012-04-19 02:23:28,666 [Thread-3] INFO HOST - Connecting to VoltDB cluster as the leader...

[java] 2012-04-19 02:23:28,666 [Thread-3] DEBUG org.voltdb.messaging.SocketJoiner - Starting Coordinator

[java] 2012-04-19 02:23:45,496 [Thread-3] INFO HOST - Maximum clock/network skew is 1 milliseconds (according to leader)

[java] 2012-04-19 02:23:45,510 [main] INFO HOST - Heartbeat timeout to host: /10.4.57.249 is 10000 milliseconds

[java] 2012-04-19 02:23:45,510 [main] INFO HOST - Heartbeat timeout to host: /10.4.57.248 is 10000 milliseconds

[java] 2012-04-19 02:23:45,629 [main] INFO ZK-SERVER - binding to port /127.0.0.1:2181

[java] 2012-04-19 02:23:45,641 [main] INFO ZK-SERVER - Created
server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout 60000

[java] 2012-04-19 02:23:45,682 [main] INFO HOST - Starting the network subsystem with 2 worker threads.

[java] 2012-04-19 02:23:45,690 [main] INFO ZK-SERVER - Initiating
client connection, connectString=127.0.0.1:2181 sessionTimeout=2000
watcher=org.voltdb.agreement.ZKUtil$1@482d59a3

[java] 2012-04-19 02:23:45,692 [main] DEBUG ZK-CLIENT - zookeeper.disableAutoWatchReset is false

[java] 2012-04-19 02:23:45,699 [main-SendThread()] INFO ZK-CLIENT - Opening socket connection to server /127.0.0.1:2181

attach22
DDL and procedure source
nshi
Apr 19, 2012
Do you mind sharing your DDL and the source of the insert procedure? You can either post them here or email them to me at nshi at voltdb.com. I'll test it on our cluster.
DDL and procedure source
dl_zhenglp
Apr 19, 2012
Do you mind sharing your DDL and the source of the insert procedure? You can either post them here or email them to me at nshi at voltdb.com. I'll test it on our cluster.


Hi nshi&#65292; Thank you very much for your help.Want to find the problem cause.

attach23
I found the problem!
dl_zhenglp
Apr 19, 2012
Hi nshi&#65292; Thank you very much for your help.Want to find the problem cause.




The problem location is there:
&#12304;Insert1.java&#12305;

@ProcInfo(
//partitionInfo = "PUB_DEPT.DEPT_ID: 0",
singlePartition = false
)

I changed it to:
@ProcInfo(
partitionInfo = "PUB_DEPT.DEPT_ID: 0",
singlePartition = true
)

Then, speed up to 300000rows/s&#65292;good!