Forum: Managing VoltDB

Post: Profiling

Profiling
Clocks
Sep 21, 2012
How do I profile execution of VoltDb using gperftools? What are the step-by-step instructions?
Profiling
rbetts
Sep 22, 2012
First you have to understand that there are two main halves to a VoltDB process -- the top half is all Java; that includes network I/O, disk I/O and all transaction management. The bottom half is all CPP. That includes storage, SQL, snapshotting, indexing, etc. I'm assuming that you want to profile the bottom half?

There are some older build targets that help profile VoltDB's CPP component. I haven't used them in some time and don't know if they still work as expected. The main idea is that you need to link the gperf tools and then enable/disable the profiler during the code you want to profile.

https://github.com/VoltDB/voltdb/blob/master/build.xml#L996

You can also build the voltdb CPP component using an IPC interface instead of Java JNI. This will, in terms of total transactions, be much slower -- but if your goal is just to profile execution engine/sql costs, that doesn't really matter - the executed code paths in the execution engine will be essentially identical to normal execution.


https://github.com/VoltDB/voltdb/blob/master/build.xml#L977

We do not have step by step directions available. Let us know what you discover as you explore.


Thanks,
Ryan.
Profiling
Clocks
Sep 24, 2012
I read your post on Tumblr on http://rbetts.tumblr.com/page/2 regarding profiling with gperftools. I cannot find the perfctl.py file you refer to after running ant ee_profile. Was there a change to the code or I'm doing something wrong?
blog post
rbetts
Sep 25, 2012
I read your post on Tumblr on http://rbetts.tumblr.com/page/2 regarding profiling with gperftools. I cannot find the perfctl.py file you refer to after running ant ee_profile. Was there a change to the code or I'm doing something wrong?


It looks like someone removed those scripts from the test directory. Those scripts just called a system stored procedure, @ProfCtl. Looks like that still exists and is added to the built-in system catalog.

https://github.com/VoltDB/voltdb/blob/master/src/frontend/org/voltdb/sysprocs/ProfCtl.java

It looks like the pieces to turn the profiler on and off are still wired through the system. Here's the native JNI call that should results from calling ProfCtl:

https://github.com/VoltDB/voltdb/blob/master/src/ee/voltdbjni.cpp#L897

This isn't doing anything too magical - it just controls the gperf collectors. I think I used this to avoid profiling a loader - I was trying to profile a specific part of the workload.

Would love to hear what you're working on. If you are still stuck at the end of the week, I can probably make some time on Friday to get gperf setup again and offer a more concrete guide.
Cannot produce gprof.prof
yksuh
Sep 26, 2012
It looks like someone removed those scripts from the test directory. Those scripts just called a system stored procedure, @ProfCtl. Looks like that still exists and is added to the built-in system catalog.


We're wondering why /tmp/gprof.prof specified in the profiler code is not produced while running the tpcc examples.
Should we invoke the stored procedure from the tpcc java source code, to produce the gprof file?
ProfilerStart() definition
Clocks
Sep 26, 2012
Managed to add some code to see which C++ code is executed when running the benchmark. Got the following error:

libvoltdb-2.8.2.so: undefined symbol: ProfilerStart

Interestingly, the toggleProfiler() method in the VoltDBIPC class contains the following comment:
// actually, the engine doesn't implement this now.
// m_engine->ProfilerStart();

I wonder if the ProfilerStart() method is still defined anywhere in the code, because I cannot locate the definition.
From the VoltDB shared
Clocks
Sep 29, 2012
Managed to add some code to see which C++ code is executed when running the benchmark. Got the following error:

libvoltdb-2.8.2.so: undefined symbol: ProfilerStart

Interestingly, the toggleProfiler() method in the VoltDBIPC class contains the following comment:
// actually, the engine doesn't implement this now.
// m_engine->ProfilerStart();

I wonder if the ProfilerStart() method is still defined anywhere in the code, because I cannot locate the definition.


Now been able to profile the code. Only problem is that the JVM crashes whenever I run the tpcc client. Perhaps a conflict with the library package...