Forum: Building VoltDB Clients

Post: python voltdb client

python voltdb client
reidmeonce
Jul 18, 2011
Hi, download "voltdb-client-python-1.3.4.tar.gz" and found it use synchronous voltdb procedure call, is asynchronous voltdb call available in Python library ?
Just like ClientKV.java example
Python library is synchronous
nshi
Jul 18, 2011
reidmeonce,

The Python library only supports synchronous stored procedure calls now. Since Python has a global interpreter lock, multi-threading in the Python client library may not make it a ton faster. A single Python program can only take up one CPU core, no matter how many threads you have. However, you can try to spawn multiple Python processes to utilize the number of cores you have in CPU.

In one of our benchmarks using the Voter application, it achieved 4k transactions/sec with a single Python process. If your application needs orders of magnitudes higher throughput, it is recommended that you try the Java, C++, or the C# client library.

--
Ning
Found timestamp issue in Python library "voltdbclient.py"
reidmeonce
Jul 29, 2011
reidmeonce,

The Python library only supports synchronous stored procedure calls now. Since Python has a global interpreter lock, multi-threading in the Python client library may not make it a ton faster. A single Python program can only take up one CPU core, no matter how many threads you have. However, you can try to spawn multiple Python processes to utilize the number of cores you have in CPU.
--
Ning


Hi, Ning,
We are currently having some prototype that need very dynamic development so we are using python 2.7
1) Found voltdbclient.py:603 , I think and change the following
def writeDate(self, value):
if value is None:
val = self.__class__.NULL_BIGINT_INDICATOR
else:
#### seconds = int(value.strftime("%s"))
seconds = long(time.mktime(value.timetuple()))
val = seconds * 1000000 + value.microsecond

However, I still found when the value pass to VoltDB procedure with "org.voltdb.types.TimestampType" and being inserted, the local time pass from python will store in DB as gmttime?
2) We can achieve ~3.8 tps using single python process with 8 threads
Can we scale out by using multiple python processes , can we except linear scale out.
3) Python library seems only support single connection to Voltdb instead of multiple connections to multiple Voltdb in Java, C++, C#, isn't it?
timestamp is always in local timezone
nshi
Jul 29, 2011
Hi, Ning,
We are currently having some prototype that need very dynamic development so we are using python 2.7
...


Hi reidmeonce,
The Python client library should work fine on Python 2.7
1. I tested with your change to converting datetime objects into seconds, it yielded same results as datetime.strftime() with both "naive" and "aware" datetime objects.
datetime objects are always de/serialized in local timezone. The TIMESTAMP schema type is agnostic to timezones in VoltDB. (see http://community.voltdb.com/node/85).
Can you give an example that resulted in incorrect timezone? I can try to reproduce it here.
2. Do you mean ~3.8k tps?
Adding more python processes will make it faster. However, there is an issue in the Python client where it uses more memory than it should (https://issues.voltdb.com/browse/ENG-1545).
3. Yes, the Python client only support single connection to the server.
--
Ning