Forum: Managing VoltDB

Post: Large catalog

Large catalog
David Best
Apr 7, 2012
Our catalog.jar is up to 2.2 MB, so @UpdateApplicationCatalog fails. What to do?

voltdb-2.1.1
Hi David
rbetts
Apr 8, 2012
David,

I don't have an immediate answer for this -- needs a little research. Is this something that needs to be addressed urgently?

Ryan.
Yes, we are going into alpha
David Best
Apr 8, 2012
David,

I don't have an immediate answer for this -- needs a little research. Is this something that needs to be addressed urgently?

Ryan.



Yes, we are going into alpha test next week. Up to now we
have been reinitializing the database between tests. A big part of
alpha testing is learning to manage high availability.
Ticket created
nshi
Apr 10, 2012
Yes, we are going into alpha test next week. Up to now we
have been reinitializing the database between tests. A big part of
alpha testing is learning to manage high availability.





Hi David,

I have created a ticket for this https://issues.voltdb.com/browse/ENG-2757. We will evaluate the ticket and plan a fix for it accordingly.
Thanks, Any suggestions for a
David Best
Apr 10, 2012
Hi David,

I have created a ticket for this https://issues.voltdb.com/browse/ENG-2757. We will evaluate the ticket and plan a fix for it accordingly.


Thanks,

Any suggestions for a quick work-around until then? Could we recompile with the size limit bumped up?

David
Increasing the limit
nshi
Apr 10, 2012
Thanks,

Any suggestions for a quick work-around until then? Could we recompile with the size limit bumped up?

David

Hi David,

You can try to increase the limit here https://github.com/VoltDB/voltdb/blob/master/src/frontend/org/voltdb/VoltType.java#L129 to see if it works. Please be aware that this is a global limit on string and varbinary types, it also affects intra-cluster communication. We haven't looked into the actual fix for this yet, but it's worth a try as a quick workaround.
Ning
What is the exact failure? I
aweisberg
Apr 10, 2012
What is the exact failure? I am wondering what is enforcing the 2 megabyte limit since it isn't the Java parameter set. Maybe it is NValue?
Hi; I'm part of the project
chungy
Apr 11, 2012
What is the exact failure? I am wondering what is enforcing the 2 megabyte limit since it isn't the Java parameter set. Maybe it is NValue?



Hi; I'm part of the project being worked on here with the large catalog. The exact response back from the server is:

Status: -2
Information: Exception while deserializing procedure params
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.io.IOException: Serializable varbinary values cannot be longer then 1048576 bytes
at org.voltdb.StoredProcedureInvocation.getParams(StoredProcedureInvocation.java:101)
at org.voltdb.ClientInterface.handleRead(ClientInterface.java:1068)
at org.voltdb.ClientInterface.access$1200(ClientInterface.java:85)
at org.voltdb.ClientInterface$ClientInputHandler.handleMessage(ClientInterface.java:655)
at org.voltdb.network.VoltPort.call(VoltPort.java:208)
at org.voltdb.network.VoltNetwork$4.run(VoltNetwork.java:598)
at org.voltdb.network.VoltNetwork.invokeCallbacks(VoltNetwork.java:632)
at org.voltdb.network.VoltNetwork.run(VoltNetwork.java:451)
at java.lang.Thread.run(Unknown Source)
Caused by: java.util.concurrent.ExecutionException: java.io.IOException: Serializable varbinary values cannot be longer then 1048576 bytes
at java.util.concurrent.FutureTask$Sync.innerGet(Unknown Source)
at java.util.concurrent.FutureTask.get(Unknown Source)
at org.voltdb.StoredProcedureInvocation.getParams(StoredProcedureInvocation.java:96)
... 8 more
Caused by: java.io.IOException: Serializable varbinary values cannot be longer then 1048576 bytes
at org.voltdb.messaging.FastDeserializer.readVarbinary(FastDeserializer.java:246)
at org.voltdb.ParameterSet.readOneParameter(ParameterSet.java:404)
at org.voltdb.ParameterSet.readExternal(ParameterSet.java:94)
at org.voltdb.messaging.FastDeserializer.readObject(FastDeserializer.java:114)
at org.voltdb.StoredProcedureInvocation$2.call(StoredProcedureInvocation.java:136)
at org.voltdb.StoredProcedureInvocation$2.call(StoredProcedureInvocation.java:132)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at org.voltdb.StoredProcedureInvocation.getParams(StoredProcedureInvocation.java:94)
... 8 more
I got the response back from the following python script, using the voltdb-client-python-2.5 module. I have tried a Java version as well using the updateApplicationCatalog method with no change in behavior.

from voltdbclient import *

if __name__ == '__main__':
with open('catalog.jar', 'rb') as f:
catalog = f.read()
f.close()
with open('deployment.xml', 'rb') as f:
deployment = f.read()
f.close()
del f

client = FastSerializer('voltdb-prod1', 21212, 'admin', 'testpasswd')
proc = VoltProcedure(
client,
'@UpdateApplicationCatalog',
[FastSerializer.VOLTTYPE_VARBINARY, FastSerializer.VOLTTYPE_STRING])
response = proc.call([catalog, deployment])
client.close()

print response
Hi, So Ning is right, if you
aweisberg
Apr 11, 2012
Hi; I'm part of the project being worked on here with the large catalog. The exact response back from the server is:




Hi,

So Ning is right, if you change VoltType.MAX_VALUE_LENGTH it will get you past that piece of validation.

-Ariel
voltdb-2.7
David Best
Jun 22, 2012
As of version 2.7 we need to specify -Djute.maxbuffer=3000000 when starting the server so that zookeeper has enough buffer space for the catalog.