Forum: Building VoltDB Applications

Post: Load data into Voltdb

Load data into Voltdb
dl_zhenglp
May 15, 2012
I use example_file_loader, found a problem:
When the datafile is very large, 20G example, then the Client need large memory to read the datafile?
Which file loader?
nshi
May 16, 2012
Hi,
I am not sure which file loader you used. Could you point me to the source or a download link of the file loader you used?
Thanks.
Ning
Url
dl_zhenglp
May 16, 2012
The file loader use this:



http://forum.voltdb.com/showthread.php?322-Loading-data-into-VoltDB



Thank you very much.
Whether there are other kinds of method?
dl_zhenglp
May 16, 2012
Whether there are other kinds of method?
Java max heap size
nshi
May 16, 2012
The example file loader in that forum post doesn't need that much memory to run. If you specify the max Java heap when you start the file loader to 100m or 256m, it should work without using more memory than that. To specify the max heap size, you pass "-Xmx100m" as a command line argument to java before the file loader class name.

One thing you can simplify the file loader is that VoltCompiler will now generate insert procedures for partitioned tables automatically. So if the table you are insert the data into is partitioned, you don't have to write your own insert stored procedure, you can just use the generated one, which will be named in the form "TABLENAME.insert". ChapSimplify-SimpleAutoProcs

Ning
How to define the "default stored procedures" in project.xml
dl_zhenglp
May 17, 2012
When i try to call the "default stored procedures", an error output occured:

org.voltdb.client.ProcCallException: User does not have permission to invoke KH_KHDA.select
at org.voltdb.client.ClientImpl.callProcedure(ClientImpl.java:222)
at org.voltdb.client.ClientImpl.callProcedure(ClientImpl.java:182)
at mytest1.SelectTest.main(SelectTest.java:56)
Exception in thread "main" java.lang.NullPointerException
at mytest1.SelectTest.main(SelectTest.java:66)


How to define the "default stored procedures" in project.xml?

<procedure class="mytest1.procedures.KH_KHDA.select" groups="dbuser,admin" />


And When That Fails.
Default procedures not supported with users/groups
rbetts
May 17, 2012
When i try to call the "default stored procedures", an error output occured:

And When That Fails.


Currently default procedures can not be invoked when you've enabled users and groups. This came up recently in another thread that you might find useful:
http://forum.voltdb.com/showthread.php?199-*-insert-procedures-have-no-permissions-by-default-solution&highlight=procedures+permissions

Supporting the combination of groups and default procedures is tracked by this story:

https://issues.voltdb.com/browse/ENG-3001
No need to define default procedures
nshi
May 17, 2012
When i try to call the "default stored procedures", an error output occured:

<procedure class="mytest1.procedures.KH_KHDA.select" groups="dbuser,admin" />

And When That Fails.


Besides what Ryan said about the permissions, I want to mention one more thing. You don't have to define default stored procedures in the project.xml file, they will be created for you automatically when you compile the catalog. In your case, you just need to invoke "KH_KHDA.select" in your client code.

Ning
Still have questions
dl_zhenglp
May 17, 2012
First of all thank you for your help.
When i invoke "KH_KHDA.select" in my client code, then run the client, Error persists:
Connecting to server: '10.4.56.251'
Connecting to server: '10.4.56.252'
org.voltdb.client.ProcCallException: User does not have permission to invoke KH_KHDA.select
at org.voltdb.client.ClientImpl.callProcedure(ClientImpl.java:222)
at org.voltdb.client.ClientImpl.callProcedure(ClientImpl.java:182)
at mytest1.SelectTestZ.main(SelectTestZ.java:78)
I referred to the top of the answer, But still confused.
Can you help me?

attach16
Access Permission
nshi
May 17, 2012
First of all thank you for your help.
When i invoke "KH_KHDA.select" in my client code, then run the client, Error persists:
Connecting to server: '10.4.56.251'
Connecting to server: '10.4.56.252'


It's because you have users and groups enabled in the project.xml and the deployment.xml file. As Ryan mentioned earlier, the default stored procedures currently don't work if users and groups are enabled.

You can either remove all groups and users from the project.xml file and the deployment.xml file, or you can write your own stored procedures with the correct groups to do insert/update/select/delete.
There is a ticket that addresses this issue https://issues.voltdb.com/browse/ENG-3001 .

Ning
thank you very much
dl_zhenglp
May 20, 2012
thank you very much