Retrying to run query in a stored procedure callback

[gideon caller]

Hi everyone,

I'm currently working on a program that inserts a lot of rows into a partitioned table (~75K insertions per sec)
These stored procedures are invoked asynchronously using a custom callback class that I wrote.
What I'm trying to do is in the callback class if I see that a stored procedure invocation failed (inside the callback) I want it to retry invoking the same SP that failed for X amount of times. What I tried doing is pass the VoltDB client to the callback and if the client response status is different than SUCCESS invoke the same stored procedure X more times synchronously (in order to not create X more callbacks on each failing stored procedure). What happens when an asynchronous invocation occurs is that the callback starts retrying but fails when invoking the stored procedure synchronously with an IO exception saying: "Can't invoke a procedure synchronously from with the client callback thread without deadlocking the client library (which makes sense since it can wait for a timeout and block for quite a while)

And so my question is: if I want to attempt several retries (of the same SP) in the callback, whats the best way of doing that?

Thanks in advance

[jhugg]

You may want to look at our bulk loader code here: https://github.com/VoltDB/voltdb/blo...ulkLoader.java

A typical pattern to consider here is to use a thread-safe queue of writes on the client side and put things back into the queue when they fail. Or keep two queues, one for failures and one for new tuples and always pull from failure first.

Then you might want to keep an "outstanding ops" counter. Increment when you async fire and insert and decrement when it succeeds. Pick a max acceptable value and use this value for backpressure.

There are a few ways to do this stuff. You could also keep a "retries counter" in the callback object, then re-call when things fail with an incremented re-try. If you get a failure with retry > max acceptable, then you do whatever you do then.

Let us know if this helps.