Forum: Installation

Post: Start in background require user input to contiue?

Start in background require user input to contiue?
alexlzl
Feb 5, 2011
I am trying to start VoltDB 1.2.1.02 in background with the following command. Unfortunately it requires a user manual input of new line return to continue, otherwise the process is just stuck. It is horrible if I am trying to invoke it through ssh.


$JAVA_HOME/bin/java -server -Xmx512m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp -XX:-ReduceInitialCardMarks -Dlog4j.configuration=file:$VOLT_LIVE/config/log4j.xml -cp .:/glass/sfw/voltdb/voltdb/* -Djava.library.path=/glass/sfw/voltdb/voltdb org.voltdb.VoltDB catalog $VOLT_LIVE/config/catalog.jar deployment $VOLT_LIVE/config/deployment.xml &



INFO 2011-02-05 01:36:39,412 [main] HOST: INITIALIZING INITIATOR ID: 1, SITEID: 0
INFO 2011-02-05 01:36:39,430 [main] HOST: Starting the network
INFO 2011-02-05 01:36:39,431 [main] HOST: --------------------------------
Server completed initialization.



[process stuck here waiting for a new line return]



I couldn't find this logic in the source code, and am running out of options about how to make the process continue. Don't tell me I need to write an "expect" script to handle this case. Please help.


Thank you
alex
No user input is required
nshi
Feb 6, 2011
Hi Alex,

VoltDB does not require any user input after start-up. Normally, it would stop after the "Server completed initialization" line, which indicates that it is ready to accept client connections.

Your shell had probably given control back to you immediately after you executed the command to launch VoltDB, but the subsequent output from VoltDB was fast enough so that you did not see the shell prompt. By the time the it stopped after the line "Server completed initialization", it was actually the shell prompt waiting for more commands. This happened because stdout is not redirected to a pipe or file.

If you do not want to loose any output from VoltDB when it is running in background, you can either edit the log4j.xml configuration file that comes with the release to output to a file or you can redirect stdout and stderr to a file manually.

Hope the explanation helps.

Ning
"ssh -t" solved it
alexlzl
Feb 7, 2011
Hi Alex,

VoltDB does not require any user input after start-up. Normally, it would stop after the "Server completed initialization" line, which indicates that it is ready to accept client connections.

Your shell had probably given control back to you immediately after you executed the command to launch VoltDB, but the subsequent output from VoltDB was fast enough so that you did not see the shell prompt. By the time the it stopped after the line "Server completed initialization", it was actually the shell prompt waiting for more commands. This happened because stdout is not redirected to a pipe or file.

If you do not want to loose any output from VoltDB when it is running in background, you can either edit the log4j.xml configuration file that comes with the release to output to a file or you can redirect stdout and stderr to a file manually.

Hope the explanation helps.

Ning


Turns out the issue was related to ssh tty (I was trying to control voltdb through remote ssh). I had to pass a "-t" to force pseudo-tty allocation (i.e. ssh -t user@host "start-voltdb.sh"), otherwise the ssh session will simply hang there. Not sure why VoltDB requires a tty though, anyway this is a workaround. Thank you!
Daemonization
nshi
Feb 10, 2011
Turns out the issue was related to ssh tty (I was trying to control voltdb through remote ssh). I had to pass a "-t" to force pseudo-tty allocation (i.e. ssh -t user@host "start-voltdb.sh"), otherwise the ssh session will simply hang there. Not sure why VoltDB requires a tty though, anyway this is a workaround. Thank you!


Hi Alex,

VoltDB sends outputs and error messages to standard out and standard error by default. If you start the VoltDB process and kills the parent process, it will also kill VoltDB. There are different tools for daemonizing processes. For example, you can probably use 'nohup' to daemonize VoltDB process. It will redirect outputs and errors to files. This way you don't need to create pseudo-tty for the process.

Ning
Thank you for the tip, I just
alexlzl
Feb 10, 2011
Hi Alex,

VoltDB sends outputs and error messages to standard out and standard error by default. If you start the VoltDB process and kills the parent process, it will also kill VoltDB. There are different tools for daemonizing processes. For example, you can probably use 'nohup' to daemonize VoltDB process. It will redirect outputs and errors to files. This way you don't need to create pseudo-tty for the process.

Ning


Thank you for the tip, I just tried with nohup. Unfortunately the remote startup script still got stuck if I don't use pseudo-tty.

Here is the startup script:

#!/bin/bash
nohup $JAVA_HOME/bin/java -server -Xmx512m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp -XX:-ReduceInitialCardMarks \
-Dlog4j.configuration=file:$VOLT_LIVE/config/log4j.xml \
-cp .:/glass/sfw/voltdb/voltdb/* -Djava.library.path=/glass/sfw/voltdb/voltdb org.voltdb.VoltDB \
catalog $VOLT_LIVE/config/catalog.jar \
deployment $VOLT_LIVE/config/deployment.xml \
2>&1 >> $VOLT_HOME/log/voltdb-startup.log < /dev/null &
echo $! > $PID_FILE
You want &>> - not 2>&1 >>
sebc
Mar 8, 2011
Thank you for the tip, I just tried with nohup. Unfortunately the remote startup script still got stuck if I don't use pseudo-tty.

Here is the startup script:

#!/bin/bash
nohup $JAVA_HOME/bin/java -server -Xmx512m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp -XX:-ReduceInitialCardMarks \
-Dlog4j.configuration=file:$VOLT_LIVE/config/log4j.xml \
-cp .:/glass/sfw/voltdb/voltdb/* -Djava.library.path=/glass/sfw/voltdb/voltdb org.voltdb.VoltDB \
catalog $VOLT_LIVE/config/catalog.jar \
deployment $VOLT_LIVE/config/deployment.xml \
2>&1 >> $VOLT_HOME/log/voltdb-startup.log < /dev/null &
echo $! > $PID_FILE



Hey Alex,
Ran into the same issue and noticed when I dropped by the blog that there didn't seem to be a definite conclusion, so I figured I'd post it here (if anything for others to read) - I'm sorry if this comes late in the game! You almost had it right, but, as Ning suggested, you need to redirect both STDERR and STDOUT, so in your specific case, the last line becomes

&>> $VOLT_HOME/log/voltdb-startup.log < /dev/null &


Though launching through ant isn't something you'd want to do in production cases (one additional java process for the ant launcher itself), for those of you who use ant in development and find themselves - like me - wanting to not have to bother with the separate server terminal window, you can run:

nohup ant server &>> server.log < /dev/null &


(this obviously assumes a build.xml organized in the same model as our samples).

Alternatively, for ant users, you can add spawn="true" on the java task, as in:

<java classname="org.voltdb.VoltDB" fork="yes" spawn="true">

However, doing this means you cannot capture the log (everything goes to /dev/null), so it remains limited to development/test usage... The long java command is your only 1-VM-only, spawn-able, with-log-capture alternative (at least that I know of).
Now, I am still not fan of the long command line (especially not something I want to do when I'm struggling to rejoin a node to my cluster!), so I played around with that.

In your case Alex, you have the following raw command to launch your app:

$JAVA_HOME/bin/java -server -Xmx512m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp -XX:-ReduceInitialCardMarks -Dlog4j.configuration=file:$VOLT_LIVE/config/log4j.xml -cp .:/glass/sfw/voltdb/voltdb/* -Djava.library.path=/glass/sfw/voltdb/voltdb org.voltdb.VoltDB catalog $VOLT_LIVE/config/catalog.jar deployment $VOLT_LIVE/config/deployment.xml

What I came up with is the following (all pretty obvious when you know your way around Linux):

Export the command parameters as an environment variable (you can do this in your .bashrc / startup scripts eventually):

export MYAPPSERVER=" -server -Xmx512m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp -XX:-ReduceInitialCardMarks -Dlog4j.configuration=file:$VOLT_LIVE/config/log4j.xml -cp .:/glass/sfw/voltdb/voltdb/* -Djava.library.path=/glass/sfw/voltdb/voltdb org.voltdb.VoltDB catalog $VOLT_LIVE/config/catalog.jar deployment $VOLT_LIVE/config/deployment.xml"

Then, in scripts or on the command line, you can run any of the following:

-- Will launch the server in regular mode
java $MYAPPSERVER


($JAVA_HOME/bin/java if you want, but if your configuration is correct you shouldn't have to be so specific)
-- Will launch the server in the background, logging all output to the specified log. Daemon will survive a logout
nohup java $MYAPPSERVER &>> $VOLT_HOME/log/voltdb-startup.log < /dev/null &


-- Will launch the server in regular mode, performing a rejoin to the specified host
java $MYAPPSERVER rejoinhost 10.10.180.132


-- Will launch the server in the background, ordering a rejoin the specified host, and logging to a separate file
nohup java $MYAPPSERVER rejoinhost 10.10.180.132 &>> $VOLT_HOME/log/voltdb-rejoin.log < /dev/null &

This is about how compact I could get things while keeping the benefits of being able to run things both interactively and as background tasks at will.

Hope this helps!

Cheers,
Seb
Thank you for the detailed
alexlzl
Mar 10, 2011
Hey Alex,
Ran into the same issue and noticed when I dropped by the blog that there didn't seem to be a definite conclusion, so I figured I'd post it here (if anything for others to read) - I'm sorry if this comes late in the game! ...


Thank you for the detailed explanation. The script itself worked quite well.

Though after changing my startup script as exactly as you described, if I invoke the script through remote ssh command, the ssh session still hangs waiting to be killed. I still have to do "ssh -t 'sudo /etc/init.d/voltdb restart'. It is not bad, just a bit annoying.
I'll look into that
sebc
Mar 11, 2011
Thank you for the detailed explanation. The script itself worked quite well.

Though after changing my startup script as exactly as you described, if I invoke the script through remote ssh command, the ssh session still hangs waiting to be killed. I still have to do "ssh -t 'sudo /etc/init.d/voltdb restart'. It is not bad, just a bit annoying.


I confess I didn't try through SSH, but there must be some solution... I'll find it and report back!
Can you clarify...?
sebc
Mar 14, 2011
I confess I didn't try through SSH, but there must be some solution... I'll find it and report back!


Hey Alex,
Sorry for the delay - I got back to some experiments today, and this works great for me:

ssh volt7a "nohup nice -n 5 java -XX:-ReduceInitialCardMarks -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp -Dvolt.rmi.agent.port=9190 -Dvolt.rmi.server.hostname=volt7a -Djava.library.path=/tmp/sebc -Dlog4j.configuration=file:/tmp/sebc/log4j.properties -Xmn1536m -Xmx2048m -cp /tmp/sebc/voltdb-1.3.trunk.jar -server org.voltdb.VoltDB port 21312 internalport 3121 catalog /tmp/sebc/catalog.jar deployment /tmp/sebc/deployment.xml &> /dev/null < /dev/null &"

No pause, no nothing.

If I create the script locally with the command:

nohup nice -n 5 java -XX:-ReduceInitialCardMarks -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp -Dvolt.rmi.agent.port=9190 -Dvolt.rmi.server.hostname=volt7a -Djava.library.path=/tmp/sebc -Dlog4j.configuration=file:/tmp/sebc/log4j.properties -Xmn1536m -Xmx2048m -cp /tmp/sebc/voltdb-1.3.trunk.jar -server org.voltdb.VoltDB port 21312 internalport 3121 catalog /tmp/sebc/catalog.jar deployment /tmp/sebc/deployment.xml &> /dev/null < /dev/null &

(to tell you the truth, I'm so new with this I even forgot to add the #!/bin/bash at the top because I just echoed the pasted command - apparently not needed).

And run this:

ssh volt7a "/tmp/sebc/run.sh"

Same thing - works flawlessly.

I have the suspicion you might have forgotten to quote the remote command maybe? That would probably confuse the shell: where is & being run: local host or remote host? If you haven't quoted the command, try it and let me know if that solves your issue.

If that's not it... Are you using password-less SSH? Any security control that might cause that issue?

Cheers,

Seb
More details
alexlzl
Mar 16, 2011
Hey Alex,
Sorry for the delay - I got back to some experiments today, and this works great for me...


Thank you, Seb. My environment involves an extra middle step: sudo and su:

- This command will hang: ssh host "sudo /etc/init.d/voltdb start" (pass "-t" fixed it)
- Content of /etc/init.d/voltdb: su nobody -s /bin/bash -c "/opt/voltdb/bin/start.sh"
- Content of /opt/voltdb/bin/start.sh (as you described earlier):

nohup $JAVA_HOME/bin/java -server -Xmx512m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp -XX:-ReduceInitialCardMarks \
-Dlog4j.configuration=file:$VOLT_LIVE/config/log4j.xml \
-cp .:/glass/sfw/voltdb/voltdb/* -Djava.library.path=/glass/sfw/voltdb/voltdb org.voltdb.VoltDB \
catalog $VOLT_LIVE/project/catalog.jar \
deployment $VOLT_LIVE/config/deployment.xml $1 & >> $VOLT_HOME/log/voltdb-startup.log < /dev/null &

echo $! > $PID_FILE
exit 0


We also use the similar pattern to start Jetty and Grizzly, none of them hangs over ssh.
Extra space...
sebc
Mar 22, 2011
Thank you, Seb. My environment involves an extra middle step...


Gets me all the time - you have:

$VOLT_LIVE/config/deployment.xml $1 & >> $VOLT_HOME/log/voltdb-startup.log < /dev/null & You want:

$VOLT_LIVE/config/deployment.xml $1 &>> $VOLT_HOME/log/voltdb-startup.log < /dev/null & (remove that extra space between & and >>)


That should fix it.
Finally worked
alexlzl
Mar 24, 2011
Gets me all the time - you have:

$VOLT_LIVE/config/deployment.xml $1 & >> $VOLT_HOME/log/voltdb-startup.log < /dev/null & You want:

$VOLT_LIVE/config/deployment.xml $1 &>> $VOLT_HOME/log/voltdb-startup.log < /dev/null & (remove that extra space between & and >>)

That should fix it.


Actually "&>>" gives a syntax error. The root cause is that I tried to do "append" and added an extra ">" to your original script, it didn't work, thus I added an extra space, which caused ssh to hang. :-)

Details can be found here: http://www.unix.com/shell-programming-scripting/24155-redirect-stdout-stderr-append-file.html

I figured in realty most people want to do "append" to log files, so here is the final version worked:

deployment $VOLT_LIVE/config/deployment.xml $1 >> $VOLT_HOME/log/voltdb-startup.log 2>&1 < /dev/null &

Thank you for following it through.
&> or > + 2>&1
sebc
Mar 25, 2011
Actually "&>>" gives a syntax error. The root cause is that I tried to do "append" and added an extra ">" to your original script, it didn't work, thus I added an extra space, which caused ssh to hang. :-)

Details can be found here: http://www.unix.com/shell-programming-scripting/24155-redirect-stdout-stderr-append-file.html

I figured in realty most people want to do "append" to log files, so here is the final version worked:

deployment $VOLT_LIVE/config/deployment.xml $1 >> $VOLT_HOME/log/voltdb-startup.log 2>&1 < /dev/null &

Thank you for following it through.


I'm a lazy typer, so I picked &> (and &>> to append) which should be equivalent to > then 2>&1. Seems to work for me (I have a couple apps for which I built a "daemon" script that lets me start/stop/rejoin my DB nodes (I'll publish it in a blog post one of these days soon, have no fear :))

In any case, glad you figured it out and things are finally working properly!