Run a nohup command over SSH, then disconnect

Question

I want to execute a script, start.sh on a remote server which runs this:

nohup node server.js &

Naively, I call SSH like this:

ssh myserver <<EOF
./start.sh &
EOF

This starts the script, but leaves the session connected. I want to follow this step with other commands in a script, so that's no good.

How can I SSH to the remote machine, launch a nohup command into the background, then disconnect? I suppose I could put the SSH process itself into the background, but that doesn't seem right.

Answer 1

You have already found the right way, here document.

NOTE: you can put the ssh (client) into background by placing a & at the end, but you will not see the output. If you really want to do this, redirect the stdout/stderr to a file in case you need to check the response from the remote host.

Basically you can do it in either way:

Directly run the command{,s}

ssh user@host "nohup command1 > /dev/null 2>&1 &; nohup command2; command3"

OR

ssh user@host "$(nohup command1 > /dev/null 2>&1 &) && nohup command2 >> /path/to/log 2>&1 &"

NOTE: && requires the first command to return 0 before executing the second

Use Here document

ssh user@host << EOF
nohup command1 > /dev/null 2>&1 &
nohup command2 >> /path/to/command2.log 2>&1 &
......
EOF

The above 3 options should work for you.

In addition, take a look at the answer here: https://askubuntu.com/a/348921/70270

Answer 2

ssh node "nohup sleep 10 &"

is not running in daemon mode, keeping your ssh session connected. Ssh session will return in 10 seconds, despite you used nohup.

Reason is that remote stdout and stderr still connected to your session. It keeps ssh session alive, nohup does not help.

This:

ssh node "nohup sleep 10 1>/dev/null 2>/dev/null &"

returns immediately. It does start remote process with nohup and exits ssh session immediately.

Answer 3

Why not just tmux or screen and be done with it? For example:

$ tmux new -s SessionNameHere
$ nohup /path/to/your/script.sh

This is practical if it's something that will continuously loop or take a while to finish. You can disconnect from the session and it will remain active.

Answer 4

Shorter form:

ssh host "(command 1; command 2; ...) &>/dev/null &"

It appears, that bash itself performs detaching from terminal, so no nohup needed. I run Ubuntu 14.04 x86_64, bash 4.3.11.

Answer 5

And last but not least, remember not to allocate a tty for your session.
If you do that, it will not work.

So if you got any -t or -tt or -ttt and so on in your command. Thats the reason why it is not working for you.

So instead of

ssh -tt host "(command 1; command 2; ...) &>/dev/null &"

Use

ssh host "(command 1; command 2; ...) &>/dev/null &"

Answer 6

terry wang's answer helped me to get to mine, but i wanted to post this separately for anyone else looking to run more complex scripts:

    EXIT_CODE_FILE="/tmp/exit_code_file"
    TIMING_FILE="/tmp/timing_file"
    LOG_FILE="/tmp/log_file"
ssh "${SSH_USER}@${HOST_ADDRESS}" "nohup bash -c '
    # Source .bashrc because that doesn't happen automatically for ssh commands
    source ~/.bashrc

    echo "START: $(date +%s)" > ${TIMING_FILE}

    # Run your command / script and capture its stdout/stderr streams
    my_command.sh --with args > ${LOG_FILE} 2>&1

    # Save the exit code
    echo \$? > ${EXIT_CODE_FILE}

    echo "END: $(date +%s)" >> ${TIMING_FILE}
' > /dev/null 2>&1 &"

The nohup ensures that the entire wrapped command provided to bash runs to completion even when we disconnect, while its standard output streams are redirected to /dev/null and the & runs everything as a background process. In this way, only the command we're interested in prints its output to the given log file.

At this point it's easy enough to implement resumable polling behavior by polling the EXIT_CODE_FILE until we have a result.

echo "Waiting for remote operation to complete..."
while true; do
    # Check if the exit code file exists and has a value
    EXIT_CODE=$(ssh "${SSH_USER}@${HOST_ADDRESS}" "cat ${EXIT_CODE_FILE} 2>/dev/null || echo 'pending'")
if [ "$EXIT_CODE" != "pending" ]; then
    break
fi

echo -n "."
sleep 5

done
Operation completed
if [ "$EXIT_CODE" -eq 0 ]; then
    echo "Remote operation completed successfully"
else
    echo "Remote operation failed with exit code: $EXIT_CODE"
fi
echo ""
echo "Timing file:"
ssh "${SSH_USER}@${HOST_ADDRESS}" "cat ${TIMING_FILE}"
echo "Remote log output:"
ssh "${SSH_USER}@${HOST_ADDRESS}" "cat ${LOG_FILE}"
if [ "$EXIT_CODE" -ne 0 ]; then
    exit $EXIT_CODE
fi