Why my process is still running after I log out?

Question

After logging in through ssh, I type this command in bash:

sleep 50000000000000 &

Then I kill -9 the sleep process’s parent process (i.e., bash). Then the terminal window disconnects simultaneously.

When I log in again, I find that the sleep process is still alive.

Question: Why can the sleep process survive when I log out and the terminal is closed? In my mind, everything except daemons and nohup programs will be killed during logout. If sleep can survive in this way, does it mean that I can use this method instead of the nohup command?

Answer 1

Tl;dr:

Why can the sleep process survive when I log out and the terminal is closed? In my mind, everything except daemons and nohup programs will be killed during logout. If sleep can survive in this way, does it mean that I can use this method instead of the nohup command?

Unless the bash instance spawned by ssh has the huponexit option set, no process will be terminated by any mean upon exit / log out, and when the huponexit option is set, using kill -9 on the shell is not a good alternative to using nohup on the shell's child processes; nohup on the shell's child processes will still protect them from SIGHUPs not coming from the shell, and even when that is not important nohup is still to be preferred because it allows the shell to be terminated gracefully.

---

In bash there's an option called huponexit, which if set will make bash SIGHUP its children upon exit / logout;

In interactive non-login bash instances, such as in a bash instance spawned by gnome-terminal, this option is ignored; whether huponexit is set or unset, bash's children will never be SIGHUPped by bash upon exit;

In interactive login bash instances, such as in a bash instance spawned by ssh, this option is not ignored (however it is unset by default); if huponexit is set, bash's children will be SIGHUPped by bash upon exit / logout; if huponexit is unset, bash's children will not be SIGHUPped by bash upon exit / logout;

So in general exiting / logging out from an interactive login bash instance, unless the huponexit option is set, won't make the shell SIGHUP its children, and exiting / logging out from an interactive non-login bash instance won't make the shell SIGHUP its children regardless;

This is, however, irrelevant in this case: using kill -9 sleep will survive regardless, because killing its parent process (bash) won't leave a chance for the latter to do anything to the former (i.e., e.g., if the current bash instance was a login bash instance and the huponexit option was set, to SIGHUP it).

Adding to this, unlike other signals (like a SIGHUP signal sent to bash), a SIGKILL signal is never propagated to a process's child processes, hence sleep is not even killed;

nohup starts a process immune to SIGHUP signals, which is something different; it will prevent the process hanging up upon the reception of a SIGHUP signal, which in this case could be received by the interactive login bash instance in case the huponexit option was set and the shell exited; so technically using nohup to start a process in an interactive login bash instance with the huponexit option unset will prevent the process from hanging up upon the reception of a SIGHUP signal, but exiting / logging out of the shell won't SIGHUP it regardless;

In general, however, when nohup is needed to prevent SIGHUP signals coming from the parent shell, there's no reason to prefer the kill -9 on the parent method to the nohup on the child method; instead, it should be the opposite.

Killing the parent using the kill -9 method doesn't leave a chance for the parent to exit gracefully, while starting the child using the nohup method allows the parent to be terminated by other signals, such as SIGHUP (to make an example which makes sense in the context of a child started using nohup), which allow it to exit gracefully.

Answer 2

bash by default does not send down the HUP signal to child processes when exiting. More in detail (thanks @kos), it never does it for non-login shells.

You can configure bash to do it for login shells if set the option huponexit. In a terminal, do:

[romano:~] % bash -l

(this start a new "login" shell)

romano@pern:~$ shopt -s huponexit
romano@pern:~$ sleep 1234 &
[1] 32202
romano@pern:~$ exit
logout

Now check for the sleep process:

[romano:~] % ps augx | grep sleep
romano   32231  0.0  0.0  16000  2408 pts/11   S+   15:23   0:00 grep sleep

...not running: it has received the HUP signal and exited as requested.

Answer 3

If sleep can survive in this way, if it means that I can use this method instead of nohup command?

kill -9 is really not the way one should go. It's like shooting with a gun at the TV to turn it off. Besides the comedic component there is no advantage. Processes can't catch or ignore SIGKILL. If you don't give the process a chance to finish what it's doing and clean up, it may leave corrupted files (or other state) around and won't be able to start again. kill -9 is the last hope, when nothing else works.

---

Why sleep process can survive when I logout and terminal is closed. In my mind, everything except daemon and nohup program will be killed when logout.

What in your case happens:

The parent process of sleep is the currently running bash shell. When you kill -9 that bash, the bash process has not the chance to send a SIGHUP to any of its child processes, because SIGKILL (which is sent by kill -9) is not catchable by the process. The sleep process continues running. sleep now became an orphan process.

The init (PID 1) process does a mechanism called reparenting. That means that the init process now becomes the parent of that orphaned process. init is an exception, processes can become it's child as it collects processes that lost their original parent process. Btw: A daemon (like sshd) does that when "going in the background".

If that whould not happen, the orphaned process would later (when finished) become a zombie process. This is what happens when waitpid() is not called (a responsibility of the parent process which cannot be fullfilled when when that process got killed). init calls waitpid() in an specific intervall to avoid zombie childs.

Answer 4

The & starts the process in background. If you type ps -ef, you will see that parent process ID (PPID) of your sleep is your bash. Then logout and login again. The process will remain running after you logout. After you login for the second time you run ps -ef again. You will see that now the parent of your sleep process will be process with id "1". That is init, the parent of all processes.

Answer 5

There's two kinds of SIGHUP that can be involved here.

First, because in this kind of situation, the shell will be a so called Controlling Process (a session leader shell in ownership of a controlling terminal--a pseudoterminal created by the ssh login), a terminal driver will be obligated to send SIGHUP to all members of the foreground process group of the terminal when the terminal is disconnected.

(It will also send SIGHUP+SIGCONT to stopped backgrounded processes (orphan process groups)).

You've ruled such a system-sent SIGHUP out by backgrounding the process while being in an interactive shell. Such backgrounding will put the process(es) in a separate process group that won't be the foreground process group of the terminal, and so the driver won't send SIGHUP to such processes.

Backgrounding by itself isn't the issue, but you need to background from a shell that's noninteractive (interactive shells put backgrounded processes in separate process groups) and part of the foreground process group:

#from a ssh login shell
bash -c 'echo $$; sleep 1000 & sleep 1000 && wait' 
#a backgrounded and a nonbackgrounded sleep, both in the foreground process group

Now if you were to kill -9 the echoed shell pid from another terminal, both the sleep processes would receive SIGHUP from the terminal driver (a sigaction-registered signal handler would read the sender's pid as si_pid==0).

Second there's SIGHUPs sent by the shell depending on its settings (si_pid!=0). An interactive bash resends received SIGHUPs to all its foreground jobs. This behavior allows you to have several interactive bashes on top of each other (like if you were to run bash from the login bash and possibly some bash instances (no & at the end) from it) and rest assured a SIGHUP from the first (controlling) bash gets sent down even though each controlling bash will create its own process group and so the SIGHUP sent by the terminal driver wouldn't be able to reach the descendants. (Not all shell have this. E.g., Dash doesn't.) Bash will also resend SIGHUP to its backgrounded processes, but only if the huponexit setting is on and the job isn't disowned. None of these resends can apply if the shell was forcibly killed by SIGKILL (kill -9).

Edit:

Just found a very relevant answer https://stackoverflow.com/a/32786211/1084774 by gavv which has a couple more details.

Answer 6

Using & will cause program to run as background. To see the program in the background use bg command , and to make it running as foreground again, run fg.

Yes, there are many ways to keep program running even the main terminal exited.