When terminating a system, the shutdown module attempts to unmount all
file systems from under /oldroot. This reaps remaining file systems that
systemd cannot unmount and detaches /oldroot itself.
In case that running umount for some file system repeatedly fails, the
module reports this error and continues the processing in order to
shutdown the system. This handles a condition when the umount command
actually terminates but it can happen in some cases that it waits
indefinitely.
An example with NFS mounts:
# mount -t nfs 192.168.0.1:/srv/nfs/dir /mnt/nfs
# mkdir /mnt/nfs/dir2
# mount -t nfs 192.168.0.1:/srv/nfs/dir2 /mnt/nfs/dir2
# touch /mnt/nfs/dir2/file
# systemd-run -pKillMode=none -pSendSIGKILL=no tail -f /mnt/nfs/dir2/file
Running as unit: run-r367825c967ca4d88a793ae4793c02f8b.service
# systemctl poweroff
The invoked tail command escapes normal termination by systemd and
prevents stopping mnt-nfs.mount and mnt-nfs-dir2.mount as it makes the
mounts busy. Systemd then again attempts to unmount these file systems
in systemd-shutdown but this fails as well. The utility tries to unmount
/mnt/nfs/dir2 but the kernel waits indefinitely doing a path lookup for
/mnt/nfs because network is no longer available at that point. The
systemd-shutdown gives up after 90 seconds. Finally, the control is
transferred to dracut which tries to unmount the file systems in the
same way and ends up indefinitely waiting on umount to finish.
This situation causes that the system hangs during shutdown. The patch
improves the shutdown module to add a timeout of 90 seconds for the
umount operation and continue with the shutdown if it gets reached,
similarly to what systemd-shutdown does.
If a process (maybe plymouth) was still pinning /oldroot, then shutdown
would
- kill -9 $pid
- umount_a
- umount_a
in a very short timeframe. A small sleep hopefully lets the scheduler free
up /oldroot in the mean time.
Harald Hoyer