fsck, mountall, /var and Lucid

So yesterday I rebooted a Lucid server I administer, and fsck ran. Ok, that’s cool. Granted it takes about 45 minutes on my RAID1 terabyte filesystem, but so be it. As in the past, I was slightly annoyed again that upstart/plymouth did not tell me that it was fscking my drive like my desktop does (may be it’s because this was an upgrade from Hardy and not a fresh install? It would be nice if I looked into why), but I knew that was what was happening, so I went about my business .

Until… there was a failure that had to be manually resolved by fsck. Looking at the path, it was no big deal (easily restoreable), so I just needed to run ‘fsck /dev/md2’. Hmmm, /dev/md2 is /var on this system, and mountall got stuck cause /var couldn’t be mounted. Getting slightly more annoyed, I tried to reboot with ‘Ctrl+Alt+Del’, but that didn’t work, so I had to SysRq my way out (using Alt+SysRq+R, Alt+SysRq+S, Alt+SysRq+U, and finally Alt+SysRq+B) and boot into single usermode. Surely I could reboot to runlevel 1 and get a prompt…. 45 minutes later (ie, after another failed fsck on /var) I was shown to be wrong. Thankfully I had an amd64 10.04 Server CD handy and booted into rescue mode, which in the end worked fine for fscking my drive manually.

Why was running fsck manually so hard? Why is plymouth/upstart so quiet on my server?

It turns out because I removed ‘splash quiet’ from my kernel boot options, plymouth wouldn’t show the message to ‘S’ (skip) or ‘M’ (maunually recover) /var. It was still running, so I could press ‘m’ to get to a maintenance shell (you might need to change your tty for this).

For the plymouth/upstart silence I came across the following:

• http://staff.adams.edu/~cdmiller/posts/Ubuntu-Lucid-server-text-console/
• http://staff.adams.edu/~cdmiller/posts/Ubuntu-Lucid-server-disable-plymouth/
• http://staff.adams.edu/~cdmiller/posts/Ubuntu-Lucid-server-upstart/

In short, the above has you add to /etc/modprobe.d/blacklist-custom.conf:
blacklist vga16fb

Then adjust your kernel boot line to have:
ro nosplash INIT_VERBOSE=yes init=/sbin/init noplymouth -v

This is not as pretty as SysV bootup, but is verbose enough to let you know what is happening. The problem is that because there is no plymouth, there is no way enter a maintenance shell when you need to, so the above is hard to recommend. :(

To me, it boils down to the following two choices:

• Boot with ‘splash’ but without ‘quiet’ and lose boot messages but gain fsck feedback
• Boot without ‘splash quiet’ and lose fsck feedback and remember you can press ‘m’ to enter a maintenance shell when there is a problem

It would really be nice to have both fsck feedback and no splash, but this doesn’t seem possible at this time. If someone knows a way to do this on Lucid, please let me know. In the meantime, I have filed bug #613562.