umountall during a reboot runs for 1.5 hours

[S.Rider]

one of SuperDomes has a vpar with 2,988 disks and 778 filesystems (406 under Veritas volume manager, the rest are straight LVM). The "umountall -l" step during a shutdown runs for about 1.5 hours. Anyone know of a way to speed this up ? Are others experiencing the same issue ?

[Patrick Wallek]

Can you try to umount some of these filesystems before rebooting?

[A. Clay Stephenson · Accepted solution]

I can't imagine a system with that many filesystems. Backup and recovery must be a nightmare and I would hate to write that document and the testplan would be a monster. The real solution is to break this into more servers or at least more vpar's. You could replace the umountall with a scripts that get a list of the current filesystems (and hope that that doesn't hang) and then divide that into about 20 subprocesses that unmount the filesyetms in the background with a "wait" command in the parent script to wait for all the child processes to finish. A divide and conquer approach will help to some extent but I suspect that are just a handful of filesystems that are causing the real delays.

I assume that long before the filesystems are unmounted, your normal shutdown scripts have run and killed any processes which might hold the filesystems in use. Before your umountall runs, I would run a lsof on each filesystem to see what filesystems are actually in use. That will go a long way towards reducing the time if you can actually free the filesystems.

[TwoProc]

I'm guessing that all or most of these volumes are Oracle database raw partitions (hey, it's only a guess). This is because this is the only reason I can think of why someone would want to have so many different filesystems (but I'm allowing that there could be many others).

The following applies to the theory that you're using all of those separate areas as Oracle data (well, at least 3000-780 or so of them).

If so, you've got all the tools you need to fix your problems in your quest to separate your data by function. You've got Veritas!

You can now run asynchronous I/O on cooked (that is real) file systems with Oracle on HPUX with the Veritas file system. So, you don't need so many of these.

If the above is not the case, then please disregard.

[S.Rider]

A reboot from run-level-4 to run-level-4 takes about 3 hours on this server, so we don't get too many changes to experiment but...
I can watch it during the "Umount all filesystems" step, using a "bdf | wc -l" and it seems to go down pretty steadily so I think it's slow on everything not just on a few filesystems.
We don't have the option of splitting the system up into multiple servers, (the customer designs them, we just support them). It runs several large oracle instances. On one of the other vPars on this box, we just added a 1 TB filesystem. Like I said, we don't design these things. 8-)
We were actually doing some hardware changes on this server over this past weekend, so I had an opportunity to experiment. Nothing was running on the server but me. "ps -ef" didn't show anything else out there. I did a "umount -a" seperately before the reboot and that took an hour & a half. I'm guessing "umount" and "umountall" are both single-threaded. I'm thinking (like Clay suggested) I'll have to substitute my own umountall process that will do multiples at once. But since it's so difficult to get test time on this server, I was hoping someone was going to say "just use the option, and you get parallel umounts".
We're 11.11 by the way, most of the disks are on a XP12000 with 8 paths between the SuperDome - switch - XP12000.