Mount /etc in a separate LV

[mogiro]

Hi,
I've got a system with /etc under /
Now / is 91% used and I would like to create a new LV (there is enough space at the VG to do so), unmount /etc and mount it again on the new LV.
Is it a good solution? Can i do it with the machine up?
Thanks in advance for your help and if you need further data don´t hesitate to ask me for.

[MarkSyder]

Don't!

Your system can only mount filesystems it finds in /etc/fstab. It can't mount /etc if it can's see /etc/fstab and it can't see /etc/fstab if /etc isn't mounted. Catch 22 anyone?

Mark Syder (like the drink but spelt different)

[A. Clay Stephenson · Accepted solution]

/etc MUST be in the / filesystem because that is where /etc/fstab which tells the system how to mount the other filesystems. How are you going to mount filesystems when /etc isn't mounted? Moreover, /etc/passwd and /etc/group won't be found.

Now you make filesystems of some of the directories under /etc, (e.g. /etc/opt, /etc/lp) and /etc/opt can be rather large on a system running Openview Operations, for example.

However, you are really treating the symptoms rather than the problem. Once a system is stable, the size of / should be all be static --- so there is nothing wrong with 91% as long as you don't allow the system to fill up (e.g. a core dump while cd'ed into the / filesystem could put you over the top). You probably need to do some cleanup (look for core files immediately and also regulkar files under /dev) and that will solve your problem.

[mogiro]

A little bit more of info:

c200:/# du -kx | sort -rn | head -20
180560 .
133840 ./etc
64216 ./etc/opt
63192 ./etc/vx
51440 ./etc/opt/OV
51432 ./etc/opt/OV/share
42336 ./sbin
32736 ./etc/vx/type
29440 ./etc/vx/static.d
28728 ./etc/vx/static.d/build
28256 ./etc/opt/OV/share/conf
15480 ./etc/opt/OV/share/bitmaps/C
15480 ./etc/opt/OV/share/bitmaps
14840 ./etc/vx/type/static
8888 ./etc/vx/type/gen
7088 ./etc/opt/resmon
6720 ./etc/vx/type/raid5
5928 ./sbin/fs
4672 ./etc/opt/OV/share/conf/stacks/pmd
4672 ./etc/opt/OV/share/conf/stacks

So, you mean I can create a new lv and move /etc/opt with the machine up?
And thanks for the quick replies, I didn´t keep in mind /etc/fstab, but you are absolutely true.

[Tim Nelson]

What version of HPUX and what is the size for the root filesystem ? 91% full for a standard 11.0 install is normal, 91% full for an 11.11 may not, etc.. All depends on how big root is.

What you see may be normal at 91%, as stated earlier unless something or someone is putting files in / then it is pretty static and not an issue. Save yourself time and effort by not messing with root.

How do you know what is in /etc is what needs addressing, how about a rougue file in /dev, /sbin, or / itself ?

[TwoProc]

I've got /etc/opt and /etc/lp mounted on other volumes with no problems. This was to handle basically the same space problem you're having.

However, I implemented this change with a reboot, I really don't know about doing this without any downtime. I'd suppose that if the print spooler is shutdown you could possibly do /etc/lp while still up. I wouldn't suspect that /etc/opt would be a good candidate for doing without downtime unless you can turn off lots and lots of services, but if that is the case, you could come down to single user mode and proceed normally just as well.

[Bill Hassell]

> 51440 ./etc/opt/OV

Really, really large at 51 MB!!

> So, you mean I can create a new lv and move /etc/opt with the machine up?

Normally, /etc/opt/OV is just a few KB, so this means you are probably running something like OpenView Node Manager or similar. Although large data files should be in /var/opt or possibly /opt/OVsomething,the good news is that OV can't do anything until all the filesystems are mounted and networking is running, so you should be able to create a 100 to 200 MB lvol, run newfs and get it ready to copy the files and directories. Also add the new mountpoint to fstab (ie, /dev/vg00/lvolXYZ /etc/opt/OV ...)

Notice I said copy, not move? Always be safe and copy so you can count files and directories to make sure everything is there. You can test your copy and count commands while you are running, but it is very likely that the data is constantly changing and there are lots of open files and directories. So the real copy will take place in single user mode. You boot to single user, mount your new lvol to a temporary directory (ie, mkdir /mnt; mount /dev/vg00/lvol99 /mnt) and copy the files. Then count them:

find /etc/opt/OV -type d | wc -l
find /etc/opt/OV -type f | wc -l
find /mnt -type d | wc -l
find /mnt -type f | wc -l

The counts will match except /mnt will have 1 additional direcory. Do NOT expect du to match. There are several reasons why the size does not match (may be larger or smaller). Once the counts are the same, remove the old files and directories:

rm -r /etc/opt/OV

Now re-create OV as an empty directory:

mkdir /etc/opt/OV
chmod 755 /etc/opt/OV

Now unmount /mnt1 and test fstab by issuing:

mount /etc/opt/OV

It should mount to the right location and you'll see the copied files. You should have more than 50MB of additional free space in /. Now you can reboot into multiuser mode: reboot -q

[A. Clay Stephenson]

Ok, you do appear to be run OV/O (or VP/O or IT/O --- whatever the marketers have decided to call the product today) which includes Network Node Manager. Since your /etc/opt is ~ 60MiB, you can free up that much space in / and bring up to a reasonable value.

You can do this "on the fly" but you will have to shutdown OV/O, Data Protector (or OB2), and anything else that uses /etc/opt.

I'll assume that all you are using is OV/O.

ovstop -c
Create a new LVOL (e.g. /dev/vg00/lvol10 although it doesn't have to be in vg00) and create a new filesystem on this LVOL and create a temporary mountpoint for it, e.g. /tmpetcopt and mount the filesystem.

Do NOT use cp -R (even with -p) because we want to preserve directory ownerships and modes as well as all other file attributes.

Use a temporary location that has at least 70MiB or so free, e.g. /xxx/yyy

# Backup the files under /etc/opt
cd /etc/opt
find . -print | cpio -ocBv > /xxx/yyy/etcopt.cpio

# Restore the files to /tmpetcopt
cd /tmpetcopt
chown root:sys .
chmod 555 .
cpio -icvdum < /xxx/yyy/etcopt.cpio

#Now, rename the original /etc/opt

mv /etc/opt /etc/opt.orig

mkdir /etc/opt
chown root:sys /etc/opt
chmod 555 /etc/opt
umount /tmpetcopt

# Now create an /etc/fstab for /etc/opt using vi


mount /etc/opt # This will read your fstab entry to insure it's correct.

ovstart -c

When everything looks ok,
rm -r /etc/opt.orig

[Tim Nelson]

Two more ideas.

1) what is in /etc/opt/OV, If it is just some big log file clean it up ?

2) copy/move /etc/opt/OV to somewhere else ( i.e. /var/opt/OV then replace /etc/opt/OV with a link to /var/opt/OV ??

[Murat SULUHAN]

Hi

.Create a lv as big as you need (lvEtcOptOV)
.mount just created lv to temporary directory (/etcOptOV
.stop OV related processes
.backup /etc/opt/OV directory and subdirectories
.mv /etc/opt/OV /etcOptOV
.umount /etcOptOV
.mount lvEtcOptOV on /etc/opt/OV
.start OV

Best Regards
Murat

[mogiro]

I will take into account all the solutions you have gave me for the next maintenance stop, because I´m afraid that unmounting /etc/opt with the systema up is not a good idea.
In the meantime, I´ve found out two files that surely can be removes:
c200:/etc/lvmconf# ll
total 5440
---------- 1 root root 0 Feb 3 2003 lvm_lock
-rw------- 1 root sys 330752 Sep 17 20:21 vg00.conf
-rw------- 1 root sys 330752 Sep 10 19:40 vg00.conf.old
-rw-r--r-- 1 root sys 64 Sep 17 20:21 vg00.mapfile
-rw------- 1 root sys 1042432 Sep 17 20:21 vg_tradex.conf
-rw------- 1 root sys 1042432 Sep 10 19:40 vg_tradex.conf.old
-rw-r--r-- 1 root sys 25 Sep 17 20:21 vg_tradex.mapfile


Do you believe I can remove the .old config files? I think so, but I prefer to count with your expert point of view.
Thanks again!!

[Bill Hassell]

The .old files you show are less than 3 MB so I wouldn't bother -- you won't gain any significant space. So there's not much you can do until you shutdown the OpenView application(s) (which does not mean shutting down the server).

[A. Clay Stephenson]

None of the files you listed amount to any significant data volume. It just won't make any difference. As I said earlier, you can move /etc/opt "on the fly" as long as you shutdown OV/O and/or NNM first. OV/O would really only have to be down for 5 minutes or so and in the meantime the clients will simply cache any messages until the OV/O server becomes available again.

[Steven E. Protter]

Shalom,

I'd suggest:

1) changing the configuration of the program storing data in /etc/opt/OV to store elsewhere. /var seems a good candidate.

2) Mount a logical volume/filesystem with the target /etc/opt/OV /dev/vg01/lvolov /etc/opt/OV . That might fool it.

I can't test suggestion 2 and would do #1 if it were my system.

SEP

[TwoProc]

Wow, it's funny how these things go around and around.

One of my admins just proved that this is do-able.

Just this morning, one of the sysadmins tells me that a box has /root full. I advised him to move /etc/opt and /etc/lp to new mount points in single user mode. He wanted to avoid scheduling the downtime and do it live. Since it's test, and we were just discussing it in the forums - I told him to give it a try by incrementally stopping processes attached to /etc/opt and of course, the lp scheduler.

Well, he stopped lp scheduler, asked developers to log out of Unix (most work from Oracle anyways) because parts of the ldap client software is located in /etc/opt, stopped the ems monitor, and then stopped the insight processes.

He then created his lvols, made his directory moves, fixed /etc/fstab, brought everything back up, let the developers back on, and he was done.

So, he proved that this can be done live, at least as far as /etc/opt and /etc/lp.

BUT! I think what allowed us to do this is that we don't have the account information for the Oracle database server user and any applications servers/services users that use it hosted by the ldap server. Had those critical server users been part of ldap, I'm thinking it probably would not have worked.