increasing root file system

[mark_238]

i wanted to increase the root file system but met with the below messages

* Performing task "Unmount a file system on a logical volume.": ^ ³
³ ³ Unmounting the file system on /dev/vg00/lvol3. ³
³³*Executing the following ommand: ³
³ ³ /usr/sbin/umount /dev/vg00/lvol3 ³
³ ³ * Command completed with exit status 1. ³
³ ³ ERROR umount: cannot unmount / : Device busy


i guess the problem is that
1) i dont have online jfs
2) because it is root, i need to g

[Michael Tully]

You cannot increase the root filesystem while the system is up, even with the online-JFS product. The extents of the logical volumes 1 2 and 3 (/ /stand and primary swap) must be contiguous, meaning unbroken extents of the disk. The only way to extend any of these volumes is to create an ignite image of the current /dev/vg00, boot from the tape created and extend the logical volume(s) that way.

# /opt/ignite/bin/make_tape_recovery -x inc_entire=vg00 -I -v -a /dev/rmt/?mn

You can the latest ignite version from here, it is free.
http://www.software.hp.com/products/IUX/download.html

[mark_238]

not even in the single user mode which i can increase the file system?

[Rajeev Shukla]

Hi,
It is not easy to increase the root filesystem (/dev/vg00/lvol3) even if you had OnlineJFS. Because root LV should be contiguous, you need to have free space after the end of /dev/vg00/lvol3 which you probably wont have since you might have created other LV just after that. So there are to options to increase root LV.
1. Move the other LV which is just after lvol3 say lvol4 to another free place. and then extend the root by booting the system in single user mode.
2. Take a Ignite backup of you system and then boot through the make_recovey tape and in advanced options change the layout, which allows you to choose different filesystem sizes compared to what you had originally.
I hope this makes clear, if not let me know

Cheers
Rajeev

[Michael Tully]

Not even in single user mode, as /, /stand and primary swap are actually used. Ignite is the way to do it and it actually does the job quite well. The whole operation including the system going down and then up should only be around 1.5 hours, depending on the size of the /dev/vg00 volume and the model of your system.

[Karthik S S]

Refer to the following links,

http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=86340

http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=242644

http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=112663


http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=112393

-Karthik S S

[Rajeev Shukla]

Ya i got you Michael, i realised just after posting the answer,
Thanks for correcting me

Cheers
Rajeev

[mark_238]

i have 2 files which are on my root file system

-rw------- 1 root sys 8615436 Sep 25 2002 ./dev/vg00/core
---------- 1 root root 0 Dec 9 09:57 core

1) i just trim the core file. basically, i change the mode of the file to 000 so that it is not writable. but the system can still write to that file as evidenced by the timestamp. what is wrong?

2) also, can i trim the ./dev/vg00/core without causing any hapzard to the system?

thanks

[Michael Tully]

Basically you have a process that is writing consistently to core. This process will need to be terminated. Try:

# fuser /core

This should give you the process(es) to investigate.

Yes you can safely remove /dev/vg00/core

[mark_238]

okay.. i issued the command as belows


>fuser /core
/core:

[Bill Hassell]

Identify the source of the core file (the program that crashed) with the file command:

file core

The zero-length file is of no value, remove it. The core file in /dev/vg00 does not belong there either. Use the file command so you know the program that is not working properly. Then remove the core file. The only value for a core file is when you have the source code for the program, knowledgeable programmers and compiler+tools to find the bad code and fix it.

To avoid core files inthe future, put this in /etc/profile:

ulimit -Sc 0

[mark_238]

thanks for your guidance...

currently, my ulimit -a gives "coredump(blocks) 4194303"

my shell is /sbin/sh. so in this case, /etc/profile is sourced. if its other shells, it will not be sourced?

all i need is to add this "ulimit -Sc 0" to my /etc/profile so that my core file will not grow?

thanks for your advise

[Bill Hassell]

Whenever you login with a POSIX-style shell (/sbin/sh, /usr/bin/sh, /usr/bin/ksh) then /etc/profile will be sourced. With ulimit -Sc 0 in you /etc/profile, the ulimit -a command will now show 0 as the size of a core file generated by your current session. However, it will not affect some other session nor will it affect cron jobs or daemons. Since a core file means something has broken, start by finding out what program caused this core file:

file core

Then post the results and we can help you find the bad program and perhaps a fix for it. As far as other shells, if you loaded bash on your system and some users are running that, it will also use /etc/profile. For csh users, the file is /etc/csh.login and of course the syntax is quite different. Any other shells, you'll have to read the man page for it to see how profiles are setup.

AS far as the current core file that seems to grow, remove the corefile after running the file command on it. Then make sure /dev and /dev/vg00 both have 755 permissions. If either directory has 777 permissions, you have a huge security problem..change /dev to 555 and bin:bin, and vg00 to 755 root:root. And if /dev is wrong, there may be many other bad directory and file permissions throughout your system. Type the command: umask as root and if it says 00 then you have a lot of work to do. First, put umask 022 in /etc/profile. Then have an HP-UX security specialist look over your system. You can do it yourself but it will take a while and you'll need to ask a lot of questions. A quick way to see how bad things are is to run this report:

find / -perm -002 \( type f -o -type d \) -exec ll -d {} \;

Now you can ignore man page directories as these must be writable by all, and also ignore /var/tmp and /tmp for the same reason. The rest will require a knowledgeble decision on what the correct permissions should be.

[mark_238]

hi.. thanks for replying.. actually, i find the root cause.. the root file system was full cause i had a file /etc/rc.log which grew tremendously.. each night.

and the situation happened on tuesday, since from friday, there are 4 days of interval.. and it grew beyond what it can contain.. and hence full..

i had to go to
gsp> rs

at the risk of my oracle, application to get it restart.

[Bill Hassell]

Ouch!!! Using the GSP to reset the computer is the same as turning power off, something that is very detrimental to database engines like Oracle. Since you found the large file, look at the end of the file (tail /etc/rc.log) to see what junk is in the file. Most liekly, someone wrote a startup script and did not code the required steps correctly. Once you know the reason for the root filesystem being full, you can zero out the file with:

cat /dev/null > /etc/rc.log

and assuming that rc.log is dozens of megabytes in size, the root filesystem will be closer to normal. Now find out what other problems there are in the / filesystem:

du -kx / | sort -rn | more

This should produce a list something like this:

71408 /
32984 /sbin
32208 /etc
14736 /etc/vx
13448 /etc/opt
12536 /etc/vx/type
8056 /etc/opt/resmon
6368 /etc/opt/resmon/lbin

If /sbin and /etc are *NOT* the first directories after /, then those directories are misplaced and must be moved.

[doug mielke]

You can consider using some soft links
ln -s logfile /emptyfilesystem/file2

to create a link from a filesystem with space to files or directories in the root dir, like some of your log files.
This way, if they grow quickly, they won't fill up root.

[mark_238]

Ouch!!! Using the GSP to reset the computer is the same as turning power off, something that is very detrimental to database engines like Oracle

>> yup. i agree. but i only knew this way to handle this root file system full issue

Once you know the reason for the root filesystem being full, you can zero out the file with:

cat /dev/null > /etc/rc.log


>> we have to do this manually, right?

du -kx / | sort -rn | more

>> gives the below
88111 /
41917 /etc
18690 /sbin


You can consider using some soft links
ln -s logfile /emptyfilesystem/file2

to create a link from a filesystem with space to files or directories in the root dir, like some of your log files.
This way, if they grow quickly, they won't fill up root.

>> i think this is fine , right?

basically, now i do manually housekeeping.. of /etc/rc.log

[Con O'Kelly]

Hi Mark

A few points that may help with understanding the / filesystem.

The / fileystem on 11.00 doesn't need to be more than around 120-130MB. If your / fileystem is around this size there should be no need to increase it.

Secondly there should only be a few log files in / fileystem (under the /etc directory) & these should be very small in size (eg /etc/shutdownlog & /etc/rc.log).
The rc.log file is used at system boot & that's it. Nothing else should ever write to this log. Therefore once the system is up the size remains static. As a result you shouldn't need to do regular maintenance on this log file. There shouldn't be a need to link log files from the / filesystem. Almost all log files should be in /var.

As Bill already mentioned I'd look very carefully at /etc/rc.log & ensure that only startup scripts write to this log. As I said once the system has booted this log should never be written to.

The / fileystem should be almost static in size and only fills up if some process(es) accidentally writes to it or core files are dumped there. From your du output, your / filesystem look pretty healthy at about 88MB.

Cheers
Con

[mark_238]

thanks for the explanation..

i have monitor for a few days. and this file keeps growing each day.

-rw-r--r-- 1 root root 33476148 Dec 11 02:01 ./etc/rc.log

and yes , u are right, some applications are writing to this log file.

[Con O'Kelly]

OK looks like it is being written to at around 2.00am.
Can you identify the script/application?
What is being written? This should give you a clue as to the application/script.
Possibly a root cronjob?
Check root cron jobs that run around this time & see if anything is there.

Once you fix this problem you'll be fine.
33MB for this file is way too big - should be around 100K or so.

Cheers
Con

[Bill Hassell]

The rc.log file is only written to once during the bootup process. Therefore, you have a bad startup script located in /sbin/init.d or possibly (but incorrectly) in the /sbin/rc*.d directories. Most likely this script is fairly new so you can sort the directories by the change date:

ll -t /sbin/init.d | more

find /sbin/rc?.d -type f

In the first listing, the items at the top of the listing were changed most recently. In the second listing, there should be no entries at all since these directories should have only symlinks. And of course, the information wastefully logged into rc.log should be examined...it will tell you the program that is not working properly. My guess is that someone has added a startup script that was not designed correctly.

[mark_238]

ll -t /sbin/init.d | more
gives the files with 2002 entries at the top


find /sbin/rc?.d -type f
gives no entries

my proposed solution is to have a crontab entry that housekeep this file. what do u think?

[Con O'Kelly]

Hi Mark

As I said before nothing should be writing to /etc/rc.log after system boot.
If this file is growing on a daily basis then it is not a startup script that is responsible.
I'd advise fixing the problem as opposed to housekeeping on this file.

What is being written to the file?
Find the application/script that is writing to /etc/rc.log and stop it from writing to it.

Cheers
Con