topic Re: increasing root file system in Operating System - HP-UX

increasing root file system

mark_238 — Mon, 08 Dec 2003 19:31:32 GMT

i wanted to increase the root file system but met with the below messages

* Performing task "Unmount a file system on a logical volume.": ^ Â³
Â³ Â³ Unmounting the file system on /dev/vg00/lvol3. Â³
Â³Â³*Executing the following ommand: Â³
Â³ Â³ /usr/sbin/umount /dev/vg00/lvol3 Â³
Â³ Â³ * Command completed with exit status 1. Â³
Â³ Â³ ERROR umount: cannot unmount / : Device busy

i guess the problem is that
1) i dont have online jfs
2) because it is root, i need to g

Re: increasing root file system

Michael Tully — Mon, 08 Dec 2003 19:36:00 GMT

You cannot increase the root filesystem while the system is up, even with the online-JFS product. The extents of the logical volumes 1 2 and 3 (/ /stand and primary swap) must be contiguous, meaning unbroken extents of the disk. The only way to extend any of these volumes is to create an ignite image of the current /dev/vg00, boot from the tape created and extend the logical volume(s) that way.

# /opt/ignite/bin/make_tape_recovery -x inc_entire=vg00 -I -v -a /dev/rmt/?mn

You can the latest ignite version from here, it is free.
http://www.software.hp.com/products/IUX/download.html

Re: increasing root file system

mark_238 — Mon, 08 Dec 2003 19:40:31 GMT

not even in the single user mode which i can increase the file system?

Re: increasing root file system

Rajeev Shukla — Mon, 08 Dec 2003 19:40:42 GMT

Hi,
It is not easy to increase the root filesystem (/dev/vg00/lvol3) even if you had OnlineJFS. Because root LV should be contiguous, you need to have free space after the end of /dev/vg00/lvol3 which you probably wont have since you might have created other LV just after that. So there are to options to increase root LV.
1. Move the other LV which is just after lvol3 say lvol4 to another free place. and then extend the root by booting the system in single user mode.
2. Take a Ignite backup of you system and then boot through the make_recovey tape and in advanced options change the layout, which allows you to choose different filesystem sizes compared to what you had originally.
I hope this makes clear, if not let me know

Cheers
Rajeev

Re: increasing root file system

Michael Tully — Mon, 08 Dec 2003 19:43:58 GMT

Not even in single user mode, as /, /stand and primary swap are actually used. Ignite is the way to do it and it actually does the job quite well. The whole operation including the system going down and then up should only be around 1.5 hours, depending on the size of the /dev/vg00 volume and the model of your system.

Re: increasing root file system

Karthik S S — Mon, 08 Dec 2003 19:46:27 GMT

Refer to the following links,

http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=86340

http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=242644

http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=112663

http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=112393

-Karthik S S

Re: increasing root file system

Rajeev Shukla — Mon, 08 Dec 2003 19:54:25 GMT

Ya i got you Michael, i realised just after posting the answer,
Thanks for correcting me

Cheers
Rajeev

Re: increasing root file system

mark_238 — Mon, 08 Dec 2003 20:09:31 GMT

i have 2 files which are on my root file system

-rw------- 1 root sys 8615436 Sep 25 2002 ./dev/vg00/core
---------- 1 root root 0 Dec 9 09:57 core

1) i just trim the core file. basically, i change the mode of the file to 000 so that it is not writable. but the system can still write to that file as evidenced by the timestamp. what is wrong?

2) also, can i trim the ./dev/vg00/core without causing any hapzard to the system?

thanks

Re: increasing root file system

Michael Tully — Mon, 08 Dec 2003 20:16:28 GMT

Basically you have a process that is writing consistently to core. This process will need to be terminated. Try:

# fuser /core

This should give you the process(es) to investigate.

Yes you can safely remove /dev/vg00/core

Re: increasing root file system

mark_238 — Mon, 08 Dec 2003 20:29:00 GMT

okay.. i issued the command as belows

>fuser /core
/core:

Re: increasing root file system

Bill Hassell — Mon, 08 Dec 2003 20:45:15 GMT

Identify the source of the core file (the program that crashed) with the file command:

file core

The zero-length file is of no value, remove it. The core file in /dev/vg00 does not belong there either. Use the file command so you know the program that is not working properly. Then remove the core file. The only value for a core file is when you have the source code for the program, knowledgeable programmers and compiler+tools to find the bad code and fix it.

To avoid core files inthe future, put this in /etc/profile:

ulimit -Sc 0

Re: increasing root file system

mark_238 — Mon, 08 Dec 2003 21:37:37 GMT

thanks for your guidance...

currently, my ulimit -a gives "coredump(blocks) 4194303"

my shell is /sbin/sh. so in this case, /etc/profile is sourced. if its other shells, it will not be sourced?

all i need is to add this "ulimit -Sc 0" to my /etc/profile so that my core file will not grow?

thanks for your advise

Re: increasing root file system

Bill Hassell — Tue, 09 Dec 2003 10:05:59 GMT

Whenever you login with a POSIX-style shell (/sbin/sh, /usr/bin/sh, /usr/bin/ksh) then /etc/profile will be sourced. With ulimit -Sc 0 in you /etc/profile, the ulimit -a command will now show 0 as the size of a core file generated by your current session. However, it will not affect some other session nor will it affect cron jobs or daemons. Since a core file means something has broken, start by finding out what program caused this core file:

file core

Then post the results and we can help you find the bad program and perhaps a fix for it. As far as other shells, if you loaded bash on your system and some users are running that, it will also use /etc/profile. For csh users, the file is /etc/csh.login and of course the syntax is quite different. Any other shells, you'll have to read the man page for it to see how profiles are setup.

AS far as the current core file that seems to grow, remove the corefile after running the file command on it. Then make sure /dev and /dev/vg00 both have 755 permissions. If either directory has 777 permissions, you have a huge security problem..change /dev to 555 and bin:bin, and vg00 to 755 root:root. And if /dev is wrong, there may be many other bad directory and file permissions throughout your system. Type the command: umask as root and if it says 00 then you have a lot of work to do. First, put umask 022 in /etc/profile. Then have an HP-UX security specialist look over your system. You can do it yourself but it will take a while and you'll need to ask a lot of questions. A quick way to see how bad things are is to run this report:

find / -perm -002 \( type f -o -type d \) -exec ll -d {} \;

Now you can ignore man page directories as these must be writable by all, and also ignore /var/tmp and /tmp for the same reason. The rest will require a knowledgeble decision on what the correct permissions should be.

Re: increasing root file system

mark_238 — Tue, 09 Dec 2003 22:22:14 GMT

hi.. thanks for replying.. actually, i find the root cause.. the root file system was full cause i had a file /etc/rc.log which grew tremendously.. each night.

and the situation happened on tuesday, since from friday, there are 4 days of interval.. and it grew beyond what it can contain.. and hence full..

i had to go to
gsp> rs

at the risk of my oracle, application to get it restart.

Re: increasing root file system

Bill Hassell — Wed, 10 Dec 2003 08:27:38 GMT

Ouch!!! Using the GSP to reset the computer is the same as turning power off, something that is very detrimental to database engines like Oracle. Since you found the large file, look at the end of the file (tail /etc/rc.log) to see what junk is in the file. Most liekly, someone wrote a startup script and did not code the required steps correctly. Once you know the reason for the root filesystem being full, you can zero out the file with:

cat /dev/null > /etc/rc.log

and assuming that rc.log is dozens of megabytes in size, the root filesystem will be closer to normal. Now find out what other problems there are in the / filesystem:

du -kx / | sort -rn | more

This should produce a list something like this:

71408 /
32984 /sbin
32208 /etc
14736 /etc/vx
13448 /etc/opt
12536 /etc/vx/type
8056 /etc/opt/resmon
6368 /etc/opt/resmon/lbin

If /sbin and /etc are *NOT* the first directories after /, then those directories are misplaced and must be moved.

Re: increasing root file system

doug mielke — Wed, 10 Dec 2003 09:41:05 GMT

You can consider using some soft links
ln -s logfile /emptyfilesystem/file2

to create a link from a filesystem with space to files or directories in the root dir, like some of your log files.
This way, if they grow quickly, they won't fill up root.

Re: increasing root file system

mark_238 — Wed, 10 Dec 2003 20:09:26 GMT

Ouch!!! Using the GSP to reset the computer is the same as turning power off, something that is very detrimental to database engines like Oracle

>> yup. i agree. but i only knew this way to handle this root file system full issue

Once you know the reason for the root filesystem being full, you can zero out the file with:

cat /dev/null > /etc/rc.log

>> we have to do this manually, right?

du -kx / | sort -rn | more

>> gives the below
88111 /
41917 /etc
18690 /sbin

You can consider using some soft links
ln -s logfile /emptyfilesystem/file2

to create a link from a filesystem with space to files or directories in the root dir, like some of your log files.
This way, if they grow quickly, they won't fill up root.

>> i think this is fine , right?

basically, now i do manually housekeeping.. of /etc/rc.log

Re: increasing root file system

Con O'Kelly — Wed, 10 Dec 2003 21:00:19 GMT

Hi Mark

A few points that may help with understanding the / filesystem.

The / fileystem on 11.00 doesn't need to be more than around 120-130MB. If your / fileystem is around this size there should be no need to increase it.

Secondly there should only be a few log files in / fileystem (under the /etc directory) & these should be very small in size (eg /etc/shutdownlog & /etc/rc.log).
The rc.log file is used at system boot & that's it. Nothing else should ever write to this log. Therefore once the system is up the size remains static. As a result you shouldn't need to do regular maintenance on this log file. There shouldn't be a need to link log files from the / filesystem. Almost all log files should be in /var.

As Bill already mentioned I'd look very carefully at /etc/rc.log & ensure that only startup scripts write to this log. As I said once the system has booted this log should never be written to.

The / fileystem should be almost static in size and only fills up if some process(es) accidentally writes to it or core files are dumped there. From your du output, your / filesystem look pretty healthy at about 88MB.

Cheers
Con

Re: increasing root file system

mark_238 — Wed, 10 Dec 2003 22:19:36 GMT

thanks for the explanation..

i have monitor for a few days. and this file keeps growing each day.

-rw-r--r-- 1 root root 33476148 Dec 11 02:01 ./etc/rc.log

and yes , u are right, some applications are writing to this log file.

Re: increasing root file system

Con O'Kelly — Wed, 10 Dec 2003 22:28:13 GMT

OK looks like it is being written to at around 2.00am.
Can you identify the script/application?
What is being written? This should give you a clue as to the application/script.
Possibly a root cronjob?
Check root cron jobs that run around this time & see if anything is there.

Once you fix this problem you'll be fine.
33MB for this file is way too big - should be around 100K or so.

Cheers
Con