topic Re: Root at 97%. in Operating System - HP-UX

Root at 97%.

J.Kimbrell — Wed, 19 Jan 2011 20:49:20 GMT

Just came into this and found on one of my systems that the root file system was at 97% and there's nothing to remove. When they brought up the system, they most have missed a number.
Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol3 1179648 1148184 31376 97% /
/dev/vg00/lvol1 1776056 31248 1567200 2% /stand
/dev/vg00/lvol8 20971520 6218592 14637784 30% /var
/dev/vg00/lvol7 4587520 1712568 2853528 38% /usr
/dev/vg00/lvol4 524288 270896 251584 52% /tmp
/dev/vg00/lvol6 5111808 3996016 1107136 78% /opt
/dev/vg00/lvol5 114688 8728 105200 8% /home

Any ideas how I can allocate more space to the root without days of aggravation and without much production impact?
It has been this way for about 1.5 years now without much of a problem but having root at 97% is not a good thing.

Re: Root at 97%.

James R. Ferguson — Wed, 19 Jan 2011 21:08:51 GMT

Hi:

You might remove old LVM configuration backups in '/etc/lvmconf'. These would be suffixed with '.old'.

You can look for any 'core' files:

# find / -xdev -type f -name core -exec ls -l {} +

You can evaluate which directories are the largest and then examine their contents:

# du -xk /|sort -nrk1,1|more

Regards!

...JRF...

Re: Root at 97%.

John Jimenez — Thu, 20 Jan 2011 00:42:52 GMT

If you do not mind me asking what version of HPUX do you have? If its version 11.11 or 11.23 yours looks pretty decent size.

Are you sure you do not have any files you can delete?
In the past I found some core files
from \ directory do the following:
find . -name \core -print
of course this will find cores on your whole system. But maybe some of these are somwere on root.

I have also found some in.... I thin it was /etc/opt/resmon.... not sure it was a long time ago

But from /etc you can do this
# find /etc -size +1800 -exec ls -l {} \;
you can play around and make 1800 bigger or smaller depanding on size you are looking for. Do any of these files look like they do not belong?
maybe do this same command from /dev
maybe you can find something that can be deleted. maybe someone with root permissions accidently put something in /etc /bin or maybe even /sbin

Re: Root at 97%.

John Jimenez — Thu, 20 Jan 2011 00:46:42 GMT

Also forgot to state the obvious but if you do an ll from / are there any files there that do not belong or maybe a directory but not a file systems? Sometimes someone with root permissions might make a directory under /

Re: Root at 97%.

Bill Hassell — Thu, 20 Jan 2011 02:42:11 GMT

Start by finding the big directories. Looking for files is not the right way to approach the problem. Use this command:

du -kx / | sort -rn | head -20

The two largest directories are /etc and /sbin. Anything else (like /dev or /specialApp) is wrong and needs to be fixed. If /dev/ is large, then there is a garbage file (usually a bad backup command) that must be removed:

find /dev -type f

Remove anything that find reports, especially things like /dev/rmt/om

Paste a copy od the du command here and we can find where the problem is located.

Re: Root at 97%.

Manix — Thu, 20 Jan 2011 09:50:06 GMT

try this as well ..

# find / -size +10000c -xdev -exec ll {} \; | sort -rn -k 5

Re: Root at 97%.

Jose Mosquera — Thu, 20 Jan 2011 12:05:46 GMT

Hi,

If by chance you're working with databases in raw mode, somebody may mistakenly have created a formated device occupying file system space into /dev directory.

Rgds.

Re: Root at 97%.

J.Kimbrell — Thu, 20 Jan 2011 12:44:29 GMT

Hello All, thanks for the replies. Below you'll find the information you requested.

1. It is 11.11

these are the only core files I found

# find . -name \core -print
./opt/omni/gui/help/C/help_topics/core
./var/adm/syslog/core
./var/sam/core

# du -xk /|sort -nrk1,1|more
1130616 /
536928 /cdrom/DataProtector6.0
536928 /cdrom
283632 /mnt
283440 /mnt/dd/ptc
283440 /mnt/dd
202896 /etc
201536 /cdrom/DataProtector6.0/core6.0patcheshpux
187288 /cdrom/DataProtector6.0/Cellconsolepatch6.0hpuxpatch

# ll
total 816
-r--r--r-- 1 bin bin 994 Nov 11 2009 .profile
dr-x------ 3 root sys 96 Jun 13 2008 .secure
-rw------- 1 root sys 0 May 11 2010 .sh_history
drwxr-xr-x 2 root sys 96 Apr 30 2008 .ssh
drwxr-xr-x 6 root sys 96 Apr 30 2008 .sw
lr-xr-xr-t 1 root sys 8 Apr 30 2008 bin -> /usr/bin
drwxr-xr-x 3 root sys 96 Nov 19 2008 cdrom
dr-xr-xr-x 15 bin bin 8192 Jan 19 15:18 dev
-rw-r--r-- 1 root sys 0 May 30 2008 devnull
dr-xr-xr-x 30 bin bin 8192 Jan 19 15:19 etc
drwxr-xr-x 9 root root 8192 Jan 19 11:50 home
lr-xr-xr-t 1 root sys 8 Apr 30 2008 lib -> /usr/lib
drwxr-xr-x 2 root root 96 Apr 30 2008 lost+found
-rw-r--r-- 1 root sys 12252 May 27 2010 mbox
drwxrwxrwx 5 root sys 8192 May 11 2010 mnt
dr-xr-xr-x 1 root root 1 Jan 19 15:16 net
-rw-rw-rw- 1 root root 2691 Nov 12 2009 null
dr-xr-xr-x 80 bin bin 8192 May 30 2008 opt
-rw-rw-rw- 1 root sys 146959 Jan 19 12:51 out.txt
-rw-rw-rw- 1 root sys 146959 Jan 19 12:52 out1.txt
dr-xr-xr-x 14 bin bin 8192 Nov 9 2009 sbin
drwxr-xr-x 2 root sys 8192 Sep 16 2009 searchdir
dr-xr-xr-x 7 bin bin 1024 Jan 19 15:15 stand
drwxrwxrwt 5 root root 8192 Jan 20 07:39 tmp
-rw-r--r-- 1 root sys 0 Sep 22 20
dr-xr-xr-x 24 bin bin 8192 May 11 2010 usr
dr-xr-xr-x 25 bin bin 8192 May 11 2010 var

# du -kx / | sort -rn | head -20
1130616 /
536928 /cdrom/DataProtector6.0
536928 /cdrom
283632 /mnt
283440 /mnt/dd/ptc
283440 /mnt/dd
202896 /etc
201536 /cdrom/DataProtector6.0/core6.0patcheshpux
187288 /cdrom/DataProtector6.0/Cellconsolepatch6.0hpuxpatch
172208 /etc/vx
107776 /cdrom/DataProtector6.0/mediaagent6.0hpuxpatch
105608 /etc/vx/type
97288 /sbin
65312 /etc/vx/static.d
62336 /etc/vx/static.d/build
42408 /etc/vx/type/static
27992 /etc/vx/type/gen
27120 /cdrom/DataProtector6.0/diskagent6.0hpuxpatch
26632 /etc/opt
21344 /etc/vx/type/raid5

Re: Root at 97%.

Hakki Aydin Ucar — Thu, 20 Jan 2011 12:53:47 GMT

Any CD mounted in your system now ? if not you can clear under /cdrom , apparently seemed lots of files there ;
for example DP6 source files not needed if you installed already

Re: Root at 97%.

J.Kimbrell — Thu, 20 Jan 2011 14:45:13 GMT

Thanks for the replies and suggestions. What I ended up doing, for contingency was to move the patches that were in the mounted /cdrom dir to another volume then tar'g them up. All is well.
Regards,
J.

Re: Root at 97%.

Bill Hassell — Thu, 20 Jan 2011 15:21:16 GMT

> # du -xk /|sort -nrk1,1|more
> 1130616 /
> 536928 /cdrom/DataProtector6.0
> 536928 /cdrom
> 283632 /mnt
> 283440 /mnt/dd/ptc
> 283440 /mnt/dd
> 202896 /etc

MAJOR errors here! The du -x option looks only at the / filesystem, so /cdrom is not a mountpoint for a CD, it is apparently a copy of a DataProtector CD stored in the / filesystem. Blow it away. You'll get 536 MB back immediately. It is not needed in / and should never have been put there. This is what /tmp and /var are for.

The second mistake is /mnt which has something that should not be in /...move it to /var or some other location and optionally put in a symlink to keep the functionality (if any). Personally, I would just move /mnt to /var and wait for something to break. That way you'll know what is there and ask why it is in root.

The third problem is /etc: 200 MB is too large for /etc unless you have hundreds of LUNs with lots of alternate paths. /etc/vx seems to verify this, but post the results of:

du -kx /etc | sort -rnk1 | head -20

Then also look in /etc for the bigger files:

ll /etc | sort -rnk5 | head -20

Post the results to see what is occupying space in the /etc directory.

Core files...
Before you remove the core files, find out what program created them like this:

file /opt/omni/gui/help/C/help_topics/core
file /var/adm/syslog/core
file /var/sam/core

This will tell you the name of the failing program. Look for patches or contact the owner of the program to get the problem fixed. Then you can blow them away.

/ directory:
Lots of mistakes here. They aren't occupying a lot of space but do indicate that root users are making mistakes and not repairing the problem:

> -r--r--r-- 1 bin bin 994 Nov 11 2009 .profile
> -rw------- 1 root sys 0 May 11 2010 .sh_history
> drwxr-xr-x 2 root sys 96 Apr 30 2008 .ssh
> drwxr-xr-x 6 root sys 96 Apr 30 2008 .sw
> -rw-r--r-- 1 root sys 12252 May 27 2010 mbox

These indicate that root's home directory is /, the worst possible location, but unfortunately the default for many Unix systems. Root's home should be in /root and all the root 'droppings' (files, directories) moved there. Then change /etc/passwd to reflect root's new home.

More notes:
> -rw-r--r-- 1 root sys 0 May 30 2008 devnull
> -rw-rw-rw- 1 root root 2691 Nov 12 2009 null

These are spelling errors. A root user made the mistake and used /null and /devnull when it should have been /dev/null. Blow them away.

-rw-rw-rw- 1 root sys 146959 Jan 19 12:51 out.txt
-rw-rw-rw- 1 root sys 146959 Jan 19 12:52 out1.txt

Junk files that should not be in /. Probably because root said to create these files with no path (bad habit. Should have been /tmp/out.txt. Moving root's home will mitigate the mistake but root users need to manage their junk files.

-rw-r--r-- 1 root sys 0 Sep 22 20

This is a filename with special characters in it. Use the command:

ll -b /

and you'll see the non-display characters. Needs to be removed.

Re: Root at 97%.

John Jimenez — Fri, 21 Jan 2011 18:05:31 GMT

Also no big deal but there was some *.txt in / out.txt etc..... do a more on those files, if not important move them or delete them, root is no place for text files.

Re: Root at 97%.

John Jimenez — Fri, 21 Jan 2011 18:15:04 GMT

Wow Bill took a lot of time and found many other things. He is right and when you have time you need to investigate those one at a time.

Re: Root at 97%.

J.Kimbrell — Fri, 21 Jan 2011 19:14:46 GMT

Thx All !