topic Re: Root directory full after crash in Operating System - HP-UX

Root directory full after crash

Martin Talero — Mon, 10 May 2010 15:51:45 GMT

Hi,
I have a root directory full after two crash dump on HP-UX 11.31.

Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol3 1048576 1048504 72 100% /
I can't determine where are the file created on root directory (/). I already delete the crash dump allocated on /var/adm/crash after send to HP.

Could you help me please?

Re: Root directory full after crash

Pete Randall — Mon, 10 May 2010 15:58:14 GMT

Run find:

find / -name core

then remove.

Pete

Re: Root directory full after crash

Martin Talero — Mon, 10 May 2010 16:03:33 GMT

Pete thanks for your response, but I has been already deleted the crash dump allocated on /var/adm/crash after send to HP.

And there are no more core.

Re: Root directory full after crash

Don Morris_1 — Mon, 10 May 2010 16:12:54 GMT

The kernel configuration at the time of the crash is typically saved to a "crashconfig" configuration under /stand. If you don't have much space for extra configurations, this could be your issue.

`kconfig` (Lists configurations -- see if there is a crashconfig listed).

`du /stand/crashconfig` (See how much space it is taking if present).

Since you already sent the crash off to HP, and especially if the crashconfig is simply your current configuration anyway (you've made no changes to tunables, patches, etc.) and you don't need the extra copy:

`kconfig -d crashconfig` (Will delete said backup crash configuration).

Re: Root directory full after crash

Steven E. Protter — Mon, 10 May 2010 16:18:31 GMT

Shalom Martin,

/var/adm/crash is normally mounted on the /var file system and clearing the crash dump has no impact on / the root file system.

cd /

du -k | sort -rn | more

Find the big directories, make sure they are not part of mounted file systems (bdf) and find the place to look for stuff to delete.

Finding and locating core files is very likely to help with / 100% condition.

SEP

Re: Root directory full after crash

John Guster — Mon, 10 May 2010 17:45:38 GMT

can you post the output of "ll /"?

Re: Root directory full after crash

Suraj K Sankari — Mon, 10 May 2010 17:55:32 GMT

Hi,

some open file will be there use lsof command.

If possible take one more reboot.

Suraj

Re: Root directory full after crash

nazishjamal — Mon, 10 May 2010 18:11:15 GMT

check this

# du -xk / | sort -rn | more

also look for

/root/.local/share/Trash/trash_files

to clean up all these filess

Re: Root directory full after crash

Martin Talero — Mon, 10 May 2010 18:50:07 GMT

Hi,
Thanks again for yours response, here is more iformation about my situation:

SARC #kconfig
Configuration Title
backup Automatic Backup
crashconfig
last_install Created by last OS install
SARC #du /stand/crashconfig
55648 /stand/crashconfig/mod
208 /stand/crashconfig/krs
0 /stand/crashconfig/bootfs/stand/current/krs
16 /stand/crashconfig/bootfs/stand/current/mod
16 /stand/crashconfig/bootfs/stand/current
16 /stand/crashconfig/bootfs/stand
16 /stand/crashconfig/bootfs
297920 /stand/crashconfig
SARC #bdf
Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol3 1048576 1048504 72 100% /
/dev/vg00/lvol1 1835008 320104 1503152 18% /stand
/dev/vg00/lvol8 10485760 1481936 8936680 14% /var
/dev/vg00/lvol7 8388608 2847888 5497464 34% /usr
/dev/vg01/u01 9437184 5614072 3588618 61% /u01
/dev/vg00/lvol4 524288 314288 208912 60% /tmp
/dev/vg00/lvol6 10485760 5212120 5232512 50% /opt
/dev/vg04/monito 20971520 11076530 9299343 54% /monitor
/dev/vg04/m09 65961984 64895315 1000007 98% /m09
/dev/vg04/m08 26214400 24506442 1601217 94% /m08
/dev/vg04/m07 52428800 41295301 10437661 80% /m07
/dev/vg04/m06 15728640 15093457 595490 96% /m06
/dev/vg02/m05 10485760 7699667 2611969 75% /m05
/dev/vg03/m04 11534336 9321956 2074113 82% /m04
/dev/vg02/m03 11534336 10538468 933633 92% /m03
/dev/vg02/m02 10485760 5262547 4896769 52% /m02
/dev/vg02/m01 10485760 8851692 1531945 85% /m01
/dev/vg00/lvol5 3145728 113544 3008616 4% /home
DevFS 11 11 0 100% /dev/deviceFileSystem
/dev/vg05/backups 359940096 307437368 52095072 86% /backups
/dev/vg06/Compartido
123731968 103602192 18871672 85% /Compartido
SARC #du -k | sort -rn | more
625142805 .
102160 ./sbin
37688 ./sbin/fs
27312 ./sbin/fs/vxfs
23960 ./etc/opt/resmon/lib
5992 ./etc/opt/resmon/lbin
5832 ./etc/opt/resmon/lib/hpux64
5768 ./sbin/fs/hfs
3016 ./.mozilla/firefox
3016 ./.mozilla
3008 ./.mozilla/firefox/cj1392w9.default
1504 ./etc/lvmconf
SARC #ll /
total 1510
-rw-rw-rw- 1 root sys 9 Jan 18 11:08 .firefox-license
-rw-rw-rw- 1 root sys 646765 Jan 18 12:34 .fonts.cache-1
drwx------ 3 root sys 96 Jan 18 11:08 .mozilla
-r--r--r-- 1 bin bin 992 Jan 22 10:03 .profile
-rw------- 1 root sys 1024 Jan 16 2009 .rnd
-rw------- 1 root sys 1100 Nov 6 2009 .sh_history
drwx------ 2 root sys 96 Jan 14 15:13 .ssh
drwx------ 6 root sys 8192 Dec 4 12:03 .sw
drwxrwxrwx 2 root sys 96 Nov 6 2009 CD
drwxr-xr-x 3 oracle dba 96 Mar 17 14:34 Compartido
drwxr-xr-x 2 root sys 96 Dec 4 11:55 SD_CDROM
drwxr-xr-x 7 oracle dba 8192 May 5 17:18 backups
lr-xr-xr-x 1 bin bin 8 Jan 16 2009 bin -> /usr/bin
drwxrwxrwx 2 root sys 96 Nov 9 2009 cdrom
dr-xr-xr-x 30 bin bin 8192 May 4 12:12 dev
drwxrwxrwx 2 root sys 96 Nov 6 2009 dvdrom
dr-xr-xr-x 36 bin bin 8192 May 5 16:44 etc
drwxr-xr-x 10 root root 8192 Jan 14 15:11 home
lr-xr-xr-x 1 bin bin 8 Jan 16 2009 lib -> /usr/lib
drwxr-xr-x 2 root root 96 Jan 16 2009 lost+found
drwxr-xr-x 4 oracle dba 96 Jan 16 11:28 m01
drwxr-xr-x 4 oracle dba 96 Jan 16 11:59 m02
drwxr-xr-x 4 oracle dba 96 Jan 16 11:31 m03
drwxr-xr-x 4 oracle dba 96 Jan 16 11:31 m04
drwxr-xr-x 4 oracle dba 96 Jan 16 11:32 m05
drwxr-xr-x 4 oracle dba 96 Jan 16 11:34 m06
drwxr-xr-x 4 oracle dba 1024 Apr 16 14:58 m07
drwxr-xr-x 4 oracle dba 96 Jan 16 11:32 m08
drwxr-xr-x 4 oracle dba 96 Jan 16 11:33 m09
-rw-rw-rw- 1 root sys 0 Mar 17 13:26 mapfile
drwxr-xr-x 12 monitor dba 2048 May 4 07:22 monitor
dr-xr-xr-x 1 root root 1 May 4 08:59 net
dr-xr-xr-x 78 bin bin 8192 Feb 13 14:06 opt
dr-xr-xr-x 15 bin bin 8192 Feb 13 14:26 sbin
dr-xr-xr-x 8 bin bin 8192 May 10 14:18 stand
drwxrwxrwt 9 root root 8192 May 10 14:18 tmp
drwxr-xr-x 4 oracle dba 96 Feb 18 07:58 u01
dr-xr-xr-x 22 bin bin 8192 Jan 16 2009 usr
dr-xr-xr-x 26 bin bin 8192 Oct 23 2009 var

Re: Root directory full after crash

Suraj K Sankari — Tue, 11 May 2010 04:24:19 GMT

Hi,
I can see there is 2 directory CD and SD_CDROM check this out I think some thing is there in these directories because I can see there is no mount point in bdf output.

Post this out put

find / -size +10000 â exec ls â lrt {} \; | sort â rn | more

Suraj

Re: Root directory full after crash

atul2701 — Tue, 11 May 2010 06:39:14 GMT

Hi Pls provide the output of below commnd

find / -xdev -size +10000 -exec ls -alrt {} \; | sort -rn

Atul

Re: Root directory full after crash

rariasn — Tue, 11 May 2010 11:25:45 GMT

Hi,

find / -xdev -type f -exec ll {} \; | awk '
{ printf "%9d %3s %2d %5s %s\n" , $5, $6 , $7, $8, $9 }'|sort -r -n|more

rgs

Re: Root directory full after crash

Martin Talero — Tue, 11 May 2010 13:05:38 GMT

CD and SD_CDROM directories where I mounted DVS before but now don't have nothing, only directories.

There are the result of commands:

SARC #find / -xdev -size +10000 -exec ls -alrt {} \; | sort -rn
-r-xr-xr-x 1 bin bin 16292880 Jun 13 2008 /etc/opt/resmon/lib/libxerces-c.so.23

SARC #find / -xdev -type f -exec ll {} \; | awk '
> { printf "%9d %3s %2d %5s %s\n" , $5, $6 , $7, $8, $9 }'|sort -r -n|more
16292880 Jun 13 2008 /etc/opt/resmon/lib/libxerces-c.so.23
4619600 Jun 13 2008 /etc/opt/resmon/lib/hpux64/libEMSHAProvider.so.1
4086524 Jun 13 2008 /etc/opt/resmon/lbin/mibmond
3428840 Jan 12 2007 /sbin/fs/vxfs/fsck
3336976 Jul 7 2008 /sbin/lvmcmd
3186716 Jan 12 2007 /sbin/fs/vxfs/fsck_logv9
3132340 Jul 7 2008 /sbin/lvmpcmd
2894224 Dec 14 2007 /sbin/fs/vxfs/mkfs
2823560 Jan 12 2007 /sbin/fs/vxfs/mount
2747356 Jan 12 2007 /sbin/fs/vxfs/extendfs
2673060 Jan 12 2007 /sbin/fs/vxfs/vxumount
2670972 Jan 12 2007 /sbin/fs/vxfs/vxenablef
2395404 Feb 15 2007 /sbin/kcboot
2384592 Feb 15 2007 /sbin/kcshutdown
1969596 Feb 15 2007 /sbin/init
1853656 Jul 7 2008 /sbin/lvm_wrapper
1708224 Jan 18 11:10 /.mozilla/firefox/cj1392w9.default/XPC.mfasl
1680264 Feb 15 2007 /sbin/fs/fsdaemon
1403704 Jun 18 2008 /sbin/sh
1280564 Jun 28 2008 /sbin/rmsf
1280564 Jun 28 2008 /sbin/mksf
1280564 Jun 28 2008 /sbin/lssf
1280564 Jun 28 2008 /sbin/insf
1246216 Feb 15 2007 /sbin/awk
1221300 May 23 2008 /sbin/pax
1140552 Jan 12 2007 /sbin/fs/vxfs/unsupported/metasave
1133808 Jun 28 2008 /sbin/ioscan
1112380 Jan 12 2007 /sbin/vxdump
1097776 Jan 12 2007 /sbin/fs/vxfs/fsdb
1092940 Jan 12 2007 /sbin/fs/vxfs/vxfsconvert
1027608 Feb 15 2007 /sbin/savecrash
1026272 May 15 2008 /sbin/frecover
971720 Jan 12 2007 /sbin/vxrestore
951132 Feb 15 2007 /sbin/dasetup
935276 Jun 28 2008 /sbin/ioinit

Thanks again.

Re: Root directory full after crash

R.O. — Tue, 11 May 2010 13:20:15 GMT

Hi,

Maybe a file under "/" was removed or renamed while it was being accesed by a process. In such cases, the space in the file system is not released until the process dies. To find it (if it is the case) try with:

# lsof +aL1 /

It will shows files deleted by still opened by the system.

Regards,

Re: Root directory full after crash

Martin Talero — Tue, 11 May 2010 19:00:39 GMT

Hi,
Do you know where can I download the lsof for hp-ux 11.31?

Re: Root directory full after crash

Robert-Jan Goossens — Tue, 11 May 2010 19:12:30 GMT

http://hpux.connect.org.uk/hppd/hpux/Sysadmin/lsof-4.83/

Re: Root directory full after crash

TwoProc — Tue, 11 May 2010 19:21:13 GMT

Look for a large file that happened on the day you crashed, or just before. This would eliminate a lot of the valid files you have in the binary areas that are valid, and not changed since 2007.

Let's say you crashed yesterday:

find / -type f -xdev -mtime -1 \
-size +10000 -exec ls -al {} \;

or you crashed today:
find / -type f -xdev -mtime 0 \
-size +10000 -exec ls -al {} \;

To narrow and expand the list, just play with the size number (removing/adding zeroes) and the =-mtime argument until you see the file you're looking for.