topic Re: root file system at 99% in Operating System - HP-UX

root file system at 99%

NDO — Mon, 30 Jan 2012 10:15:24 GMT

I know that are any posts regarding this issue, but I dont seem to get it right. I have the "/" at 99% I quite dont understand why. See an output of ls -lrt:

#bdf
Filesystem          kbytes    used   avail %used Mounted on
/dev/vg00/lvol3    1048576 1039800    8712   99% /
/dev/vg00/lvol1    1835008 478928 1345552   26% /stand
/dev/vg00/lvol8    34078720 25387536 8628264   75% /var
/dev/vg00/lvol7    5373952 2880024 2474456   54% /usr
/dev/vg00/lvol4    5242880 4217536 1017472   81% /tmp

#pwd
/
itc01[486]/ #ls -lrt
total 528
drwxr-xr-x   2 root       root            96 Sep 15 2010 lost+found
lr-xr-xr-x   1 bin        bin              8 Sep 15 2010 bin -> /usr/bin
lr-xr-xr-x   1 bin        bin              8 Sep 15 2010 lib -> /usr/lib
dr-xr-xr-x 22 bin        bin           8192 Sep 15 2010 usr
-r--r--r--   1 bin        bin           1039 Sep 23 2010 .profile
-rw-rw-rw-   1 root       sys             36 Sep 24 2010 .rhosts
drwxr-xr-x   6 root       sys           8192 Sep 24 2010 .sw
dr-xr-xr-x 28 bin        bin           8192 Sep 24 2010 var
-rw-r--r--   1 root       sys              9 Sep 28 2010 .firefox-license
drwx------   3 root       sys             96 Sep 28 2010 .mozilla
drwxrwxrwx   3 root       sys             96 Sep 29 2010 ora
drwxrwxrwx   3 root       sys             96 Sep 29 2010 data
drwxr-xr-x 10 root       root          8192 Oct 15 2010 home
srwx------   1 root       sys              0 Oct 18 2010 fsdaemonSocket4090
srwx------   1 root       sys              0 Oct 18 2010 fsdaemonSocket4611
-rw-r--r--   1 root       sys          13987 Oct 20 2010 sanlun.txt
drwxr-xr-x   2 root       sys             96 Nov 12 2010 interconnect_processed
-rw-r--r--   1 root       sys              8 Nov 18 2010 mapfile
dr-xr-xr-x 15 bin        bin           8192 Feb 7 2011 sbin
-rw-r--r--   1 root       sys           8816 Mar 4 2011 RAPIN
drwx------   2 root       sys           8192 Mar 8 2011 .ssh
drwxr-xr-x   9 root       sys           8192 Mar 17 2011 bmdtmp
-rw-------   1 root       sys           1212 Mar 18 2011 .sh_history
dr-xr-xr-x 83 bin        bin           8192 May 17 2011 opt
-rw-r--r--   1 root       sys            442 Jul 26 2011 tmpFile.1133
-r--r--r--   1 root       sys           5320 Oct 11 17:25 q4rc.pl
-r--r--r--   1 root       sys           5320 Oct 11 17:29 .q4rc.pl
-rw-r--r--   1 ictprd     ict           6248 Oct 15 23:59 Oct11
dr-xr-xr-x   1 root       root             1 Nov 9 11:52 net
dr-xr-xr-x   9 bin        bin           8192 Nov 9 11:52 stand
dr-xr-xr-x 35 bin        bin           8192 Nov 9 11:53 dev
-rw-r--r--   1 ictprd     ict          48564 Nov 30 23:59 Nov11
dr-xr-xr-x 37 bin        bin           8192 Dec 9 08:50 etc
drwxrwxrwt 13 root       root          8192 Jan 29 09:00 tmp
-rw-r--r--   1 root       sys           1302 Jan 30 12:20 diskaldata
-rw-------   1 root       sys           6970 Jan 30 12:25 history
itc01[487]/ #

Pls can you help?

Re: root file system at 99%

Dennis Handly — Mon, 30 Jan 2012 10:24:34 GMT

>I have the "/" at 99% I quite don't understand why.

What do you not understand? If it says you are using 99%, you are. If you can't add up the file space, you have removed a file that is still open.

What does this show: du -kxs /ora /data /interconnect_processed /sbin /bmdtmp /opt /dev /.mozilla /home

If these take a lot of space, they shouldn't be in /:

drwxrwxrwx 3 root sys 96 Sep 29 2010 ora
drwxrwxrwx 3 root sys 96 Sep 29 2010 data

Re: root file system at 99%

NDO — Mon, 30 Jan 2012 11:57:51 GMT

thank you for your reply, I meant that I could not see anything that was big enough, because normally that is under 50% figure.

But, the outputs requested by you:

#du -kxs /ora
16 /ora

#du -kxs /data
851336 /data

#du -kxs /interconnect_processed
0 /interconnect_processed

#du -kxs /sbin
103136 /sbin

#du -kxs /bmdtmp
13224 /bmdtmp

#du -kxs /opt
14783248 /opt

#du -kxs /dev
80 /dev

#du -kxs /home
183880 /home

Re: root file system at 99%

Alexander Ketshvili — Mon, 30 Jan 2012 12:39:31 GMT

Try to find files with big sizes

ex:

find / -size +1000000000c -exec ls -l {} \;

files>1gb

Re: root file system at 99%

NDO — Mon, 30 Jan 2012 13:01:43 GMT

The files that found with the comand "find / -size +1000000000c -exec ls -l {} \;" are all outside the root (/).

The files are in /var, /tmp and other file systems, not in the root:

find / -size +1000000000c -exec ls -l {} \; | more
-rw-r--r--   1 root       sys        1047341056 Oct 15 2010 /tmp/fr/patch/hpux__11.31_10140125.tar
-rw-r-----   1 oracle     oinstall   1049632768 Jan 30 15:02 /var/opt/oracle/ICTTST/system01.dbf
-rw-r-----   1 oracle     oinstall   1049632768 Jan 30 15:02 /var/opt/oracle/ICTTST/undotbs01.dbf
-rw-r-----   1 oracle     oinstall   1049632768 Jan 28 06:00 /var/opt/oracle/ICTTST/temp01.dbf
-rw-r-----   1 oracle     oinstall   1049632768 Jan 23 22:30 /var/opt/oracle/ICTTST/ict_summary_data01.dbf
-rw-r-----   1 oracle     oinstall   1049632768 Jan 23 22:30 /var/opt/oracle/ICTTST/ict_work_data01.dbf
-rw-r-----   1 oracle     oinstall   2098208768 Jan 23 22:30 /var/opt/oracle/ICTTST/ict_cdr_data01.dbf
-rw-r-----   1 oracle     oinstall   2260738048 Jan 23 22:30 /var/opt/oracle/ICTTST/ict_cdr_index01.dbf
-rw-r-----   1 oracle     oinstall   1049632768 Jan 23 22:30 /var/opt/oracle/ICTTST/cdr_restored_data01.dbf
-rw-r-----   1 oracle     oinstall   2098208768 Jan 28 03:03 /ora/ICTPRD/data/ict_error_data01.dbf

Re: root file system at 99%

Alexander Ketshvili — Mon, 30 Jan 2012 13:44:05 GMT

Did you try to find the files for ex 500mb?
May be you have audit running on the root. Check /etc/rc.config.d/auditing file. There you can find path to audit folders

Re: root file system at 99%

Torsten. — Mon, 30 Jan 2012 13:57:15 GMT

#bdf
Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol3 1048576 1039800 8712 99% /
/dev/vg00/lvol1 1835008 478928 1345552 26% /stand
/dev/vg00/lvol8 34078720 25387536 8628264 75% /var
/dev/vg00/lvol7 5373952 2880024 2474456 54% /usr
/dev/vg00/lvol4 5242880 4217536 1017472 81% /tmp

I really wonder where is lvol5 and lvol6.

lvol6 usually is used for /opt, but you obviously has /opt directly in /.

Could you please post your

# vgdisplay -v vg00

Re: root file system at 99%

NDO — Mon, 30 Jan 2012 14:18:19 GMT

this is the output of vgdisplay vg00:

#vgdisplay -v vg00
--- Volume groups ---
VG Name                     /dev/vg00
VG Write Access             read/write
VG Status                   available
Max LV                      255
Cur LV                      9
Open LV                     9
Max PV                      16
Cur PV                      1
Act PV                      1
Max PE per PV               4353
VGDA                        2
PE Size (Mbytes)            32
Total PE                    4343
Alloc PE                    3916
Free PE                     427
Total PVG                   0
Total Spare PVs             0
Total Spare PVs in use      0
VG Version                  1.0
VG Max Size                 2228736m
VG Max Extents              69648

   --- Logical volumes ---
   LV Name                     /dev/vg00/lvol1
   LV Status                   available/syncd
   LV Size (Mbytes)            1792
   Current LE                  56
   Allocated PE                56
   Used PV                     1

   LV Name                     /dev/vg00/lvol2
   LV Status                   available/syncd
   LV Size (Mbytes)            32768
   Current LE                  1024
   Allocated PE                1024
   Used PV                     1

   LV Name                     /dev/vg00/lvol3
   LV Status                   available/syncd
   LV Size (Mbytes)            1024
   Current LE                  32
   Allocated PE                32
   Used PV                     1

   LV Name                     /dev/vg00/lvol4
   LV Status                   available/syncd
   LV Size (Mbytes)            5120
   Current LE                  160
   Allocated PE                160
   Used PV                     1

   LV Name                     /dev/vg00/lvol5
   LV Status                   available/syncd
   LV Size (Mbytes)            5120
   Current LE                  160
   Allocated PE                160
   Used PV                     1

   LV Name                     /dev/vg00/lvol6
   LV Status                   available/syncd
   LV Size (Mbytes)            24576
   Current LE                  768
   Allocated PE                768
   Used PV                     1

   LV Name                     /dev/vg00/lvol7
   LV Status                   available/syncd
   LV Size (Mbytes)            5248
   Current LE                  164
   Allocated PE                164
   Used PV                     1

   LV Name                     /dev/vg00/lvol8
   LV Status                   available/syncd
   LV Size (Mbytes)            33280
   Current LE                  1040
   Allocated PE                1040
   Used PV                     1

   LV Name                     /dev/vg00/lvol9
   LV Status                   available/syncd
   LV Size (Mbytes)            16384
   Current LE                  512
   Allocated PE                512
   Used PV                     1

   --- Physical volumes ---
   PV Name                     /dev/disk/disk1_p2
   PV Status                   available
   Total PE                    4343
   Free PE                     427
   Autoswitch                  On
   Proactive Polling           On

and the bdf for all O.S. related F.S.:

bdf | grep vg00
/dev/vg00/lvol3    1048576 1039816    8696   99% /
/dev/vg00/lvol1    1835008 478928 1345552   26% /stand
/dev/vg00/lvol8    34078720 24313416 9694024   71% /var
/dev/vg00/lvol7    5373952 2880024 2474456   54% /usr
/dev/vg00/lvol4    5242880 4217536 1017472   81% /tmp
/dev/vg00/lvol6    25165824 14839600 10245568   59% /opt
/dev/vg00/lvol9    16777216 8138509 8098824   50% /opt/oracle
/dev/vg00/lvol5    5242880 190080 5013352    4% /home

Re: root file system at 99%

NDO — Mon, 30 Jan 2012 14:24:13 GMT

auditing is not enable

Re: root file system at 99%

Torsten. — Mon, 30 Jan 2012 14:27:07 GMT

So you just did not post the complete "bdf" output in the beginning ... confusing!

Question is still, why /data is directly in root???

Re: root file system at 99%

NDO — Mon, 30 Jan 2012 14:37:26 GMT

This is the complete bdf:

#bdf
Filesystem          kbytes    used   avail %used Mounted on
/dev/vg00/lvol3    1048576 1039808    8704   99% /
/dev/vg00/lvol1    1835008 478928 1345552   26% /stand
/dev/vg00/lvol8    34078720 24313432 9694008   71% /var
/dev/vg00/lvol7    5373952 2880024 2474456   54% /usr
/dev/vg00/lvol4    5242880 4217536 1017472   81% /tmp
/dev/vg05/lvol1    18874368 14569796 4035542   78% /ora/ICTPRD/undotbs
/dev/vg07/lvol1    18874368 5810065 12247789   32% /ora/ICTPRD/temp
/dev/vg06/lvol1    3145728 1573069 1474375   52% /ora/ICTPRD/system
/dev/vg15/lvol1    3145728 1000910 2010774   33% /ora/ICTPRD/redo4
/dev/vg14/lvol1    3145728 1000910 2010774   33% /ora/ICTPRD/redo3
/dev/vg13/lvol1    3145728 1000910 2010774   33% /ora/ICTPRD/redo2
/dev/vg12/lvol1    3145728 1000910 2010774   33% /ora/ICTPRD/redo1
/dev/vg09/lvol1    153092096 136319471 15724390   90% /ora/ICTPRD/index
/dev/vg16/lvol1    67108864 57406650 9095826   86% /ora/ICTPRD/fm
/dev/vg11/lvol1    214958080 172189194 40095881   81% /ora/ICTPRD/edr_index
/dev/vg10/lvol1    450887680 393543345 53760338   88% /ora/ICTPRD/edr_data
/dev/vg08/lvol1    319815680 272692657 44177872   86% /ora/ICTPRD/data
/dev/vg00/lvol6    25165824 14839600 10245568   59% /opt
/dev/vg00/lvol9    16777216 8138509 8098824   50% /opt/oracle
/dev/vg00/lvol5    5242880 190080 5013352    4% /home
/dev/vg03/lvol1    29360128 6900586 21056081   25% /data/ICTPRD/ict_archive
/dev/vg02/lvol1    56623104 534693 52582957    1% /data/ICTPRD/dwf
/dev/vg04/lvol1    6291456 366294 5554890    6% /data/ICTPRD/bulkload
/dev/vg17/lvol1    115343360 55718343 55900917   50% /data/ICTPRD/bmd1
itc01[633]/ #

/data is a separete file system as you can see from the bdf output

Re: root file system at 99%

Torsten. — Mon, 30 Jan 2012 14:51:00 GMT

Anything else that you hide?

Re: root file system at 99%

NDO — Mon, 30 Jan 2012 14:57:11 GMT

Hi Torsten!

:) I am not hiding anything , but if you want more information I can provide it to you, I was just trying to abbreviate, sorry I was wrong.

The "/" file system used to be 69% at all time, so I come back from leave today and I found that someone has messed up with the system

Re: root file system at 99%

Torsten. — Mon, 30 Jan 2012 18:36:33 GMT

OK, but it was confusing ...

You need to look for large files in lvol3, not in other lvols (e.g. using "du").

If there is a single large files, it is likely much smaller than 1GB, because your / is 1GB only.

So a file in question is about 300MB ?

Re: root file system at 99%

Dennis Handly — Mon, 30 Jan 2012 20:45:54 GMT

>I meant that I could not see anything that was big enough, because normally that is under 50% figure.

And I answered that question. You may have removed an open file so it is now invisible. You need to download and run lsof to find it.

And what does this show: du -kxs /

>Try to find files with big sizes: find / -size +1000000000c -exec ls -l {} +

>The files are in /var, /tmp and other file systems, not in the root:

You need to add -xdev so it only looks at /.

>but you obviously have /opt directly in /.

Yes, I also mistakenly trusted in the bdf output. ;-)

Re: root file system at 99%

Bill Hassell — Tue, 31 Jan 2012 01:33:45 GMT

Rather than looking for big files in "/" you should be looking for big directories. A directory can be large because it has one giant sized file, or it could be big because it has 50,000 junk files in it, none of which are very big. USe du to provide a useful listing:

du -kx / | sort -rn | head -20

The two largest directories are /sbin and /etc. Anything else at the top of the list is a mistake. A common mistake is /dev -- it will always be less than 100k or so. If it is megabytes, you have a bad file in it. There are *NO* ordinary files in /dev:

find /dev -type f -exec ls -l {} \*

The above should produce nothing. Remove (or move if it is a useful file) the ordinary files in /dev.

Re: root file system at 99%

NDO — Tue, 31 Jan 2012 07:21:44 GMT

Hi

I found the problem, indeed there were some 300Mb in excess in a directory under /data/ICTPRD. Thank you very much

Re: root file system at 99%

Torsten. — Tue, 31 Jan 2012 07:34:37 GMT

So it was outside of these mount points?

dev/vg03/lvol1 29360128 6900586 21056081 25% /data/ICTPRD/ict_archive
/dev/vg02/lvol1 56623104 534693 52582957 1% /data/ICTPRD/dwf
/dev/vg04/lvol1 6291456 366294 5554890 6% /data/ICTPRD/bulkload
/dev/vg17/lvol1 115343360 55718343 55900917 50% /data/ICTPRD/bmd1

Ok, then it was in /.

Re: root file system at 99%

Dennis Handly — Tue, 31 Jan 2012 08:31:04 GMT

>So it was outside of these mount points? Ok, then it was in /.

So that this was valid and not a good idea:

#du -kxs /data
851336 /data

Re: root file system at 99%

NDO — Tue, 31 Jan 2012 12:53:24 GMT

Hi!

It was a directory inside /data/ICTPRD, it was only 300Mb

I was on leave, them when I came back, I found this issue, but its now solved, thank you for your assistance