cancel
Showing results for 
Search instead for 
Did you mean: 

root file system at 99%

SOLVED
Go to solution
NDO
Super Advisor

root file system at 99%

Hi

 

I know that are any posts regarding this issue, but I dont seem to get it right. I have the "/" at 99% I quite dont understand why. See an output of ls -lrt:

 

#bdf
Filesystem          kbytes    used   avail %used Mounted on
/dev/vg00/lvol3    1048576 1039800    8712   99% /
/dev/vg00/lvol1    1835008  478928 1345552   26% /stand
/dev/vg00/lvol8    34078720 25387536 8628264   75% /var
/dev/vg00/lvol7    5373952 2880024 2474456   54% /usr
/dev/vg00/lvol4    5242880 4217536 1017472   81% /tmp

 

 #pwd
/
itc01[486]/ #ls -lrt
total 528
drwxr-xr-x   2 root       root            96 Sep 15  2010 lost+found
lr-xr-xr-x   1 bin        bin              8 Sep 15  2010 bin -> /usr/bin
lr-xr-xr-x   1 bin        bin              8 Sep 15  2010 lib -> /usr/lib
dr-xr-xr-x  22 bin        bin           8192 Sep 15  2010 usr
-r--r--r--   1 bin        bin           1039 Sep 23  2010 .profile
-rw-rw-rw-   1 root       sys             36 Sep 24  2010 .rhosts
drwxr-xr-x   6 root       sys           8192 Sep 24  2010 .sw
dr-xr-xr-x  28 bin        bin           8192 Sep 24  2010 var
-rw-r--r--   1 root       sys              9 Sep 28  2010 .firefox-license
drwx------   3 root       sys             96 Sep 28  2010 .mozilla
drwxrwxrwx   3 root       sys             96 Sep 29  2010 ora
drwxrwxrwx   3 root       sys             96 Sep 29  2010 data
drwxr-xr-x  10 root       root          8192 Oct 15  2010 home
srwx------   1 root       sys              0 Oct 18  2010 fsdaemonSocket4090
srwx------   1 root       sys              0 Oct 18  2010 fsdaemonSocket4611
-rw-r--r--   1 root       sys          13987 Oct 20  2010 sanlun.txt
drwxr-xr-x   2 root       sys             96 Nov 12  2010 interconnect_processed
-rw-r--r--   1 root       sys              8 Nov 18  2010 mapfile
dr-xr-xr-x  15 bin        bin           8192 Feb  7  2011 sbin
-rw-r--r--   1 root       sys           8816 Mar  4  2011 RAPIN
drwx------   2 root       sys           8192 Mar  8  2011 .ssh
drwxr-xr-x   9 root       sys           8192 Mar 17  2011 bmdtmp
-rw-------   1 root       sys           1212 Mar 18  2011 .sh_history
dr-xr-xr-x  83 bin        bin           8192 May 17  2011 opt
-rw-r--r--   1 root       sys            442 Jul 26  2011 tmpFile.1133
-r--r--r--   1 root       sys           5320 Oct 11 17:25 q4rc.pl
-r--r--r--   1 root       sys           5320 Oct 11 17:29 .q4rc.pl
-rw-r--r--   1 ictprd     ict           6248 Oct 15 23:59 Oct11
dr-xr-xr-x   1 root       root             1 Nov  9 11:52 net
dr-xr-xr-x   9 bin        bin           8192 Nov  9 11:52 stand
dr-xr-xr-x  35 bin        bin           8192 Nov  9 11:53 dev
-rw-r--r--   1 ictprd     ict          48564 Nov 30 23:59 Nov11
dr-xr-xr-x  37 bin        bin           8192 Dec  9 08:50 etc
drwxrwxrwt  13 root       root          8192 Jan 29 09:00 tmp
-rw-r--r--   1 root       sys           1302 Jan 30 12:20 diskaldata
-rw-------   1 root       sys           6970 Jan 30 12:25 history
itc01[487]/ #

 

Pls can you help?

 

 

19 REPLIES
Dennis Handly
Acclaimed Contributor

Re: root file system at 99%

>I have the "/" at 99% I quite don't understand why.

 

What do you not understand?  If it says you are using 99%, you are.  If you can't add up the file space, you have removed a file that is still open.

 

What does this show: du -kxs /ora /data /interconnect_processed /sbin /bmdtmp /opt /dev /.mozilla /home

 

If these take a lot of space, they shouldn't be in /:

drwxrwxrwx   3 root       sys             96 Sep 29  2010 ora
drwxrwxrwx   3 root       sys             96 Sep 29  2010 data

NDO
Super Advisor

Re: root file system at 99%

thank you for your reply, I meant that I could not see anything that was big enough, because normally that is under 50% figure.

 

But, the outputs requested by you:

#du -kxs /ora
16      /ora

 

#du -kxs /data
851336  /data

 

#du -kxs /interconnect_processed
0       /interconnect_processed

 

#du -kxs /sbin
103136  /sbin

 

#du -kxs /bmdtmp
13224   /bmdtmp

 

#du -kxs /opt
14783248        /opt

 

#du -kxs /dev
80      /dev

 

#du -kxs /home
183880  /home

 

 

 

Re: root file system at 99%

Try to find files with big sizes

ex:

 find / -size +1000000000c -exec ls -l {} \;

files>1gb 

NDO
Super Advisor

Re: root file system at 99%

The files that found with the comand "find / -size +1000000000c -exec ls -l {} \;" are all outside the root (/).

 

The files are in /var, /tmp and other file systems, not in the root:

 

find / -size +1000000000c -exec ls -l {} \; | more
-rw-r--r--   1 root       sys        1047341056 Oct 15  2010 /tmp/fr/patch/hpux__11.31_10140125.tar
-rw-r-----   1 oracle     oinstall   1049632768 Jan 30 15:02 /var/opt/oracle/ICTTST/system01.dbf
-rw-r-----   1 oracle     oinstall   1049632768 Jan 30 15:02 /var/opt/oracle/ICTTST/undotbs01.dbf
-rw-r-----   1 oracle     oinstall   1049632768 Jan 28 06:00 /var/opt/oracle/ICTTST/temp01.dbf
-rw-r-----   1 oracle     oinstall   1049632768 Jan 23 22:30 /var/opt/oracle/ICTTST/ict_summary_data01.dbf
-rw-r-----   1 oracle     oinstall   1049632768 Jan 23 22:30 /var/opt/oracle/ICTTST/ict_work_data01.dbf
-rw-r-----   1 oracle     oinstall   2098208768 Jan 23 22:30 /var/opt/oracle/ICTTST/ict_cdr_data01.dbf
-rw-r-----   1 oracle     oinstall   2260738048 Jan 23 22:30 /var/opt/oracle/ICTTST/ict_cdr_index01.dbf
-rw-r-----   1 oracle     oinstall   1049632768 Jan 23 22:30 /var/opt/oracle/ICTTST/cdr_restored_data01.dbf
-rw-r-----   1 oracle     oinstall   2098208768 Jan 28 03:03 /ora/ICTPRD/data/ict_error_data01.dbf

 

 

Re: root file system at 99%

Did you try to find the files for ex 500mb?
May be you have audit running on the root. Check /etc/rc.config.d/auditing file. There you can find path to audit folders
Torsten.
Acclaimed Contributor

Re: root file system at 99%

#bdf
Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol3 1048576 1039800 8712 99% /
/dev/vg00/lvol1 1835008 478928 1345552 26% /stand
/dev/vg00/lvol8 34078720 25387536 8628264 75% /var
/dev/vg00/lvol7 5373952 2880024 2474456 54% /usr
/dev/vg00/lvol4 5242880 4217536 1017472 81% /tmp

 

 

I really wonder where is lvol5 and lvol6.

 

lvol6 usually is used for /opt, but you obviously has /opt directly in /.

 

Could you please post your

 

# vgdisplay -v vg00


Hope this helps!
Regards
Torsten.

__________________________________________________
There are only 10 types of people in the world -
those who understand binary, and those who don't.

__________________________________________________
No support by private messages. Please ask the forum!

If you feel this was helpful please click the KUDOS! thumb below!   
NDO
Super Advisor

Re: root file system at 99%

Hi

 

this is the output of vgdisplay vg00:

 

#vgdisplay -v vg00
--- Volume groups ---
VG Name                     /dev/vg00
VG Write Access             read/write
VG Status                   available
Max LV                      255
Cur LV                      9
Open LV                     9
Max PV                      16
Cur PV                      1
Act PV                      1
Max PE per PV               4353
VGDA                        2
PE Size (Mbytes)            32
Total PE                    4343
Alloc PE                    3916
Free PE                     427
Total PVG                   0
Total Spare PVs             0
Total Spare PVs in use      0
VG Version                  1.0
VG Max Size                 2228736m
VG Max Extents              69648

   --- Logical volumes ---
   LV Name                     /dev/vg00/lvol1
   LV Status                   available/syncd
   LV Size (Mbytes)            1792
   Current LE                  56
   Allocated PE                56
   Used PV                     1

   LV Name                     /dev/vg00/lvol2
   LV Status                   available/syncd
   LV Size (Mbytes)            32768
   Current LE                  1024
   Allocated PE                1024
   Used PV                     1

   LV Name                     /dev/vg00/lvol3
   LV Status                   available/syncd
   LV Size (Mbytes)            1024
   Current LE                  32
   Allocated PE                32
   Used PV                     1

   LV Name                     /dev/vg00/lvol4
   LV Status                   available/syncd
   LV Size (Mbytes)            5120
   Current LE                  160
   Allocated PE                160
   Used PV                     1

   LV Name                     /dev/vg00/lvol5
   LV Status                   available/syncd
   LV Size (Mbytes)            5120
   Current LE                  160
   Allocated PE                160
   Used PV                     1

   LV Name                     /dev/vg00/lvol6
   LV Status                   available/syncd
   LV Size (Mbytes)            24576
   Current LE                  768
   Allocated PE                768
   Used PV                     1

   LV Name                     /dev/vg00/lvol7
   LV Status                   available/syncd
   LV Size (Mbytes)            5248
   Current LE                  164
   Allocated PE                164
   Used PV                     1

   LV Name                     /dev/vg00/lvol8
   LV Status                   available/syncd
   LV Size (Mbytes)            33280
   Current LE                  1040
   Allocated PE                1040
   Used PV                     1

   LV Name                     /dev/vg00/lvol9
   LV Status                   available/syncd
   LV Size (Mbytes)            16384
   Current LE                  512
   Allocated PE                512
   Used PV                     1


   --- Physical volumes ---
   PV Name                     /dev/disk/disk1_p2
   PV Status                   available
   Total PE                    4343
   Free PE                     427
   Autoswitch                  On
   Proactive Polling           On

 

and the  bdf for all O.S. related F.S.:

 

bdf | grep vg00
/dev/vg00/lvol3    1048576 1039816    8696   99% /
/dev/vg00/lvol1    1835008  478928 1345552   26% /stand
/dev/vg00/lvol8    34078720 24313416 9694024   71% /var
/dev/vg00/lvol7    5373952 2880024 2474456   54% /usr
/dev/vg00/lvol4    5242880 4217536 1017472   81% /tmp
/dev/vg00/lvol6    25165824 14839600 10245568   59% /opt
/dev/vg00/lvol9    16777216 8138509 8098824   50% /opt/oracle
/dev/vg00/lvol5    5242880  190080 5013352    4% /home


NDO
Super Advisor

Re: root file system at 99%

auditing is not enable
Torsten.
Acclaimed Contributor

Re: root file system at 99%

So you just did not post the complete "bdf" output in the beginning ... confusing!


Question is still, why /data is directly in root???

Hope this helps!
Regards
Torsten.

__________________________________________________
There are only 10 types of people in the world -
those who understand binary, and those who don't.

__________________________________________________
No support by private messages. Please ask the forum!

If you feel this was helpful please click the KUDOS! thumb below!   
NDO
Super Advisor

Re: root file system at 99%

Hi

 

This is the complete bdf:

 

#bdf
Filesystem          kbytes    used   avail %used Mounted on
/dev/vg00/lvol3    1048576 1039808    8704   99% /
/dev/vg00/lvol1    1835008  478928 1345552   26% /stand
/dev/vg00/lvol8    34078720 24313432 9694008   71% /var
/dev/vg00/lvol7    5373952 2880024 2474456   54% /usr
/dev/vg00/lvol4    5242880 4217536 1017472   81% /tmp
/dev/vg05/lvol1    18874368 14569796 4035542   78% /ora/ICTPRD/undotbs
/dev/vg07/lvol1    18874368 5810065 12247789   32% /ora/ICTPRD/temp
/dev/vg06/lvol1    3145728 1573069 1474375   52% /ora/ICTPRD/system
/dev/vg15/lvol1    3145728 1000910 2010774   33% /ora/ICTPRD/redo4
/dev/vg14/lvol1    3145728 1000910 2010774   33% /ora/ICTPRD/redo3
/dev/vg13/lvol1    3145728 1000910 2010774   33% /ora/ICTPRD/redo2
/dev/vg12/lvol1    3145728 1000910 2010774   33% /ora/ICTPRD/redo1
/dev/vg09/lvol1    153092096 136319471 15724390   90% /ora/ICTPRD/index
/dev/vg16/lvol1    67108864 57406650 9095826   86% /ora/ICTPRD/fm
/dev/vg11/lvol1    214958080 172189194 40095881   81% /ora/ICTPRD/edr_index
/dev/vg10/lvol1    450887680 393543345 53760338   88% /ora/ICTPRD/edr_data
/dev/vg08/lvol1    319815680 272692657 44177872   86% /ora/ICTPRD/data
/dev/vg00/lvol6    25165824 14839600 10245568   59% /opt
/dev/vg00/lvol9    16777216 8138509 8098824   50% /opt/oracle
/dev/vg00/lvol5    5242880  190080 5013352    4% /home
/dev/vg03/lvol1    29360128 6900586 21056081   25% /data/ICTPRD/ict_archive
/dev/vg02/lvol1    56623104  534693 52582957    1% /data/ICTPRD/dwf
/dev/vg04/lvol1    6291456  366294 5554890    6% /data/ICTPRD/bulkload
/dev/vg17/lvol1    115343360 55718343 55900917   50% /data/ICTPRD/bmd1
itc01[633]/ #

 

/data is a separete file system as you can see from the bdf output

Torsten.
Acclaimed Contributor

Re: root file system at 99%

Anything else that you hide?

Hope this helps!
Regards
Torsten.

__________________________________________________
There are only 10 types of people in the world -
those who understand binary, and those who don't.

__________________________________________________
No support by private messages. Please ask the forum!

If you feel this was helpful please click the KUDOS! thumb below!   
NDO
Super Advisor

Re: root file system at 99%

Hi Torsten!

 

:) I am not hiding anything , but if you want more information I can provide it to you, I was just trying to abbreviate, sorry I was wrong.

 

The "/" file system used to be 69% at all time, so I come back from leave today and I found that someone has messed up with the system

Torsten.
Acclaimed Contributor
Solution

Re: root file system at 99%

OK, but it was confusing ...

 

 

You need to look for large files in lvol3, not in other lvols (e.g. using "du").

 

If there is a single large files, it is likely much smaller than 1GB, because your / is 1GB only.

 

So a file in question is about 300MB ?


Hope this helps!
Regards
Torsten.

__________________________________________________
There are only 10 types of people in the world -
those who understand binary, and those who don't.

__________________________________________________
No support by private messages. Please ask the forum!

If you feel this was helpful please click the KUDOS! thumb below!   
Dennis Handly
Acclaimed Contributor

Re: root file system at 99%

>I meant that I could not see anything that was big enough, because normally that is under 50% figure.

 

And I answered that question.  You may have removed an open file so it is now invisible.  You need to download and run lsof to find it.

 

And what does this show: du -kxs /

 

>Try to find files with big sizes:  find / -size +1000000000c -exec ls -l {} +

>The files are in /var, /tmp and other file systems, not in the root:

 

You need to add -xdev so it only looks at /.

 

>but you obviously have /opt directly in /.

 

Yes, I also mistakenly trusted in the bdf output.  ;-)

Bill Hassell
Honored Contributor

Re: root file system at 99%

Rather than looking for big files in "/" you should be looking for big directories. A directory can be large because it has one giant sized file, or it could be big because it has 50,000 junk files in it, none of which are very big. USe du to provide a useful listing:

 

du -kx / | sort -rn | head -20

 

The two largest directories are /sbin and /etc. Anything else at the top of the list is a mistake. A common mistake is /dev -- it will always be less than 100k or so. If it is megabytes, you have a bad file in it. There are *NO* ordinary files in /dev:

 

find /dev -type f -exec ls -l {}  \*

 

The above should produce nothing. Remove (or move if it is a useful file) the ordinary files in /dev.



Bill Hassell, sysadmin
NDO
Super Advisor

Re: root file system at 99%

Hi

I found the problem, indeed there were some 300Mb in excess in a directory under /data/ICTPRD. Thank you very much
Torsten.
Acclaimed Contributor

Re: root file system at 99%

So it was outside of these mount points?

 

dev/vg03/lvol1 29360128 6900586 21056081 25% /data/ICTPRD/ict_archive
/dev/vg02/lvol1 56623104 534693 52582957 1% /data/ICTPRD/dwf
/dev/vg04/lvol1 6291456 366294 5554890 6% /data/ICTPRD/bulkload
/dev/vg17/lvol1 115343360 55718343 55900917 50% /data/ICTPRD/bmd1

 

Ok, then it was in /.


Hope this helps!
Regards
Torsten.

__________________________________________________
There are only 10 types of people in the world -
those who understand binary, and those who don't.

__________________________________________________
No support by private messages. Please ask the forum!

If you feel this was helpful please click the KUDOS! thumb below!   
Dennis Handly
Acclaimed Contributor

Re: root file system at 99%

>So it was outside of these mount points?  Ok, then it was in /.

 

So that this was valid and not a good idea:

#du -kxs /data
851336  /data

NDO
Super Advisor

Re: root file system at 99%

Hi!

It was a directory inside /data/ICTPRD, it was only 300Mb

 

I was on leave, them when I came back, I found this issue, but its now solved, thank you for your assistance