- Community Home
- >
- Servers and Operating Systems
- >
- Operating Systems
- >
- Operating System - HP-UX
- >
- Re: / 100% full - HELP !
Categories
Company
Local Language
Forums
Discussions
Forums
- Data Protection and Retention
- Entry Storage Systems
- Legacy
- Midrange and Enterprise Storage
- Storage Networking
- HPE Nimble Storage
Discussions
Forums
Discussions
Discussions
Discussions
Forums
Discussions
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
- BladeSystem Infrastructure and Application Solutions
- Appliance Servers
- Alpha Servers
- BackOffice Products
- Internet Products
- HPE 9000 and HPE e3000 Servers
- Networking
- Netservers
- Secure OS Software for Linux
- Server Management (Insight Manager 7)
- Windows Server 2003
- Operating System - Tru64 Unix
- ProLiant Deployment and Provisioning
- Linux-Based Community / Regional
- Microsoft System Center Integration
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Community
Resources
Forums
Blogs
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-28-2003 01:04 PM
09-28-2003 01:04 PM
I've got a K-class machine running on HP-UX 10.20. About two weeks ago, I noticed that the root filesystem was 100% full. Dmesg reflected the following:
vxfs: mesg 001: vx_nospace - /dev/root file system full (1 block extent)
Did a du to catch the culprit, but to no avail as most of them are mounted onto their own mount points. Tried clearing root's email in elm, and trimmed all logfiles but still does not help. Rebooted the system as well and its' still 100% full. tried remove /etc/mnttab and then perform mount -a again and still does not help. Any guidelines on what to check ? What is taking up the root filesystem ?
Many Thanks,
Shirley
Solved! Go to Solution.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-28-2003 01:10 PM
09-28-2003 01:10 PM
Re: / 100% full - HELP !
In many cases this is caused by a an incorrect backup being started to a flat file instead of a device.
# du /dev
# du /etc
Regards
Michael
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-28-2003 01:15 PM
09-28-2003 01:15 PM
Re: / 100% full - HELP !
Thanks for your prompt response !
My du /dev/ shows the following:
#du /dev |sort -rn
24 /dev
4 /dev/ptym
4 /dev/pty
2 /dev/vg01
2 /dev/vg00
2 /dev/rmt
2 /dev/pts
2 /dev/diag
0 /dev/rscsi
0 /dev/rdsk
0 /dev/dsk
0 /dev/diag/rmt
And my du /etc shows the following:
#du /etc |sort -rn |more
5504 /etc
1762 /etc/lvmconf
908 /etc/hpC2400
674 /etc/X11
488 /etc/UPS.d
228 /etc/vue
196 /etc/opt
188 /etc/rc.config.d
188 /etc/mail
184 /etc/opt/dce
166 /etc/vue/config
118 /etc/SnmpAgent.d
108 /etc/sam
Any clues what has gone wrong ?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-28-2003 01:27 PM
09-28-2003 01:27 PM
Re: / 100% full - HELP !
# find / -xdev -size 100000c -print
This will only check / and not any other filesystems.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-28-2003 01:31 PM
09-28-2003 01:31 PM
Re: / 100% full - HELP !
du -kx / | sort -rn > /var/tmp/du.root
Now the file /var/tmp will have all the directories in / sorted by size. The largest directories are:
31056 /
18979 /sbin
9331 /etc
6680 /etc/opt
3676 /sbin/fs
No other directories should show up in this list more than a few Kbytes. Growth in / is caused by sysadmin mistakes or intallers that fail to ask the sysadmin where files should be stored.
/dev should NOT be larger than about 20-40 Kbytes (the -k option for du reports in Lb) If /dev is large, there are ordinary files in /dev (there should never be any). Most common are files like /dev/rmt/om or /dev/null. Find all the files in /dev with:
find /dev -type f -exec ll {} \;
There should be nothing shown. If /dev/null shows up as an ordinary file, do this:
rm /dev/null
mknod /dev/null c 3 2
If the root user accidently removes /dev/null, then it will become an ordinary file, possibly megs in size. If the / directory is the largest in the du.root file, then sort all the / files with:
ll / | sort -rnk5 | head
There should be *no* ordinary files in /, just directories. Unfortunately, HP-UX (and a few other Unix flavors) defauot root's $HOME directory to /, probably the worst possible location. Note that logfiles, email, spool files, etc are not in the / directory so this isn't the problem.
Bill Hassell, sysadmin
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-28-2003 03:33 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-28-2003 03:34 PM
09-28-2003 03:34 PM
Re: / 100% full - HELP !
You create a folder on root called /steve
You copy in 100 MB.
You forget and decide to mount a filesystem called /steve on it. You set it up with lvm and mount it. Never remove the 100 MB.
The files are still there and you can't see them, because there is a filesytem mounted on it.
If all other measures faile the way to go is to bring the system into single user mode and start looking around.
Interupt a boot at the console at the 10 second prompt.
bo
Y Interact ...
hpux -is
A more realistic scenario is when /var failed to mount once during a boot. Operations missed it while I was out of town. Logs got written to the /var folder until the / filesystem got near full, then they booted the system. I got home to a very full / filesystem with no explanation. The only way to find it was single user mode.
SEP
Happy 5764
<
Owner of ISN Corporation
http://isnamerica.com
http://hpuxconsulting.com
Sponsor: http://hpux.ws
Twitter: http://twitter.com/hpuxlinux
Founder http://newdatacloud.com
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-28-2003 04:16 PM
09-28-2003 04:16 PM
Re: / 100% full - HELP !
Here's the results for the following commands:
#find /dev -type f
/dev/null2
#quot /
/dev/vg00/rlvol3 (/):
66335 oracle
13084 bin
3581 root
11 lp
#du -kx / | sort -rn
17093 /
14163 /sbin
3039 /sbin/fs
2752 /etc
1297 /sbin/fs/vxfs
1110 /sbin/fs/hfs
881 /etc/lvmconf
510 /sbin/lib
454 /etc/hpC2400
388 /sbin/fs/nfs
337 /etc/X11
269 /sbin/init.d
244 /etc/UPS.d
132 /sbin/fs/cdfs
122 /sbin/set_parms.d
Any idea what could have gone wrong and eating up the space ? Thanks !
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-28-2003 04:23 PM
09-28-2003 04:23 PM
Re: / 100% full - HELP !
Can you post:
# bdf /
Cheers
Con
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-28-2003 04:59 PM
09-28-2003 04:59 PM
Re: / 100% full - HELP !
I use the `find` command for situations like this. There's a couple of options that will help:
-xdev This keeps the find command from crossing to a different mount point
-size Allows you to select the size of the files you're searching for
-exec executes a command on the found files
I'd do something like:
find / -xdex -size +20000 -exec ls -l {} \;
This will give a long listing (`ls -l`) on the file that you found.
The "\;" is necessary on the find command when you're using the exec option. If this finds too many files, bump up the "-size" from +20000 to +25000. If it doesn't find anything, start using smaller numbers( +15000, then +10000, etc).
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-28-2003 05:44 PM
09-28-2003 05:44 PM
Re: / 100% full - HELP !
When issued with the following find command, here's the results...
#find / -xdev -size +1200 -exec ls -l {} \;
-r-sr-xr-x 29 root sys 696320 May 11 1999 /sbin/lvchange
-r-sr-xr-x 29 root sys 696320 May 11 1999 /sbin/lvcreate
-r-sr-xr-x 29 root sys 696320 May 11 1999 /sbin/lvmerge
-r-sr-xr-x 29 root sys 696320 May 11 1999 /sbin/lvdisplay
-r-sr-xr-x 29 root sys 696320 May 11 1999 /sbin/lvextend
-r-sr-xr-x 29 root sys 696320 May 11 1999 /sbin/lvlnboot
-r-sr-xr-x 29 root sys 696320 May 11 1999 /sbin/lvreduce
-r-sr-xr-x 29 root sys 696320 May 11 1999 /sbin/lvremove
-r-sr-xr-x 29 root sys 696320 May 11 1999 /sbin/lvrmboot
-r-sr-xr-x 29 root sys 696320 May 11 1999 /sbin/pvchange
-r-sr-xr-x 29 root sys 696320 May 11 1999 /sbin/pvcreate
-r-sr-xr-x 29 root sys 696320 May 11 1999 /sbin/pvdisplay
-r-sr-xr-x 29 root sys 696320 May 11 1999 /sbin/pvmove
-r-sr-xr-x 29 root sys 696320 May 11 1999 /sbin/vgcfgbackup
-r-sr-xr-x 29 root sys 696320 May 11 1999 /sbin/vgcfgrestore
-r-sr-xr-x 29 root sys 696320 May 11 1999 /sbin/vgchange
-r-sr-xr-x 29 root sys 696320 May 11 1999 /sbin/vgcreate
-r-sr-xr-x 29 root sys 696320 May 11 1999 /sbin/vgdisplay
-r-sr-xr-x 29 root sys 696320 May 11 1999 /sbin/vgexport
-r-sr-xr-x 29 root sys 696320 May 11 1999 /sbin/vgextend
-r-sr-xr-x 29 root sys 696320 May 11 1999 /sbin/vgimport
-r-sr-xr-x 29 root sys 696320 May 11 1999 /sbin/vgreduce
-r-sr-xr-x 29 root sys 696320 May 11 1999 /sbin/vgscan
And my bdf / output is as below:
#bdf /
Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol3 86016 85104 888 99% /
Many thanks !
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-28-2003 05:51 PM
09-28-2003 05:51 PM
Re: / 100% full - HELP !
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-28-2003 06:07 PM
09-28-2003 06:07 PM
Re: / 100% full - HELP !
Here's the bdf output for vg00.
#bdf |grep vg00
/dev/vg00/lvol3 86016 85104 888 99% /
/dev/vg00/lvol1 47829 18230 24816 42% /stand
/dev/vg00/lvol8 327680 144480 173419 45% /var
/dev/vg00/lvol7 417792 313482 97823 76% /usr
/dev/vg00/lvol4 32768 4591 27760 14% /tmp
/dev/vg00/lvol10 658197 478014 114363 81% /oradata2
/dev/vg00/lvol9 1024000 290235 687928 30% /oracle
/dev/vg00/lvol6 368640 286149 77508 79% /opt
/dev/vg00/lvol5 40960 5485 33395 14% /home
Any clue what could have caused the root filesystem to be suddenly full? This machine has been running for years and the root filesystem had never reached near 90% before.
Thanks !
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-28-2003 06:13 PM
09-28-2003 06:13 PM
Re: / 100% full - HELP !
stand
var
usr
tmp
oradata2
oracle
opt
home
Do you have anyother dir's inside root where some files are residing and not mounted probably that is using the space
Revert
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-28-2003 06:21 PM
09-28-2003 06:21 PM
Re: / 100% full - HELP !
find / -name core -ok rm {} \;
This will find any core files and prompt you to remove them. You probably don't need them.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-28-2003 06:28 PM
09-28-2003 06:28 PM
Re: / 100% full - HELP !
(as already mentioned 80MB for / ain't really big)
As no large files could be found on /, there probably must be many small files, not?
.
Try:
find / -xdev -exec ll -d {} \;
Have a quick browse through this long list, and (re)move non-system files.
.
find /dev -size +20 -exec ll -d {} \;
should not produce any output.
good luck,
Thierry.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-28-2003 06:36 PM
09-28-2003 06:36 PM
Re: / 100% full - HELP !
The main issue here is that du reports around 30MB for / while bdf shows its using 85MB.
Normally this indicates that you have an open file that has been removed but hasn't released space in /. However a reboot fixes this problem.
If you have lsof installed run the following command:
# lsof / | sort -rnk 7 |more.
(This shows all open files for / fs, sorted by order of size.)
Often its possible from this output to figure out the offending file and then kill the process that has the file open, thus releasing the space.
The other possibility is what SEP & TG mentioned. Have you mounted any new filesystems recently (run mount command to see date of filesystem mounts).
Cheers
Con
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-28-2003 07:10 PM
09-28-2003 07:10 PM
Re: / 100% full - HELP !
Using the find / -name core command manage to find one core file. Have removed it but still does not help.
Hi Con,
I agree with you. The strange thing is, du reports about 30MB, but why does a root filesystem of 85MB not sufficient ? That's strange.
Unfortunately I do not have "lsof" installed. Anyway to install it? I dont think I have any new filesystems recently created and mounted wrongly.
Normally I'll use the "SAM" method if I want to create a new lvol. I'll never create a directory if I want a new filesystem.
Any clues?
Troubled Shirley.. :-(
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-28-2003 07:16 PM
09-28-2003 07:16 PM
Re: / 100% full - HELP !
#find / -name core -exec rm {} \;
Also you could try to remove log-files
/with regards to its content/:
# > logfile;
Regards
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-28-2003 07:42 PM
09-28-2003 07:42 PM
Re: / 100% full - HELP !
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-28-2003 07:59 PM
09-28-2003 07:59 PM
Re: / 100% full - HELP !
Have a look at which scripts in /sbin/init.d have changed recently, or have a look in /etc/rc.log for anything suspicious.
To test this theory, boot up in single-user mode, and see if / is still full. Then run the other advized stuff (the du and find ideas) without anything else mounted. You might have to mount /usr manually, can't remember if single-user mounts it or not, so make sure which LVOL is for /usr (default is /dev/vg00/lvol7).
Good luck.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-28-2003 09:48 PM
09-28-2003 09:48 PM
Re: / 100% full - HELP !
Check for Big files in the root file system using SAM routine tasks
if you are using Oracle database, check for listener log may be wrongly placed in root volume
Hope this helps
A.Nagarathinam
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-28-2003 11:22 PM
09-28-2003 11:22 PM
Re: / 100% full - HELP !
Since bdf and du have a very bad discrepancy, the problem is likely caused during startup. Check /var/adm/syslog.log, and /etc/rc.log for possible error messages. You may have to go reboot into single user mode to see usage without any mounted filesystems. NOTE: in single user mode, you'll have to issue: mount /usr to get access to most commands.
Bill Hassell, sysadmin
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-28-2003 11:35 PM
09-28-2003 11:35 PM
Re: / 100% full - HELP !
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-29-2003 12:30 AM
09-29-2003 12:30 AM
Re: / 100% full - HELP !
d) find / -xdev -ctime 0 -exec ll {} \'
Regarding: "...#find /dev -type f
/dev/null2..."
'/dev/null2' can be deleted.
Regarding "...#quot /
/dev/vg00/rlvol3 (/):
66335 oracle
13084 bin
3581 root
11 lp..."
Shouldn't have oracle in /. So this is where to concentrate. In the long run make a new logical volume and copy all the oracle files in / into it. Get with oracle for help. For now, run this command:
find / -name -user oracle | xargs ll, or,
find / -name -user oracle -exec ll {} \;
Regarding: "...#du -kx / | sort -rn
17093 /
14163 /sbin
3039 /sbin/fs
2752 /etc
1297 /sbin/fs/vxfs
1110 /sbin/fs/hfs
881 /etc/lvmconf
510 /sbin/lib
454 /etc/hpC2400..."
This list indicates / sub directories are OK. /sbin, /etc, for example. But '...17093..." is too, too many files in /. Concentrate here. This command will provide a cronological list for /.
ls -lrt /
Use it also.