topic Re: Root File system 100% in Operating System - HP-UX

Root File system 100%

Navid HUSSAIN_2 — Sat, 12 Apr 2003 07:58:50 GMT

Hi Techies,

I have K370 Server with HPUX 11.0 and bdf output displayed / file system 100% usage.

Filesystem kbytes used avail %used Mounted on

/dev/vg00/lvol3 258048 256764 1268 100% /

Apart from checking
core files,Trash and any backup dump (by mistake) what else I need to check to get more available space by deleting unused files. I have used cleanup command as well.

Looking for your advise to get more space as I know some where I am wrong to check unused files.. It can not full by necessory system files..

Looking forward to hear quick response..

Thanks in advance ..

NH

Re: Root File system 100%

Paula J Frazer-Campbell — Sat, 12 Apr 2003 08:03:04 GMT

Hi

Have a look at log files - things like wtmp and btmp, also check dev for om instead of 0m.

Paula

Re: Root File system 100%

Robert-Jan Goossens — Sat, 12 Apr 2003 08:30:39 GMT

Hi,

Try to locate the "large" files in lvol3

# find / -xdev -size +1000 -exec ls -l {} \; | more

Hope ir helps,

Robert-Jan.

Re: Root File system 100%

Balaji N — Sat, 12 Apr 2003 08:50:38 GMT

http://www1.itrc.hp.com/service/cki/docDisplay.do?docLocale=en_US&admit=-682735245+1050140963317+28353475&docId=200000065011073&printable=true

may be this is of some help.

hth
-balaji

Re: Root File system 100%

James R. Ferguson — Sat, 12 Apr 2003 10:34:29 GMT

Hi:

Look for files that don't belong in '/dev' (i.e. non-special ones). If someone mis-keyed the backup device for a tape, for instance, an ordinary file would be created in '/dev' instead of a copy occuring to tape. A common mis-key is '/dev/rmt/om' instead of /dev/rmt/0m'. Use 'ls -l' to look for ordinary files.

Regards!

...JRF...

Re: Root File system 100%

John Poff — Sat, 12 Apr 2003 11:00:21 GMT

Hi,

Another thing to check for is any mountpoint directories for filesystems that aren't mounted, but where the mountpoint has world writeable permissions. I've been burned on this before where somebody thinks they are writing to a filesystem but it isn't mounted, and the mountpoint is writeable, so they wind up writing to the / filesystem. I always make my mountpoints have 750 permissions so that non-root users can't write to them.

Also, if your / filesystem stays at 100% even after looking for files and cleaning up things, you might have a process holding open a file that has been removed. The lsof utility is great for finding those problems.

JP

Re: Root File system 100%

Michael Steele_2 — Sat, 12 Apr 2003 11:09:51 GMT

find /dev -type f

And delete for these are file types that don't belong.

Re: Root File system 100%

Bill Hassell — Sat, 12 Apr 2003 15:02:52 GMT

At 258megs, this is a pretty large / directory and should never be this full...perhaps 80-100 megs is typical. So there are a lot of big directories (notice I did not say files?) that do not belong there. NEVER look for big files first! This practice always leads to removing important files (like vmunix) and completely missing the real culprits. Always look first for big directories:

du -kx | sort -rn | head -20

Of course, / will be listed first, but then you should see /sbin and /etc, something like this:

29148 /
18979 /sbin
8572 /etc
6343 /etc/opt
3676 /sbin/fs
3259 /etc/opt/samba
3235 /etc/opt/samba/codepages
2731 /etc/opt/resmon
1657 /sbin/fs/vxfs
1550 /sbin/fs/hfs

What you do NOT want to see is anything that is not part of HP-UX. This often occurs when rogue applications refuse to ask the sysadmin where they should be installed and put themselves in /, the wrong place to be.

And as you can see, /dev is no where to be found in the first few directories. /dev should be 10 to 20Kb in size...anything more and you have bad files there. Use this command to locate b ig mistakes in /dev:

find /dev -type f -exec ll {} \;

There should be nothing found by this command.

Post the results of the du command above and we can help you locate the bad directories and get them moved.

Re: Root File system 100%

Jon Finley — Sun, 13 Apr 2003 04:06:35 GMT

One of the most common errors is to tar a file to tape, but mis-type the tape device name.

Look under /dev/rmt for a large file.

Jon

Re: Root File system 100%

Navid HUSSAIN_2 — Sun, 13 Apr 2003 04:23:39 GMT

Hi,

Thx a lot for your valuable input. Let me work out and if the problem persists then I will post du -k output.

Thanks once again..

NAVID

Re: Root File system 100%

doug hosking — Sun, 13 Apr 2003 05:34:27 GMT

Navid, it would be helpful if you would post the output from the 'bdf' command so we can see the general partition layout of your system. I can't tell if, for example, you have /var/adm/crash under the / partition or some other one. If /var/adm/crash is on the root file system, that would be one of the first places to look to recover space.

Another useful piece of information would be the ouptut from the command
/usr/sbin/vgdisplay -v /dev/vg00 | grep Free

Many, many times I see people who think they need to buy new disks when they haven't even allocated all of the available space in their existing disks to logical volumes. The result is that the disk LOOKS full with bdf, etc.
when it really isn't. There is often extra
space just waiting to be allocated to a logical volume, but it's hidden and unavailable until that happens.

I agree with the others that 250 MB is a lot to have on the root partition. There is probably something simple you can do to recover. With the output from these two commands we could tell a lot better what options you might have to reconfigure your system appropriately.

Re: Root File system 100%

Navid HUSSAIN_2 — Sun, 13 Apr 2003 05:42:37 GMT

Hi
Thanks for your input. I have aleardy check and deleted crash. and We have enough free PE available to extend ,if I am not able to locate unuseful files/directories.
Meanwhile here are bdf output

Filesystem kbytes used avail %used iused ifree %iuse Mounted on
/dev/vg00/lvol3 258048 256764 1268 100% 4673 319 94% /
/dev/vg00/lvol7 3072000 865421 2068799 29% 22263 551641 4% /var
/dev/vg00/lvol6 3072000 495486 2415521 17% 19046 644126 3% /usr
/dev/vg00/lvol5 307200 25242 264400 9% 290 70486 0% /tmp
/dev/vg00/lvol1 119637 28473 79200 26% 47 19153 0% /stand
/dev/vg00/lvol4 3072000 244126 2651173 8% 3199 706965 0% /opt
/dev/vg00/lvol8 20480 1810 17563 9% 50 4666 1% /home
/dev/vg01/lvol1 2048000 36982 1885458 2% 1057 502751 0% /opt/status
srv
/dev/vg01/lvol2 2048000 1514935 500321 75% 5424 133264 4% /opt/oracle
/dev/vg01/lvol3 2048000 1658316 365386 82% 572 97420 1% /var/opt/or
acle

Thanks & regards

NH

Re: Root File system 100%

doug hosking — Sun, 13 Apr 2003 06:29:52 GMT

Thanks for the bdf output. It looks like you have generally reasonable partitions. One thing that jumps out at me is that the 'iused' value for the root file system is several times larger on your system than on an 11.00 K570 I have access to (where 'iused' is 1518). That suggests that your problem may be more a matter of inappropriate directories/files in the root file system than about any large files.

There are several reasons this could happen. One is that applications were loaded onto the root file system instead of in /opt, /usr/local/bin or some other more customary place. Another is that the home directory for some user is on the root file system. Another is that someone could have made a mistake when doing a restore from a backup tape. For example, if you did a restore in single user mode and forgot to mount /opt before doing the restore, the files would be left on the root partition instead of in the /opt partition. These problems can be hard to find, because mounting the real /opt can cover the /opt directory, efectively hiding the files that are causing the problem.

Does the result from 'find / -xdev | wc'
nearly match the 'iused' value for 'bdf -i /' ?
If so, then it's probably a simple problem.
If there is a large difference (hundreds or thousands different), then you may have a more complicated problem that may require some extra work as described below.

If the suggestions above have not found the problem and you can reboot the system in question without major headaches, I would be tempted to:
Boot the system in single user mode.
Do not mount the partitions.
Use 'ls -lRt | more' to list the files.
Look for anything that doesn't belong on
the root file system. That would include
files under var, tmp, opt, and similar.
(Any directories associated with other parititions)

Because the real partitions of these names have not been mounted at this point, you should not see files under these directories.
If you do, it's a sign of the restore problem I mentioned above.

Re: Root File system 100%

Andrew_80 — Sun, 13 Apr 2003 09:42:29 GMT

Are you using raw devices for Oracle ?
The reason I am asking, is that it hapened to me that the DBA wrote to the raw device "/dev/xxyz" and miss typed the raw device by adding a space at the end of the device name whic h of coures filled up root. so check under '/dev'

Good luck.

Re: Root File system 100%

Michael Steele_2 — Sun, 13 Apr 2003 21:58:46 GMT

Looks like you've got a runaway process still hanging on to /. Reboot you server before it crashes and panics.

Re: Root File system 100%

Steven E. Protter — Mon, 14 Apr 2003 01:51:05 GMT

You have made all the right moves and still haven't found the real offending file. The reason may date back to a problem that plagued me a few months back.

I had a system that had a disk problem and the var filesystem failed to mount. The OS however kept writing what log files it could to the unmounted folder, part of the root fs.

I fixed the disk problem, and got /var up and saw most of my root fs space was GONE.

I did everything in the post above and a few other tricks and could not find the problem.

When you write files to a folder and then mount a filesystem on it, those files are invisible until you unmount the filesystem. They take up space, but you CAN'T delete them.

So I booted the box into single user mode with no filesystems mounted. I found lots of files in the /var folder, and a pretty fair number in the /usr folder which had not to my knowledge ever failed to mount. I found 40 Megabytes of files in a folder called /download because I had done patch downloads while it was unmounted while I was making it bigger.

If you need help booting to single user mode post it, but it seems like you probably already know how to do that.

SEP

Re: Root File system 100%

Tim Sanko — Tue, 15 Apr 2003 12:30:38 GMT

First panic and get it over with.

Now look to see if your other filesystems have some room.

They do! Good.

look to see if there are any directories that are not mount points, or system directories that must be available on boot
/dev /bin /sbin /stand /opt /var /etc and so forth.

boot to single user mode.

fsck /

check in lost & found.
du -sk * | sort -n

This will tell you if anything is out under you mount points.

Look under all mount points for hidden space.

Sometimes it is hidden there.

Look for logs and the

ll /etc/wtmp* /etc/utm*
/etc/wt* should be links to
/var/adm/wt*

If they are large (many MB)edit inittab to cycle them. on reboot. (don't loose current process statuses).

du -sk * | sort -n

This will tell you if anything is out under you mount points. Clean up all straggling stuff, and reboot.

This may save you from re-igniting your machine, but if you have the space, I'd ignite it and make a larger /
partition. It is several hours, but only a few keystrokes. Bring a good book and re-Ignite is the best solution if you have free space.

Re: Root File system 100%

Pete Randall — Tue, 15 Apr 2003 12:35:08 GMT

Tim,

What's the point in dragging up these old questions? All you're doing is flooding the front page with old, outdated, already solved problems which makes it difficult to spot the new issues. While I'm sure your insights are valuable, you're hindering people with current problems from getting the support they need. One thing you'll learn out here after a while is that most questioners are "hit and run" types. They get the answer they want and log out and don't return until they have another problem months later.

Pete

Re: Root File system 100%

Michael Steele_2 — Tue, 15 Apr 2003 12:39:27 GMT

Tim has the right!

Re: Root File system 100%

Pete Randall — Tue, 15 Apr 2003 12:49:21 GMT

Absolutely he does, Michael. I was merely hoping that he might reconsider.

Pete