1834936 Members
2210 Online
110071 Solutions
New Discussion

Re: / filesystem full

 
SOLVED
Go to solution
aparna challagulla
Valued Contributor

/ filesystem full

 
If you don't have time to do it right you must have time to do it over
16 REPLIES 16
Tim Adamson_1
Honored Contributor

Re: / filesystem full

Check in the /dev/rmt directory. It is common for people to use a device file of om instead of 0m when backing up files. This causes the backup to go to disk instead of tape.

The files in /dev/rmt should be small in size.



Tim
Yesterday is history, tomorrow is a mystery, today is a gift. That's why it's called the present.
Tim Adamson_1
Honored Contributor

Re: / filesystem full

Just checked your listing again.

/ is 2Gb which is very large for /

I also note you have some large files there:

-rw-r--r-- 1 226 mqm 20736000 Oct 15 2001 hp-U474837.v11
-rw-r--r-- 1 226 mqm 9369600 Oct 15 2001 hp-U474838.v11DCE

I would use the find command to find files greater than a certain size. Make sure you don't cross mount points so that way you only check /

Refer to the find man page for more details.


Tim
Yesterday is history, tomorrow is a mystery, today is a gift. That's why it's called the present.
Tim Adamson_1
Honored Contributor

Re: / filesystem full

Us this command to find all files greater than say 10Mb.

# find / -xdev -size +10000c -exec ls -l {}\;

-xdev means don't cross mount points
-size is size in blocks (1 block = 512 bytes) or bytes if c is used.


Check the lising that comes back and determine if you can delete them or not.


Tim
Yesterday is history, tomorrow is a mystery, today is a gift. That's why it's called the present.
Bill Hassell
Honored Contributor
Solution

Re: / filesystem full

From just a quick looksee, the following files do *NOT* belong in /:

U474837.tar (30 megs!)
hp-U474837.v11
hp-U474838.v11DCE


s mentioned, 2Gb is *WAY* too large for / as the / directory should be a static volume, perhaps 200 megs i size. What has happened is the previous administrator(s) have allowed massive directories to be created in / rather than on a separate volume (the recommended method). Also, it appears that root's HOME directory is /, a very bad (but very common) problem in many flavors of Unix.

Start by creating a new directory called /root. Then move all files (not directories) to /root. This includes all dot-files (files like .profile and .exrc). Then change root's HOME directory by editing /etc/passwd with the vipw command and change root's HOME from / to /root. The reasom for this is to separate root 'droppings' from the overall root directory and make the root user 1000% less likely to make the terrible mistake of rm -rf * in the wrong directory. You'll also need to move root's local directories to /root:

/.dt /.netscape /.secure /.sw /nsmail

Login again as root (leave an existing session running just in case) and you should find /root as the HOME directory and now look at the files that are there (a lot less than /) to see if there are junk/temp files that can be removed.

As a quick check for /dev, do this:

du -s /dev

It should be just a couple of digits, something like 20 to 50. If this directory is very large (3 digits or more) then someone has likely created an ordinary file in /dev by misspelling a device file. Do this:

find /dev -type f

Remove all ordinary files that you see in this find search. Common mistakes are: /dev/rmt/om (supposed to be /dev/rmt/0m) and /dev/null1 or /dev/null2, etc.

Now it is time to locate all the directories that are part of / and not separately mounted volumes:

du -kx / | sort -rn | head -20

The *only* occupied directories in / should be:

/sbin
/etc
/dev

Nothing else!


From the above list, the previous admin(s) used root's HOME as a junk storage area (see the directories /MQHA /archive /hponly /jag /nixon /oradata) so these (probably very large) directories must be moved. It is hard to tell if any of these directories are critically important to your system's operation. NONE of these directories have anything to do with HP-UX so you can move them to another volume but I would do this during downtime if this is a production system. Then startup your usual applications (looks like Oracle and MQM) so see if all is well.

As a general rule, / is a static directory and never grows except when the root user makes a mistake. Often the mistake is allowing 3rd party software to self-install without any controls. A lot of 3rd party (and freeware) software never asks the sysadmin where the files should be installed. Like most modern SysV (layout v.4) systems, /opt is the preferred location for applications, with /usr/local and /usr/contrib less desirable.

Note also that your system needs a security review. There are several 777 directories in / (a terrible security error) such as /MQHA /archive /hponly /nixon and I suspect the the permissions of the directories in /usr/local are also 777 (actually an HP-UX installation error). All directories should be 755, just like /usr/contrib. This will fix /usr/local:

find /usr/local -type d -print -exec chmod 755 {} \;

I also see a /core file iwhich means that there may be many more scattered around the system. If you do not write, compile and test your own software, you should determine what programs caused the core file then remove them after asking the program's author for a fix. This will help locate core files:

find / -type f -name core -print -exec file {} \;

Note that this will keep HP-UX pretty busy for a fe minutes so run the command during quiet time.


Bill Hassell, sysadmin
Bill Hassell
Honored Contributor

Re: / filesystem full

And to prevent core files from occuring, put this into /etc/profile:

ulimit -Sc 0

If a developer needs to collect a core file, they can use the ulimit command to enable a large core file using:

ulimit -Sc 10000

or similar. The -S sets a variable limit. Without -S, the default is a hard limit which cannot be increased. The ulimit feature assumes that you are using the standard POSIX shell (ksh does not have ulimit options).


Bill Hassell, sysadmin
Sridhar Bhaskarla
Honored Contributor

Re: / filesystem full

Hi,

As indicated before U474837.tar hp-U474837.v11 hp-U474838.v11DCE do not belong to root. They are MQ patches. You can move them to a filesystem.

The size of / filesystem at our site is 250MB and it is every hardly 40%.

find / -xdev -type d |xargs du -sk |sort -n|tail -10

Will show you top 10 directories under / with respect to the sizes.

My bet would be on the directories like nixon, jags, hponly etc., that someone created them populated with files. The above command will should show you them.

Keep a root session open. Otherwise, the system will become inaccessable if it becomes 100%.

-Sri

-Sri
You may be disappointed if you fail, but you are doomed if you don't try
John Payne_2
Honored Contributor

Re: / filesystem full

We have a 3rd party product that likes to continue to write to /etc/rc.log when a filesystem fills or the product is started and stopped. This will occasionally make our / more full than it should be. (And fill it when no one is paying any attention.) If you have a large /etc/rc.log, you can do a 'tail /etc/rc.log > /etc/rc.log' which will empty it out and free the space.

Hope it helps

John
Spoon!!!!
Bill Hassell
Honored Contributor

Re: / filesystem full

Regarding 3rd party products that make take away your responsibilities as a sysadmin, I wouold definitely make this serious error visible in current and future negotiations. /etc/rc.log is only for startup so perhaps that 3rd party vendor's startup script is badly written. And to keep rc.log from causing serious problems, relocate it to someplace like /var/adm by using a symlink:

mv /etc/rc.log /var/adm/rc.log
ln -s /var/adm/rc.log /etc/rc.log

You will still have to trim rc.log but I would not clear the entire file regularly since problems with startup are logged to this file.


Bill Hassell, sysadmin
Steven E. Protter
Exalted Contributor

Re: / filesystem full

I liked Bill's ulimit command. I learn something new every day here.

I would like to share my approach to the root filesystem.

I keep it at around 1 GB. I keep a file in there around 100 MB. Its a slug, its an emergency escape valve.

If for some reason I get clobbered with core files or a huge file in /dev/rmt I can quickly get myself 100 MB of breathing room while I try and figure out what's going on.

Thanks to Bill I guess I won't have to worry about core files anymore. Thanks Bill.

My other approach is to keep NOTHING else in the root filesystem.

There are few reasons to keep files in there. You need the configuration files in /etc but that takes up a trivial amount of disk space.

I keep a few scripts that are useful to me in single user mode and a few text files telling me how to do things in single user mode so if I don't have internet access, I can take care of tasks that I don't do often but need to know.

I let my .profile file for root sit in /root because thats my choice on where that should sit. If I run a browser or x-windows as root, I direct all the cache and configuration files to the /home/root filesystem. Those browsers really do collect a lot of junk.

I think I was on the job as sysadmin two days when someone ran a bad script that the previous guy had written and a 650 MB file was sitting in /dev/rmt

The /etc/group file got hammered down to zero bytes(A nasty characteristic of 11.00) and I was having a lot of trouble keeping the system running while I diagnosed and corrected. Thats why I keep the 100 MB slug file, to give me breathing room.

Another thing you need to know.

If I have a folder called /stuff and I copy 200 MB of files in there and then mount a filesystem on that folder, those 200 MB are taking up space in the root filesystem and are completely invisible as long as there is a filesystem mounted in the /stuff folder.

You have to watch out for that and if you've done a thorough search and can find nothing, you might want to umount filesystems and look for junk in the folders. A good expample is due to a disk problem once /var failed to mount. A bunch of log files collected in there before I became aware of the problem. The next time the system was booted /var mounted and root was 75 MB fuller than it should have been. Because you can't umount /var on a running system, I had to take the system into single user mode and clean it out.

As a reference you can go to single user mode as follows:

At the console:

boot the box,
Interupt at the 10 second prompt.

bo

Y Interact with the ISL
hpux -is

You are in single user mode.

This is good to know in case you ever need to reset the root password and don't know it because you've been hacked or forgetful.

Good luck.

SEP

Steven E Protter
Owner of ISN Corporation
http://isnamerica.com
http://hpuxconsulting.com
Sponsor: http://hpux.ws
Twitter: http://twitter.com/hpuxlinux
Founder http://newdatacloud.com
Andi Rigauer
Regular Advisor

Re: / filesystem full

Ok hello from Germany.

go to / and type du -sk * | sort -n
this shouws you all files / directories under /
and the size. With this command you can see which file/dir is using at most space.
.tar can usually be deleted.

Regards
Andi
god, root where's the difference
Tim Sanko
Trusted Contributor

Re: / filesystem full

U474837.tar,hp-U474838.v11DCE, and hp-U474837.v11 may be removed. I would also whack
stmout.txt, testexp,core and have the DBAs look at the oracle 9i installation. If 9i is implemented look for oinstall (ll | grep oinstall ) and see if they are done with the installation. oracle:dba should be the group of any active directories.

I am going to say something that others may find heresy. 2 GB is not always too large. If you have 250 VGs or are running some of a particular list of software which includes Visibility, PeopleSoft,SAP and depending on the vendors who implemented it for you 2 GB may not be excessive. I see no reason to consider reducing the root volume at this time.
Tim Sanko
Trusted Contributor

Re: / filesystem full

Remember that a 20 MB file is 1% of a 2 GB filesystem. The problem may well be a symbolic link on a mounted filesystem pointing back to the root volume.


I would do the following:

cd /

du -sk * | egrep -v 'all|your|mount|points'

If that didn't find the offending party,

find / exec ll {} \; | grep ' -'
this will find all links. see if any are linked to the root partition.

That should be the ultimate answer...


Tim Sanko
Tim Sanko
Trusted Contributor

Re: / filesystem full

Remember that a 20 MB file is 1% of a 2 GB filesystem. The problem may well be a symbolic link on a mounted filesystem pointing back to the root volume.


I would do the following:

cd /

du -sk * | egrep -v 'all|your|mount|points'

If that didn't find the offending party,

find / exec ll {} \; | grep ' -'
this will find all links. see if any are linked to the root partition.

That should be the ultimate answer...


Tim Sanko
aparna challagulla
Valued Contributor

Re: / filesystem full

hi Tim,Steven,Bill,John & Sridhar

thanks a lot for ur help. i was able to delete a lot of unwamted large files (tar,dumps,logs....) because of ur replies. the / is now 16% full.

actually the database guys had also started two instances of MQ running which was generating a lot of log files in / :-(.

thx again. i got to know a lot more than i expected when i posted the question :-)

bye
aparna
If you don't have time to do it right you must have time to do it over
Murgesh_2
Advisor

Re: / filesystem full

Please assign the points, this will add more value to time spent.
Enjoy
Murgesh
aparna challagulla
Valued Contributor

Re: / filesystem full

hi murgesh,

advice taken. :-)

aparna
If you don't have time to do it right you must have time to do it over