root_domain 100% full

[Annette_11]

The root volume group in the system appears to be at 100%
Filesystem 512-blocks Used Available Capacity Mounted on
root_domain#root 4305184 4303900 1284 100% /
/proc 0 0 0 100% /proc
usr_domain#usr 31259872 1187862 30011904 4% /usr
usr_domain#var 31259872 26306 30011904 1% /var
kits#kits 35565072 2626946 32920592 8% /kits
data1#data1 35565072 14648990 20799040 42% /data1
data2#data2 35565072 15364496 20188000 44% /data2
data3#data3 35565072 12765488 22772480 36% /data3

Can anyone tell me if there are any files which are able to be removed. Also this /proc directory seems to have created itself. I am unable to remove this directory, I have tried rmdir -p but this does not work.

Apologies if there have been similar posts, I used the search function and couldn't see any.

[Ralf Puchner · Accepted solution]

do not delete /proc it is an api and not a stupid directory.

Please use the key "proc" within google to find an explanation and read a unix handbook explaining the basics before starting to delete something, like:

http://h30097.www3.hp.com/docs/base_doc/DOCUMENTATION/V51_HTML/MAN/MAN4/0201____.HTM

Use the search function within this forum to find a way to analyze why your filesystem is 100% full, use showfdmn instead of df (also explained if using search within this forum).
There are a lot of posting regarding this issue:

http://forums1.itrc.hp.com/service/forums/bizsupport/questionanswer.do?threadId=497590
http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=497590
etc...

[Abdul Rahiman]

Running this might help determine which directory in / is really getting filled,

ls |grep -v -E "data|kits|proc" | xargs -i du -sk {} |sort -n -r | more

regds,
Abdul.

[Johan Brusche]

One of the most common causes of a full
root_domain, is a mistake made during the creation of a tar-archive, where the user misspelled the name of the output device.
The consequence is a large file somewhere in the /dev path.

Another one is the existence of a deleted but not properly closed temporary file in /tmp. Consequence is that the filename is not visible but the space is still reserved. Killing the application that did this migth liberate the space.

Johan

[Mohamed K Ahmed]

see if you have multiple copies of vmunix under the root directory. You only need the filenamed vmunix, delete any files with name vmunix.olf, vmunix.PreUpd, vmunix.PrePatch or similar
check for big files under the /etc directory.
check for big files under the /dev directory, as mentioned before, someone might have put the name of a tape drive wrong for example vdump /dev/nrnt0h /usr, this will backup the usr directory to a "file" named nrnt0h in the dev directory.

HTH

Mohamed

[Annette_11]

Thank you for all the replies. I have checked the directories mentioned and there were no files I could remove. Can anyone tell me if I can allocate more space to the / filesystem. I am familiar with aix and I want to know if there are any commands like lsvg, lspv, chfs, etc.

Thank you

[Don Ritchey]

Another tool that will help you with this problem (and other problems as well) is 'lsof' from ftp://vic.cc.purdue.edu/pub/tools/unix/lsof.

This tool will list open files on the system and will help you located files that have been opened and then the file name removed. You can use 'lsof' to look for open files in the root file system by typing "lsof /|grep -v -E 'VCHR||VDIR| txt '", which will exclude directory and character special files, and shared library files from the output. The last three columns are of the most interest: Size/Offset, INode and File Name. Check the items where the file size is unusually large or where there is no file name associated with the inode.

Finish the investigation by running a find command to locate the inodes for the files associated with the inodes:
find / -xdev -inum nnn
which will search only the root file system for files with an inode of nnn. Stopping the process associated with a large unlinked file should free up the space used by that file.

[Annette_11]

On doing a search on how to increase the / directory. I found a statement 'The root filesystem is a static directory and never grows unless improper files are stored in it'.

The reason I am concerned about the / filesystem being 100% is that I am about to upgrade the version of Tru64 from 5.1A to 5.1B, and I am worried the / being at 100% may affect being able to do this.

Can anyone confirm whether this will affect the upgrade?

Thank you

[Mohamed K Ahmed]

your root file system is 2GB, check the requirements of root and usr sizes for 5.1B

[Don Ritchey]

The statement 'The root filesystem is a static directory and never grows unless improper files are stored in it' is not completely correct. Normally, user's files will not go into the root file system, however various programs write temporary files into /tmp, which is part of the root filesystem. Make sure that all the junk is cleaned out of root before you continue your investigation of why root is 100% full.

Your statement: "The reason I am concerned about the / filesystem being 100% is that I am about to upgrade the version of Tru64 from 5.1A to 5.1B, and I am worried the / being at 100% may affect being able to do this." is correct, with root at 100% you will not be able to upgrade from one release to another. You will need to find out the cause of the full condition and rectify it before you consider an upgrade. You may already be suffering from the side effects of a full root filesystem, in that e-mail is probably not working correctly, since most e-mail messages are managed as temporary files in /tmp.

Please use one or more of the troubleshooting methods listed in this discussion to find the cause of the full root filesystem before you consider either patches or an upgrade. The upgrade tool will probably fail without a minumum amount of space in root.

[Michael Schulte zur Sur]

Hi,

can you post
du -kx /
as attachment?

thanks,

Michael

[Mohamed K Ahmed]

Also, check the /sys directory

[Annette_11]

Thank you

[Ralf Puchner]

why not reading previous topics (links have been given)? du and df are not suitable tools to analyze advfs filesystem problems, use showfdmns, quotacheck etc. instead.

Be sure no process has files open, these files will not be seen - but this was written several times in the above links.
(so df and du output is useless)

Have a look into the upgrade guide explaining in detail howto expand the root filesystem during upgrade.

Btw. you can still delete the /.netscape directory.

[Annette_11]

Sorry, I have done: showfdmn -k root_domain and I got the following output:

Id Date Created LogPgs Version Domain Name
3bc6e7f3.0007c9f0 Fri Oct 12 13:54:11 2001 512 4 root_domain

Vol 1K-Blks Free % Used Cmode Rblks Wblks Vol Name
1L 2152592 2033488 6% on 256 256 /dev/disk/dsk0a

This seems to show that only 6% of the root filesystem is being used?

[Ralf Puchner]

it seems so depending on your output. Please post the output as attachement next time, because it is hard to read it with proportional font ;-)

So have a look into the given links and use quotacheck as described to verify the real counters.

[Norman_20]

Hi all, if i can i want to suggest one common problem.....core file.
You can go in / and launch find . -name core, and removethe core file you find.

By Norman

[Ralf Puchner]

Norman,

why not reading the threads prior to post an answer (your proposal is part of several other given answers on the named links)

[Don Ritchey]

Based on the information from your root_domain checkout, it appears to me that you have a transient file that is very large but unlinked from the file system. When was the last time that you rebooted this system? If it is not too disruptive of your operations, I suggest you reboot the system to clear any transient files and get back to a known-good baseline. In fact, to prevent any startup processes from muddying the water, start the system in single user mode and check out the root_domain while nothing else is runnning to interfere with your view.

Once you have checked out the startup conditions, you can then watch for any creeping loss of space over time. You need a known-good baseline to compare to future events. This will allow you to measure any changes and the rate of change.

[Ralf Puchner]

Don,

the advfs informations differs from the information the "unix" commands are seeing. This indicates a quota mismatch which are corrected by using quotacheck (see links, documentation etc.).

A "hidden" file will not be seen by the "unix" commands like df and du but within the showfdmn. But showfdmn gives a smaller value than the df/du commands so I don't think problem rely on "open" files of a process. A sparse file is more likely but df/du should handle this...

I'm looking forward for the real counters after using the quotacheck command.

[Annette_11]

Ralf,

Thank you for you help. I run the quotacheck earlier on and everything looks o.k. with the root domain now.

Thank you once again for all the replies and advice.

[Michael Schulte zur Sur]

Hi,

I just wanted to politely make you aware of a speciality of this forum, the points system:
http://forums1.itrc.hp.com/service/forums/helptips.do?#28

greetings,

Michael

[Ralf Puchner]

Michael,

your answer does not solve the problem so it must be assigned with 0 points ;-)

Be clever next time and post the content of the link I've given as Norman did and earn 10 points, I've got only 1 *lol*

[Mohamed K Ahmed]

Are you fighting for the points ????!!!!!!???!!!!!???!!!?

[Ralf Puchner]

Mohammed,

I will not fight for points, if all people honor my correct answers I will get additional 5000 points ;-)

Annette:

it would be wise to rate the highest number of points to the correct answer and assign less points to the answers which will be helpful during troubleshooting. Postings not leading to the solutions should be rated with 0 instead of "unassigned" state.

Advantage: Other people will see which answer was correct without reading the whole stuff.