1748259 Members
3878 Online
108760 Solutions
New Discussion юеВ

Inode problem?

 
Michael Gretton
Frequent Advisor

Inode problem?

I have a server with the /var directory full. I have cleaned out some logs that are eating space but I am still not gaining any space on the FS. I remember that this might be an inode release problem. Is there a way to release these inodes without having to reboot?

I am not certain this is the problem. If I have it wrong, pls let me know.

Thanks,

Mike
8 REPLIES 8
Martin Johnson
Honored Contributor

Re: Inode problem?

Filling /var can cause all sorts of problems. When it happened to me, whoami showed intruder alert, some users could not log on, shutdown would tell me that I not authorized to shutdown the system, etc. I finally had to use reboot to reboot the system.

I would recommend that once you get the space issue resolved, reboot the system to clear up any potential problems.

Remember: before deleting any logs, stop any processes that are writing to the logs. Otherwise, the directory entry is gone, but the file is still allocated until all processes writing to it have stopped.

HTH
Marty
MANOJ SRIVASTAVA
Honored Contributor

Re: Inode problem?

Mike

/var hosts lots of logs for various system activites specially /var/tmp which is used bu oracle /applicatiuons as a scrtach pad , all you do is

1. check /var/tmp
2. ls -lR /var | grep log
to know which log files have to be trimmed .

this will give back most of the space I assume you ahcve cleared all core files.


Manoj Srivastava
Sridhar Bhaskarla
Honored Contributor

Re: Inode problem?

Hi Mike,

Deleting log files is not sufficient. If the processes that opened the files are still active, you won't get the space. For ex., if you cleared mail.log or syslog.log, you would need to restart syslogd.

You can use "lsof" to find out the processes and the opened files in a file system. Based on it you can clean up. You can get lsof 64bit from the following site.

ftp://ftp.fu-berlin.de/pub/unix/tools/lsof/binaries/hpux/B.11.00/vxfs/64/9000_800/

Also, /var/adm/sw/save contains saved patches. You can use "cleanup" command to delete lower versions of patches. Look at the man page of cleanup command. It is recommended to take a backup of /var/adm/sw/save before running the command. I would not suggest going any less than level 2.

-Sri

You may be disappointed if you fail, but you are doomed if you don't try
nancy rippey
Trusted Contributor

Re: Inode problem?

Take a look at the files under /var/adm/sw. The swinstall and swconfig files can become very large and removed if needed. If you think you may need them for future reference you may wish to save them off somewhere. I usually just delete them. If you have vxfs installed and available space you will be able to expand /var if indeed you need more space. Did you happen to install any patches lateley that would cause var to grow?

Hope this helps

nancy
nrip
John Dvorchak
Honored Contributor

Re: Inode problem?

I agree, deleting the logs is not the answer. When I get a full file system alert the first thing I do is something that I learned from Bill Hassel:

du -k /var | sort -rn | more

This tells me where all of the disk space is being used. Because a lot of little files can be just as big of a problem as a couple of big files. Now in the case of /var, you will probably have a bunch in /var/adm/sw and you might try a cleanup but DO NOT clean up first. Cleanup is something you do when you are not in a panic situation and you have made an archive of /var/adm/sw so you can roll back patches if needed.

You have to determine which processes own which files. I do this with fuser and make a determination of what needs to be trimmed or if the process can be terminated gracefully thereby giving up the space. If they are log files and you think you need to trim them, the best way to zero them out is with:

> log.file

It is just redirecting nothing to the file. This will zero the file and recover the space and the process that is logging to it doesn't have to stop and restart.

Good luck
John
If it has wheels or a skirt, you can't afford it.
harry d brown jr
Honored Contributor

Re: Inode problem?

Of course this thread should be merged with http://forums.itrc.hp.com/cm/QuestionAnswer/1,,0x45a33a1c04ffd61190050090279cd0f9,00.html

live free or die
harry
Live Free or Die
Oleg Zieaev_1
Regular Advisor

Re: Inode problem?

Hello Mike.

You can run bdf -i and see how many free inodes you have.
In some cases you do not see space released immediately after deleting the file, due to the different reasons, like file being locked by the process writing to it, etc.
We saw on our site this - huge file deleted and you still had 100% utilization of you file system.
There are few ways to force inode table to be refreshed. Easiest is to touch file in the file system. You can run sync as well.
Agree with the note that you might have process(es) filling up your logs VERY fast, therefore lsof is a good idea mentioned hereabove.
You might in this case consider reseting logs, like
> logfile.log
And of course some patches may be commited, so their uninstall information is removed from /var/adm/sw.

Hope this helps,
0leg
Professionals will prevail ...
Rory R Hammond
Trusted Contributor

Re: Inode problem?

Mike,

When a program (process) opens a file, it uses a file dicriptor.

If you delete the file, and the process is still running. The space is not released until the file descriptor is closed. This is usually done by killing the process.

Commonly, I have seen new admins rm /var/adm/syslog/syslog.log inorder ot free space.

The problem is that the space will not be freed until syslogd is stopped.

"fuser" might have found your problem for instance
fuser -u /dev/vg00/lvol4 (which is my /tmp) You would find what processes have files opened in tmp. Which would hopefully give you a clue to the errant program.

When removing\reduceing logfiles. It is a good practice to do a fuser on the file to see if it is has attached processes. If this is the case ZERO out or truncate the file instead of removeing it
(echo "" > badlog )
Rory
There are a 100 ways to do things and 97 of them are right