/usr full and nfs mount query

[Emma_8]

The /usr filesystem is almost full on one of my servers. I ran a find for large files and the following 2 files I would like to remove:

-r-xr--r-- 2 bin bin 2475728 Oct 31 2003 /usr/contrib/bin/cmsetlog
-r-xr--r-- 8 bin bin 2893520 Oct 31 2003 /usr/sbin/stsetlog

I am not sure what these logs are used for or if I should remove them - any ideas ?

Also, does anyone happen to know how to determine the orginal mount point of a filesystem. For example I have the following nfs mount:

/dev/vgvanfs/lv_cdr_central
83857408 56899240 26753136 68% /nfs_cdr_central

nfs_va:/nfs_cdr_central
83857408 56899240 26753136 68% /cdr_central

...how do I find out where it is being mounted from ?

Cheers
Emma

[Kent Ostby]

Do not remove those files. They are binary files.

The first one is used by SG.

This is unlikely to be the cause of yoru problems as they are only 2 megabyte files.

I would suggest:

find /usr -size +1000000c

and see what turns up.

Best regards,

Kent M. Ostby

[Rick Garland]

To find the mount info you asked, can use bdf and view the information on that line or if it mounts at boot you can look at /etc/fstab.

If automounter is in play, look at maps

[Todd McDaniel_1 · Accepted solution]

Emma,

keep in mind too that filesystems can get full by tons of small files.

I had a situation once where i had 100 30mb files filling up a FS. 3gb back made a difference and wasnt just a few large files...

Try this script written by a friend of mine... it will give you a listing of the top 10 filesystems that are using space.

It is also originally intended to find disk hogs as well and has a wall feature.


Just give it arguents like this.
Usage:
#./top10 filesystem_name

-------------------------------cut here-------------------

# top10.sh
# Written by David R. Kramer 10/23/97
## Report the ten largest consumers of disk space
## for the filesystem or directory specified as $1
## Additionally, for root, produce a report of files
## which are currently open.
## Validate syntax ##
if [ "${#}" -ne 1 ]
then
echo "Usage: ${0} filesystem_name"
exit 1
fi
file_system=${1}
## Collect disk usage in background so it is ##
## ready to be reported In the Top 10 List. ##
f_s="`echo ${file_system} | tr '/' '_'`"
dufile="/tmp/${f_s}_disk_utilization.`date +%y%m%d%H%M`"
du -s ${file_system}/* >${dufile} 2>${dufile}.err &
## This was intended for good use but, only root can read
## the device files. So this will only function for ROOT
## Report on open files uses lsof to get inodes then
## ff to produce the names of the open files
special_device="`df -k ${1} | awk 'NR==2 {print $1}'`"
if [ -r ${special_device} ]
then
(
lsof | grep "${file_system}" | awk '{print $1, $2, $3, $8, $9, $10}' | s
ort -r -n +3 -4 >open_files
awk '{printf("%s ",$5)};END{printf("\n")}' open_files |
xargs ff ${special_device} -i
) 2>/tmp/open_files.err >/tmp/open_files.rpt
fi
wait
## Top 10 List ##
DATE="`date '+%D@%R'`" ; export DATE
THISSYS="`uname -n`"
case "${THISSYS}"
in
cmps* ) SYSTEM_TYPE="PRODUCTION" ;;
cmvt* ) SYSTEM_TYPE="VOLUME TEST" ;;
cmdv* ) SYSTEM_TYPE="DEVELOPMENT" ;;
esac
export SYSTEM
sort -n -r ${dufile} |
head -10 |
awk '
BEGIN{
cdate = "'${DATE}'"
system_type = "'${SYSTEM_TYPE}'"
printf("HOME Directory Disk Utilization on %\n", system)
printf("\n Top 10 List\n")
printf("\n %s\n", cdate)
printf("\n MBytes Home Directory\n")
printf(" ------- -----------------\n")
}
{
printf(" %6d %s\n", $1/(2*1024), $2)
}
END{
printf("\n\n")
}' >/tmp/wall_file

cat /tmp/wall_file

echo "\nWall Top 10 List? (y/n) \c"
read ans
if [ "${ans}" = "y" -o "${ans}" = "Y" ]
then
cat /tmp/wall_file | wall
fi


# end top10.sh

---------------------------cut here-----------------------------

[Emma_8]

Thanks for the script Todd. I just ran it and /var/stm is consuming the most space. In /var/stm/logs/tools that are about 40 sub-directories (excerpt below):

drwxr-xr-x 2 root sys 96 Jan 23 2004 0_4_0_0_1_22_39_0_0_5
drwxr-xr-x 2 root sys 96 Jan 23 2004 0_4_0_0_1_22_255_2_7
drwxr-xr-x 2 root sys 96 Jan 23 2004 0_4_0_0_1_22_39_0_0_6
drwxr-xr-x 2 root sys 96 Jan 23 2004 0_4_0_0_1_22_255_2_7_0
drwxr-xr-x 2 root sys 96 Jan 23 2004 0_4_0_0_1_23_255_14_4
drwxr-xr-x 2 root sys 96 Jan 23 2004 0_4_0_0_1_23_255_14_4_0
drwxr-xr-x 2 root sys 96 Jan 23 2004 0_4_0_0_1_23_255_14_4_1

Each one of these have a info sub-dir which contains a test_activity_log and info_log_msg files. Given these are all timestampted Jan. '04 can I removed all the above dir's or do you know what is writing to them ?

Cheers
Emma

[Sundar_7]

Emma,

cmsetlog is an un-supported utility that can be used to set the amount of logging the cluster daemons/commands log in the log files.

As mentioned above, they are very small in size and should not be removed from the system.

in /usr filesystem there is not much housekeeping you can do unless you have some huge core files or unnessary softwares installed in the system.

Find and remove the core files in the filesystem

# cd /usr
# find . -name "core" -exec ls -lrt {} \;

If the core files are significant in size, then remove them or copy them to a different filesystem.

# cd /usr
# du -sk * | sort -n

the above command will sort the directories based on their usage. Take it from there.

original mount point of the system ? - hmm - what do u mean by that ?

nfs_va:/nfs_cdr_central
83857408 56899240 26753136 68% /cdr_central

In the above case, the filesystem is from the NFS server nfs_va. server nfs_va is exporting its /nds_cdr_central filesystem and it is mounted under /cdr_central in your system.

-- Sundar.

[Emma_8]

...I think I've found it - there are about 2000 of the following files in /var/stm/logs/os:


-rw-r--r-- 1 root root 512656 Feb 17 2004 log3506.raw
-rw-r--r-- 1 root root 512656 Feb 17 2004 log3507.raw
-rw-r--r-- 1 root root 512656 Feb 17 2004 log3508.raw
-rw-r--r-- 1 root root 512656 Feb 17 2004 log3509.raw
-rw-r--r-- 1 root root 512656 Feb 17 2004 log3510.raw
-rw-r--r-- 1 root root 512656 Feb 17 2004 log3511.raw
-rw-r--r-- 1 root root 512656 Feb 17 2004 log3512.raw


Can I safely remove these ?

Thanks a mil
Emma

[Todd McDaniel_1]

Yes you can remove them down to the most recent ones...

I only have 1 log* file in my dir there.

I wouldn't keep more than a month if that...

Even still, if you have good backups, I doubt you will ever need to restore those logs.

[Emma_8]

Thanks alot. I freed up about 1.5GB by just removing those log files.

[Sundar_7]

But the bottomline here is, you should not have those many stm log files in that directory.

stm rotates the log files every 512 KB.

Do NOT remove the log file with .cur extention. That is the file currently used by the stm.

I would review one of those log files before deleting them. With so many log files, it is quiet possible that something is failing in the system

# cstm
> runutil logtool
> sr log3XXX.raw

[Dave Olker]

Hi Emma,

Did you ever figure out which server "nfs_va" is or did you still need help with that? If you do, you can try using the "nfsstat -m" command to display the IP address of the server. That should help you find the real NFS server.

However, if this file system is being served by an HA/NFS server (i.e. ServiceGuard) and "nfs_va" is the name of an NFS package, then the easiest way to find the real server is to log onto one of the systems in the ServiceGuard cluster and issue the "cmviewcl" command and look at which system the package is running on currently. That will tell you which system is acting as the NFS server at the current time.

Regards,

Dave

[Emma_8]

Hi Dave

Thanks for your response. I did eventually figure it out and just by using cmviewcl on the SG box as you suggested, I was able to find where the data was coming from.

Thanks anyway.
Emma

[Emma_8]

I have one further question about the .raw log files which are being generated on this platform at /var/stm/logs/os.

I understand that the .raw files are used by cstm. I am getting approx 50 of these files per day of approx 0.5 mb each. Through time these files are causing the /var filesystem to fill up.

I have read a couple of .raw files and they do not appear to be logging any errors or alarms.

Is there any configuration associated with the generation of these files where I would have the option of storing just one weeks worth ?

Thanks in advance
Emma