root full---not resolved

[subodhbagade]

hello all Experts... ....

we having 11.11 Hp-unix server and sap running on it.

initialy we are not able to login quality server... it showing server unexpectedly close net work connection.

and error was /etc/pam.conf is missing

so we had done the following steps.

(1)we rebooted the server into single user mode and

Cp -R –p usr/newconfig/etc/pam.conf /etc/pam.conf and set the permission
-r--r--r-- 1 root sys pam.conf

(2) we rebooted the server again.

(3)After rebooting the server root (file system ) become 100%...

(4)Is the root become 100% because of replacing /etc/pam.conf ???????? or whatever activity we had done....

(5)# ll /core*
/core* not found no core file is find ....

# pwd
ls

.Xauthority .rhosts ThrSaveSPAFields: lvoltest sapswap test
.Xauthority.new .sh_history net sbin testr
.Xauthority.old .ssh bin nsr
.gpmhp-lathstg .sw cdrom stgtmp14oct tmp_mnt
.profile SD_CDROM dev usr
.profile.bak ThCallHooks: error.file lib var
.pvprinters_ver1 ThrSaveSPAFields etc lost+found wan
#

(6)# du -kx / | sort -rn | head -20
188856 /
139872 /etc
63448 /etc/vx
48256 /sbin
34120 /etc/opt
32960 /etc/vx/type
29472 /etc/vx/static.d
28760 /etc/vx/static.d/build
25120 /etc/lvmconf
24336 /etc/opt/resmon
17808 /etc/opt/resmon/lib
14920 /etc/vx/type/static
8952 /etc/vx/type/gen
8320 /etc/lp
6776 /etc/vx/type/raid5
6560 /etc/lp/interface
6232 /etc/opt/OV/share
6232 /etc/opt/OV
5928 /sbin/fs
5512 /etc/opt/OV/share/conf

this are the large files present ...

(7)vgdisplay -v /dev/vg00 showing all lv no pv related issue.....

(8)can we move any of listed file or
can we create a seperate mount point (creating new lv ) any one of above...or
can we make soft link of above or

(9)we don't have /sbin/init.d/nohup.out log which we can move.

How could i make root FS more free / resolve this issue on urgent basis.....

Please help me i am badly stuck with this issue .....

Thanks in advance.....

Regards,
subodh.

[Ivan Krastev]

Use find and search for big files, last modified files to have more information.

regards,
ivan

[Robert-Jan Goossens]

Hi Subodh,

----
(6)# du -kx / | sort -rn | head -20
188856 /
----

And what is the actual volume size for /, is it 200MB?

# bdf /

Regards,
Robert-Jan

[Wim Rombauts]

Maybe, start trying "find / -type f -mtime -1". This will list all files modified/created the last 24 hours.

Maybe some process started looping and you have a directory with an incredible amount of small files. Thes files will not show up in your du output, becuase they are too small.

[Steven E. Protter]

Shalom,

Implement a script to remove all core files more than a day or so old from the entire system.

What are we planning to do with these anyway?

Your system should be looked at and set up with a standard set of self help scripts designed to keep it running.

You can not extend the root file system, and should not try. root is a mount point and needs very few actual files on it.

SEP

[subodhbagade]

Hi to all.......

(1)# bdf /
Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol3 262144 256880 5264 98% /

(2)But
du -kx / | sort -rn | head -20
188856 /

and i am finding the difference of 73288 kb , not sure about this memory,,,please help.

(3)ll /core*
/core* not found no core file is find

This specify no core file ....or how to check core file or script

(4)
ll /largedir |sort -rnk5 |head -20
/largedir not found

(5)i had listed the large files
du -kx / | sort -rn | head -20 in my above query

can we move any of listed file or
can we create a seperate mount point (creating new lv ) any one of above...or
can we make soft link of above or how i should i proceed?

waiting for ur valuable reply

Regards,
Subodh

[James R. Ferguson]

Hi:

The descrepency you see between the value 'bdf' reports for used space and the value 'du' reports is due to an open, but unlinked (removed) file.

That is, ther is a process that has opened a temporary file; immediately unlink()ed or removed it; and is continuing to write to the file. This is a standard technique with great value.

You can find the process by looking for a file with a zero link count. The culprit will probably have a large size/offset that more or less matches the difference in used space you see. Do:

# lsof +D / +L1

Regards!

...JRF...

[Dennis Handly]

>(2)But du -kx / => 188856 /
>I am finding the difference of 73288 kb, not sure about this space

If you just rebooted, I'm not sure why there should be this difference. Unless you have files trapped under a mount point.

>(3)ll /core* => /core* not found no core file is find

You need to use find(1) to look deeper:
find / -xdev -type f -name "core*"

>(5)i had listed the large files

These are large directories. You really can't move or symlink them.

You'll need to look inside each of these for large files, or useless logs.
Try: 139872 /etc, 63448 /etc/vx

[Johnson Punniyalingam · Accepted solution]

Hi Subodh,

Have you tried ..? >>Dennis >> suggestions

step1:-
sorted list of all dir and files in /

cd /
du -sk *|sort -krn

Find out large size files:

find . -xdev -type f -size +100000 -exec ls -lrt {} \+

step2:-


vgdisplay -v /dev/vg00 --> output


Thanks,
Johnson

[Paolo Gilli]

Hi, to free some space in root filesystem in short time, try to move the ".old" files in /etc/lvmconf to an other fs.

To extend LVM root Filesystem in HP-UX, read the thrad at http://forums11.itrc.hp.com/service/forums/questionanswer.do?threadId=1098002&admit=109447626+1227255017365+28353475

and the attached file.
"Extending the root file system with Online JFS - fsadm"

hope this help
regard
Paolo

[Paolo Gilli]

Hi, to free some space in root filesystem in short time, try to move the ".old" files in /etc/lvmconf to an other fs.

To extend LVM root Filesystem in HP-UX, read the thrad at http://forums11.itrc.hp.com/service/forums/questionanswer.do?threadId=1098002&admit=109447626+1227255017365+28353475

and the attached file.
"Extending the root file system with Online JFS - fsadm"

hope this help
regards
Paolo

[subodhbagade]

Hello Expert ...once again..

As per your suggestion i had listed the all command as you asked ..(please refer attached text.)

1) size in kb..
25120 /etc/lvmconf
it having some old files like
# cd /etc/lvmconf
# ls
lathstg.conf lathstg.mapfile lathstg1.conf.old lvm_lock vg00.conf.old vgignite.conf vgtest.conf
lathstg.conf.old lathstg1.conf lathstg1.mapfile vg00.conf vg00.mapfile vgignite.conf.old vgtest.conf.old

can we move old files ??? as paolo suggestedsuggested.

(3)
# du -sk lathstg.conf.old
3504 lathstg.conf.old
#
# du -sk lathstg1.conf.old
4160 lathstg1.conf.old
#
# du -sk vg00.conf.old
320 vg00.conf.old
#
# du -sk vgignite.conf.old
2280 vgignite.conf.old
#
# du -sk vgtest.conf.old
2280 vgtest.conf.old

which will free 12mb...

or it it possible to move any log or old 0r core files/dir move from du -kx / | sort -rn | head -20

but i find no core files........

(4)
i am confused about space?????

du -kx / | sort -rn | head -20
188856 /
and
/dev/vg00/lvol3 262144 257064 5080 98% /

have a nice timeeee!!!!!!
Subodh

[James R. Ferguson]

Hi (again):

Have you yet evaluated the possiblity of an over-mounted filesystem as Dennis suggested, or the possibliity of an open, but removed temorary file as I suggested, using 'lsof'?

As for removing files, you can remove old copies of files in '/etc/lvmconf'. That is, you *need* the most current file from an LVM change, like '/etc/lvmconf/vg00.conf' but you don't need the predecesor copy: '/etc/lvmconf/vg00.conf.old'.

Regards!

...JRF...

[Johnson Punniyalingam]

Hi,

Can you try look for below folders,.? see weather you could remove some old Ignite files, and free up some space

if you have tape-drive "attached" can you take backup of the Ignite folders,


57024992 ignite-backup1 -->>

118778 ignite-backup --->

Free space left under VG00 --> 31GB
PE Size (Mbytes) 32
Free PE 976
(PE Size * Free PE)= 31GB,

if you have online _JFS you can lvextend the / root File system other-wise you need resize the / root File system - using Ignite to resize you / root File system

Thanks,
Johnson

[subodhbagade]

hello------all
james ......lsof is not working..

(1) lsof +D / +L1
sh: lsof: not found.
#
# man lsof
No manual entry for lsof.

(2)As per ur and Dennis suggestion

it seems to be there are no over-mounted filesystem .

(3)can we make free space from etc/vx/static.d/build

pwd
/etc/vx/static.d/build
# du -sk
28760

# ls -l
total 57520
-rwxrwxrwx 1 bin bin 29443732 Jan 31 2007 vold.o
.
# file vold.o
vold.o: PA-RISC1.1 relocatable object

Subodh

.

[James R. Ferguson]

Hi:

> sh: lsof: not found.

OK, it is usually installed in '/usr/local/bin', so:

# /usr/local/bin/lsof +D / +L1

# whereis lsof
lsof: /usr/local/bin/lsof /usr/local/man/man8/lsof.8

If you don't have it, fetch and install it from:

http://hpux.connect.org.uk/hppd/cgi-bin/search?package=on&description=on&term=lsof

Installation is with 'swinstall'. no reboot is required and the binary and manpages will be placed in the paths I showed above.

Regards!

...JRF...

[subodhbagade]

johnson and all....

i don't have online JFS install on server,

ignite-backup1 -,,,ignite-backup -- both are from different lv ...

so i don't thinks so it will work ??

ignite-backup --->
(1)
/dev/vgignite/lvol1
81920000 57029840 24695768 70% /ignite-backup1
(2)/dev/vg00/lvol10 20480000 124156 19720434 1% /ignite-backup

to make free space ,, view more on some log or old file

du -kx / |sort -nr |head -20 (o/p mention above)

Regards,
Subodh

[Dennis Handly]

>it seems to be there are no over-mounted filesystem.

How did you check this? Did you umount each filesystem except for /?
Or did you export and mount / back on the same system?
(Not that I think this is your issue.)

[Johnson Punniyalingam]

Hi Subodh,

if you can't narrow-down some old files under / file system it would be bit diffcult for us to comment,

As mentioned earlier other-wise you need resize the / root File system - using Ignite

please see below thread which extactly same

http://forums11.itrc.hp.com/service/forums/questionanswer.do?threadId=983018

Thanks,
Johnson

[subodhbagade]

Hiiii Dennis and all ..

To Ensure overmounted
(1) I had neither unmounted all FS (exept /) nor exported as you told.

BUT,

(2) When server was rebooted ...
During that time i had checked all file system ..and except 2 FS all FS were mounted ..
So i guess there were no overmonted case.



Regards,
Subodh..

[Dennis Handly]

>To ensure overmounted
>(1) I had neither unmounted all FS (exept /) nor exported as you told.

Then you can't check to see if there are files trapped under the mount point.

>(2) When server was rebooted. During that time I had checked all file system and except 2 FS all FS were mounted.
>So I guess there were no overmounted case.

This also won't check. You need to unmount the filesystem and look at the mount point. Or if you can't do that, you need to remount the root filesystem somewhere else.

[James R. Ferguson]

Hi (again) Subodh:

I agree with Dennis that you really haven't ruled out the possibility of an overmount. Do:

# ff -F vxfs /dev/vg00/rlvol3|grep -vE "^\./(etc|dev|sbin)/"|more

...and examine the output for files that don't belong in the list. Notice that I filtered out things in /etc, /dev, and /sbin since there are *not* mountpoints, but rather directories that are subordinate to '/'.

You can ignore anything like '/tmp/sh*' (e.g. /tmp/sh100.1). These are files opened during startup before the '/tmp' mountpoint is mounted.

Regards!

...JRF...

[Avinash20]

Hi Subodh,

Are you still facing issue ?