topic Re: root full---not resolved in Operating System - HP-UX

root full---not resolved

subodhbagade — Thu, 20 Nov 2008 12:39:14 GMT

hello all Experts... ....

we having 11.11 Hp-unix server and sap running on it.

initialy we are not able to login quality server... it showing server unexpectedly close net work connection.

and error was /etc/pam.conf is missing

so we had done the following steps.

(1)we rebooted the server into single user mode and

Cp -R –p usr/newconfig/etc/pam.conf /etc/pam.conf and set the permission
-r--r--r-- 1 root sys pam.conf

(2) we rebooted the server again.

(3)After rebooting the server root (file system ) become 100%...

(4)Is the root become 100% because of replacing /etc/pam.conf ???????? or whatever activity we had done....

(5)# ll /core*
/core* not found no core file is find ....

# pwd
ls

.Xauthority .rhosts ThrSaveSPAFields: lvoltest sapswap test
.Xauthority.new .sh_history net sbin testr
.Xauthority.old .ssh bin nsr
.gpmhp-lathstg .sw cdrom stgtmp14oct tmp_mnt
.profile SD_CDROM dev usr
.profile.bak ThCallHooks: error.file lib var
.pvprinters_ver1 ThrSaveSPAFields etc lost+found wan
#

(6)# du -kx / | sort -rn | head -20
188856 /
139872 /etc
63448 /etc/vx
48256 /sbin
34120 /etc/opt
32960 /etc/vx/type
29472 /etc/vx/static.d
28760 /etc/vx/static.d/build
25120 /etc/lvmconf
24336 /etc/opt/resmon
17808 /etc/opt/resmon/lib
14920 /etc/vx/type/static
8952 /etc/vx/type/gen
8320 /etc/lp
6776 /etc/vx/type/raid5
6560 /etc/lp/interface
6232 /etc/opt/OV/share
6232 /etc/opt/OV
5928 /sbin/fs
5512 /etc/opt/OV/share/conf

this are the large files present ...

(7)vgdisplay -v /dev/vg00 showing all lv no pv related issue.....

(8)can we move any of listed file or
can we create a seperate mount point (creating new lv ) any one of above...or
can we make soft link of above or

(9)we don't have /sbin/init.d/nohup.out log which we can move.

How could i make root FS more free / resolve this issue on urgent basis.....

Please help me i am badly stuck with this issue .....

Thanks in advance.....

Regards,
subodh.

Re: root full---not resolved

Ivan Krastev — Thu, 20 Nov 2008 12:44:26 GMT

Use find and search for big files, last modified files to have more information.

regards,
ivan

Re: root full---not resolved

Robert-Jan Goossens — Thu, 20 Nov 2008 12:58:41 GMT

Hi Subodh,

----
(6)# du -kx / | sort -rn | head -20
188856 /
----

And what is the actual volume size for /, is it 200MB?

# bdf /

Regards,
Robert-Jan

Re: root full---not resolved

Wim Rombauts — Thu, 20 Nov 2008 13:00:51 GMT

Maybe, start trying "find / -type f -mtime -1". This will list all files modified/created the last 24 hours.

Maybe some process started looping and you have a directory with an incredible amount of small files. Thes files will not show up in your du output, becuase they are too small.

Re: root full---not resolved

Steven E. Protter — Thu, 20 Nov 2008 13:52:31 GMT

Shalom,

Implement a script to remove all core files more than a day or so old from the entire system.

What are we planning to do with these anyway?

Your system should be looked at and set up with a standard set of self help scripts designed to keep it running.

You can not extend the root file system, and should not try. root is a mount point and needs very few actual files on it.

SEP

Re: root full---not resolved

subodhbagade — Thu, 20 Nov 2008 18:07:51 GMT

Hi to all.......

(1)# bdf /
Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol3 262144 256880 5264 98% /

(2)But
du -kx / | sort -rn | head -20
188856 /

and i am finding the difference of 73288 kb , not sure about this memory,,,please help.

(3)ll /core*
/core* not found no core file is find

This specify no core file ....or how to check core file or script

(4)
ll /largedir |sort -rnk5 |head -20
/largedir not found

(5)i had listed the large files
du -kx / | sort -rn | head -20 in my above query

can we move any of listed file or
can we create a seperate mount point (creating new lv ) any one of above...or
can we make soft link of above or how i should i proceed?

waiting for ur valuable reply

Regards,
Subodh

Re: root full---not resolved

James R. Ferguson — Thu, 20 Nov 2008 18:48:20 GMT

Hi:

The descrepency you see between the value 'bdf' reports for used space and the value 'du' reports is due to an open, but unlinked (removed) file.

That is, ther is a process that has opened a temporary file; immediately unlink()ed or removed it; and is continuing to write to the file. This is a standard technique with great value.

You can find the process by looking for a file with a zero link count. The culprit will probably have a large size/offset that more or less matches the difference in used space you see. Do:

# lsof +D / +L1

Regards!

...JRF...

Re: root full---not resolved

Dennis Handly — Thu, 20 Nov 2008 18:53:47 GMT

>(2)But du -kx / => 188856 /
>I am finding the difference of 73288 kb, not sure about this space

If you just rebooted, I'm not sure why there should be this difference. Unless you have files trapped under a mount point.

>(3)ll /core* => /core* not found no core file is find

You need to use find(1) to look deeper:
find / -xdev -type f -name "core*"

>(5)i had listed the large files

These are large directories. You really can't move or symlink them.

You'll need to look inside each of these for large files, or useless logs.
Try: 139872 /etc, 63448 /etc/vx

Re: root full---not resolved

Johnson Punniyalingam — Fri, 21 Nov 2008 02:35:03 GMT

Hi Subodh,

Have you tried ..? >>Dennis >> suggestions

step1:-
sorted list of all dir and files in /

cd /
du -sk *|sort -krn

Find out large size files:

find . -xdev -type f -size +100000 -exec ls -lrt {} \+

step2:-

vgdisplay -v /dev/vg00 --> output

Thanks,
Johnson

Re: root full---not resolved

Paolo Gilli — Fri, 21 Nov 2008 08:24:12 GMT

Hi, to free some space in root filesystem in short time, try to move the ".old" files in /etc/lvmconf to an other fs.

To extend LVM root Filesystem in HP-UX, read the thrad at http://forums11.itrc.hp.com/service/forums/questionanswer.do?threadId=1098002&admit=109447626+1227255017365+28353475

and the attached file.
"Extending the root file system with Online JFS - fsadm"

hope this help
regard
Paolo

Re: root full---not resolved

Paolo Gilli — Fri, 21 Nov 2008 08:26:23 GMT

Re: root full---not resolved

subodhbagade — Fri, 21 Nov 2008 16:00:02 GMT

Hello Expert ...once again..

As per your suggestion i had listed the all command as you asked ..(please refer attached text.)

1) size in kb..
25120 /etc/lvmconf
it having some old files like
# cd /etc/lvmconf
# ls
lathstg.conf lathstg.mapfile lathstg1.conf.old lvm_lock vg00.conf.old vgignite.conf vgtest.conf
lathstg.conf.old lathstg1.conf lathstg1.mapfile vg00.conf vg00.mapfile vgignite.conf.old vgtest.conf.old

can we move old files ??? as paolo suggestedsuggested.

(3)
# du -sk lathstg.conf.old
3504 lathstg.conf.old
#
# du -sk lathstg1.conf.old
4160 lathstg1.conf.old
#
# du -sk vg00.conf.old
320 vg00.conf.old
#
# du -sk vgignite.conf.old
2280 vgignite.conf.old
#
# du -sk vgtest.conf.old
2280 vgtest.conf.old

which will free 12mb...

or it it possible to move any log or old 0r core files/dir move from du -kx / | sort -rn | head -20

but i find no core files........

(4)
i am confused about space?????

du -kx / | sort -rn | head -20
188856 /
and
/dev/vg00/lvol3 262144 257064 5080 98% /

have a nice timeeee!!!!!!
Subodh

Re: root full---not resolved

James R. Ferguson — Fri, 21 Nov 2008 16:06:35 GMT

Hi (again):

Have you yet evaluated the possiblity of an over-mounted filesystem as Dennis suggested, or the possibliity of an open, but removed temorary file as I suggested, using 'lsof'?

As for removing files, you can remove old copies of files in '/etc/lvmconf'. That is, you *need* the most current file from an LVM change, like '/etc/lvmconf/vg00.conf' but you don't need the predecesor copy: '/etc/lvmconf/vg00.conf.old'.

Regards!

...JRF...

Re: root full---not resolved

Johnson Punniyalingam — Fri, 21 Nov 2008 16:19:29 GMT

Hi,

Can you try look for below folders,.? see weather you could remove some old Ignite files, and free up some space

if you have tape-drive "attached" can you take backup of the Ignite folders,

57024992 ignite-backup1 -->>

118778 ignite-backup --->

Free space left under VG00 --> 31GB
PE Size (Mbytes) 32
Free PE 976
(PE Size * Free PE)= 31GB,

if you have online _JFS you can lvextend the / root File system other-wise you need resize the / root File system - using Ignite to resize you / root File system

Thanks,
Johnson

Re: root full---not resolved

subodhbagade — Fri, 21 Nov 2008 16:44:58 GMT

hello------all
james ......lsof is not working..

(1) lsof +D / +L1
sh: lsof: not found.
#
# man lsof
No manual entry for lsof.

(2)As per ur and Dennis suggestion

it seems to be there are no over-mounted filesystem .

(3)can we make free space from etc/vx/static.d/build

pwd
/etc/vx/static.d/build
# du -sk
28760

# ls -l
total 57520
-rwxrwxrwx 1 bin bin 29443732 Jan 31 2007 vold.o
.
# file vold.o
vold.o: PA-RISC1.1 relocatable object

Subodh

.

Re: root full---not resolved

James R. Ferguson — Fri, 21 Nov 2008 17:33:52 GMT

Hi:

> sh: lsof: not found.

OK, it is usually installed in '/usr/local/bin', so:

# /usr/local/bin/lsof +D / +L1

# whereis lsof
lsof: /usr/local/bin/lsof /usr/local/man/man8/lsof.8

If you don't have it, fetch and install it from:

http://hpux.connect.org.uk/hppd/cgi-bin/search?package=on&description=on&term=lsof

Installation is with 'swinstall'. no reboot is required and the binary and manpages will be placed in the paths I showed above.

Regards!

...JRF...

Re: root full---not resolved

subodhbagade — Fri, 21 Nov 2008 18:58:46 GMT

johnson and all....

i don't have online JFS install on server,

ignite-backup1 -,,,ignite-backup -- both are from different lv ...

so i don't thinks so it will work ??

ignite-backup --->
(1)
/dev/vgignite/lvol1
81920000 57029840 24695768 70% /ignite-backup1
(2)/dev/vg00/lvol10 20480000 124156 19720434 1% /ignite-backup

to make free space ,, view more on some log or old file

du -kx / |sort -nr |head -20 (o/p mention above)

Regards,
Subodh

Re: root full---not resolved

Dennis Handly — Fri, 21 Nov 2008 23:16:45 GMT

>it seems to be there are no over-mounted filesystem.

How did you check this? Did you umount each filesystem except for /?
Or did you export and mount / back on the same system?
(Not that I think this is your issue.)

Re: root full---not resolved

Johnson Punniyalingam — Sat, 22 Nov 2008 00:48:50 GMT

Hi Subodh,

if you can't narrow-down some old files under / file system it would be bit diffcult for us to comment,

As mentioned earlier other-wise you need resize the / root File system - using Ignite

please see below thread which extactly same

http://forums11.itrc.hp.com/service/forums/questionanswer.do?threadId=983018

Thanks,
Johnson

Re: root full---not resolved

subodhbagade — Mon, 24 Nov 2008 19:31:40 GMT

Hiiii Dennis and all ..

To Ensure overmounted
(1) I had neither unmounted all FS (exept /) nor exported as you told.

BUT,

(2) When server was rebooted ...
During that time i had checked all file system ..and except 2 FS all FS were mounted ..
So i guess there were no overmonted case.

Regards,
Subodh..