topic Re: root 100% full (urgent) in Operating System - HP-UX

root 100% full (urgent)

subodhbagade — Sat, 15 Nov 2008 00:28:51 GMT

hello expert,

i having hp-ux server 11:11.

we have rebooted the serve but after restarting the server root file system become 100% full.

can we move or delete any of unwanted files or directory so that we can make it free.

please hhhelp me..on this.

thanks in advance.
Subodh

Re: root 100% full (urgent)

Avinash20 — Sat, 15 Nov 2008 00:37:46 GMT

You could check the files which is taking lot of space in root via

# du -xk | sort -rn | more

Also you could check if you have kept any files in /

# ls -l /

Could you please post the output here ??

Re: root 100% full (urgent)

Avinash20 — Sat, 15 Nov 2008 00:38:43 GMT

Sorry the command is

# du -xk / | sort -rn | more

Re: root 100% full (urgent)

Dennis Handly — Sat, 15 Nov 2008 00:40:11 GMT

You need to be very careful which files you remove or move. Do you have a list of these?

There are a bunch of threads on this:
http://forums.itrc.hp.com/service/forums/questionanswer.do?threadId=1275669
http://forums.itrc.hp.com/service/forums/questionanswer.do?threadId=1269726
http://forums.itrc.hp.com/service/forums/questionanswer.do?threadId=1246073
http://forums.itrc.hp.com/service/forums/questionanswer.do?threadId=1192185
http://forums.itrc.hp.com/service/forums/questionanswer.do?threadId=1186813

Re: root 100% full (urgent)

subodhbagade — Sat, 15 Nov 2008 01:38:00 GMT

vgdisplay command show:---

Discrepencies as mention in attache Doc...

please help me...

subodh.

Re: root 100% full (urgent)

Patrick Wallek — Sat, 15 Nov 2008 01:47:50 GMT

What discrepancies?

I see nothing wrong in your "after reboot" section. It shows your '/' LV at 76%.

Did you accidentally attach the same screen shot twice? The 2 screen shots look exactly the same to me.

Re: root 100% full (urgent)

Avinash20 — Sat, 15 Nov 2008 01:48:17 GMT

I could see root filesystem is having 62 MB free..

What is the difference you like to address here..

Re: root 100% full (urgent)

subodhbagade — Sat, 15 Nov 2008 02:19:29 GMT

sorry i had attached wrong one..

on #vgdisplay
it showing

vgdisplay: Warning: couldn't query all of the physical volumes.

please refer attached Doc.

Regards,
subodh

Re: root 100% full (urgent)

Avinash20 — Sat, 15 Nov 2008 02:25:29 GMT

Post output of

# vgdisplay -v
# ioscan -funC disk
# strings /etc/lvmtab
# cat /var/adm/syslog/syslog.log

Probably the other VG's are not getting activated or the disk from storage are not getting CLAIMED in ioscan

Re: root 100% full (urgent)

Avinash20 — Sat, 15 Nov 2008 02:27:42 GMT

Subodh: Request you to start a new thread for the VG issue

Re: root 100% full (urgent)

Johnson Punniyalingam — Sat, 15 Nov 2008 02:28:19 GMT

>>>can we move or delete any of unwanted files or directory so that we can make it free<<

Hi Subo,

when ever your / root file system we should always look for

1. System core file

# ll /core*

2. look for some large files

# find /var -xdev -type f -size +5000000c -exec ll {} \; | sort -nk 5

or

cd /
du -sk * --> this will list down all files & dir* take look them you will know which caused / root file system reached 100%

Thanks,
Johnson

Re: root 100% full (urgent)

Johnson Punniyalingam — Sat, 15 Nov 2008 03:00:13 GMT

Hi Subo,

By checking your vgdisplay "suspect" some disk are missing or unable to query,

vgcfgrestore -n /dev/vg00 /dev/rdsk/c9t0d0
vgcfgrestore -n /dev/vg00 /dev/rdsk/c11t0d0
vgcfgrestore -n /dev/vg00 /dev/rdsk/c9t0d1
vgcfgrestore -n /dev/vg00 /dev/rdsk/c11t0d1

vgchange -a y vg00

vgsync vg00

please open another windows and run the below commands

while true
do
pvdisplay -v /dev/dsk/c9t0d0 |grep stale|wc -l
pvdisplay -v /dev/dsk/c11t0d0|grep stale|wc -l
pvdisplay -v /dev/dsk/c9t0d1 |grep stale|wc -l
pvdisplay -v /dev/dsk/c11t0d1|grep stale|wc -l

sleep 5
done

again do vgdisplay /dev/vg00

Thanks,
Johnson

Re: root 100% full (urgent)

Kapil Jha — Sat, 15 Nov 2008 03:33:19 GMT

Just check if /etc/lvmconf directory has old configuration files these can be the reason.
Try to move these files.
BR,
Kapil+

Re: root 100% full (urgent)

Avinash20 — Sat, 15 Nov 2008 03:35:10 GMT

Hi Subodh,

Please check to which VG's does disk c11* and c9* belong to..

Could you post the output which I have asked.

Re: root 100% full (urgent)

subodhbagade — Mon, 17 Nov 2008 17:41:00 GMT

Avinash and all experts......sorry for delay..

As you mention earlier i have to put command
1)vgcfgrestore -n /dev/vg00 /dev/rdsk/c9t0do and all other pv path.

(2) But what initial steps are require before doing vgcfgrestore

Do we need to unmount all LV belongs to vg vg00 and is this require to do in single user mode?
Do I need to De-activate vg by
#vgchange -a n vg00

(3) In attached Doc ,,,i had given o/p of
vgdisplay -v,,
ioscan -funC disk
strings /etc/lvmtab

Please advice.

Regards,
Subodh.

Re: root 100% full (urgent)

Johnson Punniyalingam — Thu, 20 Nov 2008 05:33:14 GMT

Hi Subodh,

what root file system reached 100% ..? are you still facing any 100 % file system after you rebooted ..?

by seeing your attachment you have only 1 disk under vg00

bdf --> output needed

Subo >>
1)vgcfgrestore -n /dev/vg00 /dev/rdsk/c9t0do and all other pv path.

Johnson>>>
(for above invalid - disk you are pointing on your vgcfgrestore of /dev/vg00 , the disk you are mentioning does not belong to vg00) <<<<

# strings /etc/lvmtab

/dev/vg00
/dev/dsk/c0t6d0 --> you have only 1 disk for vg00

/dev/lathstg
/dev/dsk/c8t0d0
/dev/dsk/c9t0d0
/dev/dsk/c10t0d0
/dev/dsk/c11t0d0
/dev/dsk/c8t0d1
/dev/dsk/c9t0d1
/dev/dsk/c10t0d1
/dev/dsk/c11t0d1
/dev/vgignite
/dev/dsk/c3t6d0

>>Subodh
(2) But what initial steps are require before doing vgcfgrestore

>>> Johnson
(please follow below steps) <<< (ioscan -funC disk shows claimed for all the disk c9 ,c11 )

# pvchange -a y /dev/rdsk/c9t0d0
# pvchange -a y /dev/rdsk/c9t0d1
# pvchange -a y /dev/rdsk/c11t0d0
# pvchange -a y /dev/rdsk/c11t0d1

# vgcfgrestore -n /dev/lathstg /dev/rdsk/c9t0d0
# vgcfgrestore -n /dev/lathstg /dev/rdsk/c9t0d1
# vgcfgrestore -n /dev/lathstg /dev/rdsk/c11t0d0
# vgcfgrestore -n /dev/lathstg /dev/rdsk/c11t0d1

# vgchange -a y /dev/lathstg

# vgsync /dev/lathstg

Subodh : - >>> Do we need to unmount all LV belongs to vg vg00 and is this require to do in single user mode?
Do I need to De-activate vg by
#vgchange -a n vg00 <<

in this case you no need becoz the disk complains are not refering to vg00

if you LV is in vg00

Yes you need to go Single User Mode

Thanks,
Johnson

Re: root 100% full (urgent)

subodhbagade — Thu, 20 Nov 2008 19:19:26 GMT

yes john and all expert.... still i am facing root problem..from /dev/vg00 and pv issue from /dev/lathstg

as per the strings /etc/lvmtab ...
physical volume are missing from other vg (/dev/lathstg) not from /dev/vg00 (/)

i belive root issue to be solve to 1st priority and then pv issue.

i started new thread
http://forums13.itrc.hp.com/service/forums/questionanswer.do?threadId=1290146

please help to solve root space urgently.
i am stuck badly..

thanks to all
subodh..

Re: root 100% full (urgent)

Ninad_1 — Thu, 20 Nov 2008 20:27:33 GMT

Hi Subodh,

3 things to check imm.

1. du -kx / | sort -nr | head -20
2. find /dev -type d -exec ls -ld +
3. find / -xdev -name core -exec ls -ld +

Please post the output for the above 3 cmds.
1. - is to check top most usage files/dirs.
2. - is to check if there is a normal file in /dev directory - which would be a suspect as /dev will have device special files
3. - to check if there are any core files

Regards,
Ninad

Re: root 100% full (urgent)

Johnson Punniyalingam — Fri, 21 Nov 2008 02:46:32 GMT

Hi subodhbagade,

Have you assigned points so fare..? for your "Dear Friends" on their Contribution of their valuable time & sharing Knowledege,

also heard from you who have opened another Thread,

Dude , hope its not Right thing do .. :(

you should have closed the Thread and Opened One

Thanks,
Johnson

Re: root 100% full (urgent)

Suraj K Sankari — Fri, 21 Nov 2008 05:22:32 GMT

Hi,

Check with lsof any open file is there or not

lsof +L1 +D /

Suraj