root 100% full (urgent)

[subodhbagade]

hello expert,

i having hp-ux server 11:11.

we have rebooted the serve but after restarting the server root file system become 100% full.

can we move or delete any of unwanted files or directory so that we can make it free.

please hhhelp me..on this.

thanks in advance.
Subodh

[Avinash20]

You could check the files which is taking lot of space in root via

# du -xk | sort -rn | more

Also you could check if you have kept any files in /

# ls -l /

Could you please post the output here ??

[Avinash20]

Sorry the command is

# du -xk / | sort -rn | more

[Dennis Handly]

You need to be very careful which files you remove or move. Do you have a list of these?

There are a bunch of threads on this:
http://forums.itrc.hp.com/service/forums/questionanswer.do?threadId=1275669
http://forums.itrc.hp.com/service/forums/questionanswer.do?threadId=1269726
http://forums.itrc.hp.com/service/forums/questionanswer.do?threadId=1246073
http://forums.itrc.hp.com/service/forums/questionanswer.do?threadId=1192185
http://forums.itrc.hp.com/service/forums/questionanswer.do?threadId=1186813

[subodhbagade]

vgdisplay command show:---

Discrepencies as mention in attache Doc...

please help me...


subodh.

[Patrick Wallek]

What discrepancies?

I see nothing wrong in your "after reboot" section. It shows your '/' LV at 76%.

Did you accidentally attach the same screen shot twice? The 2 screen shots look exactly the same to me.

[Avinash20]

I could see root filesystem is having 62 MB free..

What is the difference you like to address here..

[subodhbagade]

sorry i had attached wrong one..

on #vgdisplay
it showing

vgdisplay: Warning: couldn't query all of the physical volumes.

please refer attached Doc.

Regards,
subodh

[Avinash20]

Post output of

# vgdisplay -v
# ioscan -funC disk
# strings /etc/lvmtab
# cat /var/adm/syslog/syslog.log

Probably the other VG's are not getting activated or the disk from storage are not getting CLAIMED in ioscan

[Avinash20]

Subodh: Request you to start a new thread for the VG issue

[Johnson Punniyalingam]

>>>can we move or delete any of unwanted files or directory so that we can make it free<<

Hi Subo,

when ever your / root file system we should always look for

1. System core file

# ll /core*

2. look for some large files

# find /var -xdev -type f -size +5000000c -exec ll {} \; | sort -nk 5

or

cd /
du -sk * --> this will list down all files & dir* take look them you will know which caused / root file system reached 100%

Thanks,
Johnson

[Johnson Punniyalingam]

Hi Subo,

By checking your vgdisplay "suspect" some disk are missing or unable to query,

vgcfgrestore -n /dev/vg00 /dev/rdsk/c9t0d0
vgcfgrestore -n /dev/vg00 /dev/rdsk/c11t0d0
vgcfgrestore -n /dev/vg00 /dev/rdsk/c9t0d1
vgcfgrestore -n /dev/vg00 /dev/rdsk/c11t0d1


vgchange -a y vg00

vgsync vg00

please open another windows and run the below commands

while true
do
pvdisplay -v /dev/dsk/c9t0d0 |grep stale|wc -l
pvdisplay -v /dev/dsk/c11t0d0|grep stale|wc -l
pvdisplay -v /dev/dsk/c9t0d1 |grep stale|wc -l
pvdisplay -v /dev/dsk/c11t0d1|grep stale|wc -l

sleep 5
done

again do vgdisplay /dev/vg00

Thanks,
Johnson

[Kapil Jha]

Just check if /etc/lvmconf directory has old configuration files these can be the reason.
Try to move these files.
BR,
Kapil+

[Avinash20]

Hi Subodh,

Please check to which VG's does disk c11* and c9* belong to..

Could you post the output which I have asked.

[subodhbagade]

Avinash and all experts......sorry for delay..

As you mention earlier i have to put command
1)vgcfgrestore -n /dev/vg00 /dev/rdsk/c9t0do and all other pv path.

(2) But what initial steps are require before doing vgcfgrestore

Do we need to unmount all LV belongs to vg vg00 and is this require to do in single user mode?
Do I need to De-activate vg by
#vgchange -a n vg00

(3) In attached Doc ,,,i had given o/p of
vgdisplay -v,,
ioscan -funC disk
strings /etc/lvmtab

Please advice.

Regards,
Subodh.

[Johnson Punniyalingam · Accepted solution]

Hi Subodh,

what root file system reached 100% ..? are you still facing any 100 % file system after you rebooted ..?

by seeing your attachment you have only 1 disk under vg00

bdf --> output needed

Subo >>
1)vgcfgrestore -n /dev/vg00 /dev/rdsk/c9t0do and all other pv path.

Johnson>>>
(for above invalid - disk you are pointing on your vgcfgrestore of /dev/vg00 , the disk you are mentioning does not belong to vg00) <<<<

# strings /etc/lvmtab

/dev/vg00
/dev/dsk/c0t6d0 --> you have only 1 disk for vg00

/dev/lathstg
/dev/dsk/c8t0d0
/dev/dsk/c9t0d0
/dev/dsk/c10t0d0
/dev/dsk/c11t0d0
/dev/dsk/c8t0d1
/dev/dsk/c9t0d1
/dev/dsk/c10t0d1
/dev/dsk/c11t0d1
/dev/vgignite
/dev/dsk/c3t6d0


>>Subodh
(2) But what initial steps are require before doing vgcfgrestore

>>> Johnson
(please follow below steps) <<< (ioscan -funC disk shows claimed for all the disk c9 ,c11 )

# pvchange -a y /dev/rdsk/c9t0d0
# pvchange -a y /dev/rdsk/c9t0d1
# pvchange -a y /dev/rdsk/c11t0d0
# pvchange -a y /dev/rdsk/c11t0d1

# vgcfgrestore -n /dev/lathstg /dev/rdsk/c9t0d0
# vgcfgrestore -n /dev/lathstg /dev/rdsk/c9t0d1
# vgcfgrestore -n /dev/lathstg /dev/rdsk/c11t0d0
# vgcfgrestore -n /dev/lathstg /dev/rdsk/c11t0d1

# vgchange -a y /dev/lathstg

# vgsync /dev/lathstg


Subodh : - >>> Do we need to unmount all LV belongs to vg vg00 and is this require to do in single user mode?
Do I need to De-activate vg by
#vgchange -a n vg00 <<

in this case you no need becoz the disk complains are not refering to vg00

if you LV is in vg00

Yes you need to go Single User Mode


Thanks,
Johnson

[subodhbagade]

yes john and all expert.... still i am facing root problem..from /dev/vg00 and pv issue from /dev/lathstg

as per the strings /etc/lvmtab ...
physical volume are missing from other vg (/dev/lathstg) not from /dev/vg00 (/)

i belive root issue to be solve to 1st priority and then pv issue.

i started new thread
http://forums13.itrc.hp.com/service/forums/questionanswer.do?threadId=1290146

please help to solve root space urgently.
i am stuck badly..

thanks to all
subodh..

[Ninad_1]

Hi Subodh,

3 things to check imm.

1. du -kx / | sort -nr | head -20
2. find /dev -type d -exec ls -ld +
3. find / -xdev -name core -exec ls -ld +


Please post the output for the above 3 cmds.
1. - is to check top most usage files/dirs.
2. - is to check if there is a normal file in /dev directory - which would be a suspect as /dev will have device special files
3. - to check if there are any core files

Regards,
Ninad

[Johnson Punniyalingam]

Hi subodhbagade,

Have you assigned points so fare..? for your "Dear Friends" on their Contribution of their valuable time & sharing Knowledege,

also heard from you who have opened another Thread,

Dude , hope its not Right thing do .. :(

you should have closed the Thread and Opened One


Thanks,
Johnson

[Suraj K Sankari]

Hi,

Check with lsof any open file is there or not

lsof +L1 +D /

Suraj

[subodhbagade]

Johnson and all Dear freiends...

Pretty sorry for Delay in points....appreciating to all u experts...still Badly stuck with this issue...john thanks for realising me ..

As suggested
(1)# du -kx / |sort -nr |head -20 ( which log or old file can we move)
189040 /
140056 /etc
63448 /etc/vx
48256 /sbin
34120 /etc/opt
32960 /etc/vx/type
29472 /etc/vx/static.d
28760 /etc/vx/static.d/build
25120 /etc/lvmconf
24336 /etc/opt/resmon
17808 /etc/opt/resmon/lib
14920 /etc/vx/type/static
8952 /etc/vx/type/gen
8504 /etc/lp
6776 /etc/vx/type/raid5
6736 /etc/lp/interface
6232 /etc/opt/OV/share
6232 /etc/opt/OV
5928 /sbin/fs
5512 /etc/opt/OV/share/conf

(2) find /dev -type d -exec ls -ld +
drwxrwxrwx 17 bin bin 8192 Nov 17 13:25 /dev
drwxrwxrwx 3 bin bin 8192 Jul 12 2007 /dev/diag
drwxrwxrwx 2 bin bin 96 Jul 12 2007 /dev/diag/rmt
drwxrwxrwx 2 bin bin 8192 Apr 11 2008 /dev/dsk
drwxrwxrwx 2 bin bin 96 Jun 2 2007 /dev/krm
drwxrwxrwx 2 root sys 8192 Oct 14 05:55 /dev/lathstg
drwxrwxrwx 2 bin bin 8192 Jun 2 2007 /dev/pts
drwxrwxrwx 2 bin bin 8192 Nov 15 2000 /dev/pty
drwxrwxrwx 2 bin bin 8192 Nov 15 2000 /dev/ptym
drwxrwxrwx 2 bin bin 8192 Jan 16 2008 /dev/rdsk
drwxrwxrwx 2 bin bin 8192 Jul 12 2007 /dev/rmt
drwxrwxrwx 2 bin bin 8192 Jun 18 2007 /dev/rscsi
drwxrwxrwx 2 bin bin 96 Nov 15 2000 /dev/screen
drwxrwxrwx 2 bin bin 96 Nov 15 2000 /dev/telnet
drwxrwxrwx 2 root root 8192 Jun 25 2007 /dev/vg00
drwxrwxrwx 2 root sys 96 Jan 16 2008 /dev/vgignite
drwxrwxrwx 2 bin bin 8192 Jun 2 2007 /dev/vx

(3)# find / -xdev -name core -exec ls -ld + ( no 0/p)

(4)lsof +L1 +D / ........sh: lsof: not found.
# man lsof
No manual entry for lsof.
(5)cd /etc/lvmconf
# ls (can we move old one)

lathstg.conf lathstg1.conf.old vg00.conf.old vgtest.conf
lathstg.conf.old lathstg1.mapfile vg00.mapfile vgtest.conf.old
lathstg.mapfile lvm_lock vgignite.conf
lathstg1.conf vg00.conf vgignite.conf.old
#

[Ninad_1]

Hi,

Yes, you can move the .old conf files from /etc/lvmconf directory outside / filesystem - which will releive you of some space.
All the directory utilisations in / seems to be genuine - nothing occupying unnecessarily large space - so dont think you can do anything more here - apart from moving the conf.old files.
What you should be looking at is increasing your root filesystem - after taking an ignite image.

Regards,
Ninad