topic Re: Data loss. Why? in Operating System - HP-UX

Data loss. Why?

ViS_2 — Mon, 26 Dec 2005 09:44:39 GMT

I had mirrored logical volume
/dev/vg01/lvol1

It was placed on
/dev/dsk/c5t8d0
/dev/dsk/c4t8d0

These disks are lockated in the disk enclosure HP DS2100

The disk /dev/dsk/c5t8d0 had logged some "Midia falure" events in the /etc/opt/resmon/log/reslog.html file and it was decided to replace it with the new one.

The procedure I performed is:
1) vgcfgbackup vg01
2) physically disconnect the disk (/dev/dsk/c5t8d0) from the disk enclosure
3) insert the new disk
4) vgchange -a y vg01
5) vgcfgrestore -n vg01 /dev/dsk/c5t8d0
6) vgsync vg01

All of above was performed on running system.

But vgsync failed and the message was logged in the syslog.log:

Dec XX XX:XX:XX HOST1 vmunix: msgcnt 1941 vxfs: mesg 003: vx_mapbad - /dev/vg01/lvol1 file system free extent bitmap in au 97 marked bad

So, the file system became corrupted.

Where is mistake?

Re: Data loss. Why?

Sivakumar TS — Mon, 26 Dec 2005 10:10:30 GMT

Dear Victor,

The Procedure you have followed is OK, are you sure that the lvol data was intact after removing the c5t8d0 ?

Did you do "dd" read test ( #dd if=/dev/dsk/cxtxdx of=/dev/null )on the c5t8d0 disk before confirming the media problem?

you may also do a "dd" read test in the c4t8d0 disk also..may be that also HAS errors and due to which this situation aoocured.

With Regards,

Siva.

Re: Data loss. Why?

ViS_2 — Mon, 26 Dec 2005 10:59:54 GMT

Thans for your reply.

I'm not sure that the mirror copy of /dev/vg01/lvol1 that remained on the /dev/dsk/c4t8d0 was OK before I've removed the /dev/dsk/c5t8d0.

Moreover, after data corruption had occured I started to analyse the /etc/opt/resmon/log/reslog.html file in detail and I had discover that the disk /dev/dsk/c4t8d0 had logged the "Media failure" evets two times about a half a year before.

Eventually the /dev/dsk/c4t8d0 and /dev/dsk/c5t8d0 was replaced with the new ones.

These fault disks I tried to test on another system (rx2600) by running
dd if= of=/dev/null ...
agains them but I did not find any errors.

Re: Data loss. Why?

Shameer.V.A — Mon, 26 Dec 2005 22:39:51 GMT

Hi Victor,
The actual procedure is to do a vgcfgrestore before vgchange.

Unplugging a disk online and replugging the same mech is not a problem but inserting a
different disk instead is not always supported without reducing the mirror because
LVM may not recognize that the PV was exchanged by a new one or the new disk may
already have a valid LVM header

If data is so critical & you don't have any backup available, you may put the old hard disk alone ( /dev/dsk/c5t8d0 ) and reduce mirror & then remirror with new one.

Please find the attached doc for more reference.

Also check the SCSI connection between your server and the storage box.

Hope this will help you,

Shameer

Re: Data loss. Why?

ViS_2 — Tue, 27 Dec 2005 03:00:55 GMT

Thanks Shameer.V.A

Nice brochure.
Is it possible to download the full version somewhere?

Re: Data loss. Why?

Shameer.V.A — Tue, 27 Dec 2005 03:04:38 GMT

Hi Victor,
I'm having the full pdf edition of this handbook. If you can give me your mail id, I can send you the same...

Smile!!!!!!

Shameer

Re: Data loss. Why?

ViS_2 — Tue, 27 Dec 2005 03:18:25 GMT

Thanks Shameer.
My email is
pilot(_AT_)todes(_DOT_)by

Re: Data loss. Why?

Shameer.V.A — Tue, 27 Dec 2005 03:52:33 GMT

Hi Victor,
I have sent the documents to the " pilot@todes.by" id..

Please check at your side.

Smile!!!!!!!

Shameer

Re: Data loss. Why?

ViS_2 — Tue, 27 Dec 2005 07:38:04 GMT

Shameer, thanks a lot!

Re: Data loss. Why?

Rory R Hammond — Tue, 27 Dec 2005 11:33:53 GMT

Victor,
I can't add any thing about why you lost data.

I use dd all of the time to test access to disks. I do not recommend using it on known bad disk, your command can hang. I use dd to "light up" adjancent disk to help figure out location. I also run vgcfgbackup via cron nightly on all my volume group. see sample code.

Rory;

#ident "@(#) Rory Hammond - /usr/bin/vgback 1.0"
# Purpose: HP recommends that volume groups be saved daily
# problems arise when volume groups
# are added to or changed
# vgback solves this problem just run it daily
# via cron and it backs up all volume groups.
#
# example of crontab entry
#0 22 * * * /usr/bin/vgback >> /var/adm/syslog/syslog.log
#
# 07/28/94 released as good

for arg in $(/etc/vgdisplay |grep "VG Name")
do
case $arg in
*dev*) /etc/vgcfgbackup $arg;;
*) continue;;
esac
done

Re: Data loss. Why?

Ted Buis — Wed, 28 Dec 2005 21:45:56 GMT

There is a document called when good disks go bad. It is attached and should help.