Worst system corruption ever experienced..

[Stefan Farrelly]

Hi Marc,

I eagerly await the xstm logtool results when you can get them.

The ncheck for fstab and services actually look good - theres multiple clusters for each so their is indeed content on disk for each file - which is good. This points more to a simple directory listing corruption rather than a physical corruption on the disk, which is a little more reassuring.

[Marc Dijkstra]

Stefan,

I am not sure if I understand, you say that there are multiple clusters for services?????

# ncheck -F vxfs -S - /dev/marc/lvol3 |grep services
UNNAMED 999 1048 2793 /etc/services.window
UNNAMED 999 283 11504-11506 /etc/ppp/Examples/services.ex
UNNAMED 999 283 11462-11477 /etc/ppp/Examples/services.ex
UNNAMED 999 1530 10159 /etc/net/ticlts/services
UNNAMED 999 1532 12113 /etc/net/ticots/services
UNNAMED 999 1534 12181 /etc/net/ticotsord/services
# ncheck -F vxfs -S - /dev/vg00/lvol3 |grep services
UNNAMED 999 69 9860 /etc/services
UNNAMED 999 69 9838-9845 /etc/services
UNNAMED 999 69 9723 /etc/services
UNNAMED 999 205 4513 /etc/services.window
UNNAMED 999 1242 10277 /etc/net/ticlts/services
UNNAMED 999 1244 18581 /etc/net/ticots/services
UNNAMED 999 1044 10707-10709 /etc/ppp/Examples/services.ex
UNNAMED 999 1044 10688-10703 /etc/ppp/Examples/services.ex
UNNAMED 999 1246 1789 /etc/net/ticotsord/services

The top ncheck (/dev/marc/lvol3)is for the affected filesystem, services is not there at all, the second one (/dev/vg00/lvol3) is for the good, just built OS and there are multiple (correct) entries.

The same for all lv commands in /sbin on the affected filesystem, in the /dev/vg00/lvol3 they are there, but NOT in the bad one. (see above attachment ncheck3.txt)

MND

[Stefan Farrelly]

Hi Marc,

sorry, I didnt realise you have both good and bad outputs in your attachments. I cant match a name to an attachment (like ncheck3) as when its uploaded to the itrc it gets a huge long winded name of random numbers+letters.

Anyway, the only file there on the old drive is fstab (multiple clusters). All else are gone - so theyre not even on the physical disk anymore. So it looks like a major corruption of the data on the disk. Still looking forward to the logtool results.

[Bill McNAMARA_1]

What is your disk system..
if it is a disk array such as an XP, make sure that you have used lvcreate with the -r no option.

This will say to lvm not to relocate bad blocks to the bad block relocation area of the PV. The reason being is that the XP will do that at the hardware level and you can imagine the confusion that would result in.

You can remove the lv, then recreate it and lvextend it without loosing data on the overlying filesystem, but you need to be very exact to get that hack to work, but it would rule out/correct this lvm problem if it was an XP.

I don't believe any other arrays do hardware bad block relocation.

in either case, lvdisplay -v the lvol and pvdisplay -v the pv and attach the result.

Later,
Bill

[Bill McNAMARA_1]

what's 16:03 and 16:00 in binary?

I can recall seeing errors 7, 255 and trying to decode them - it eventually triggered that they were all 1's - which meant that the error was meaningless, I'm wondering if the 16:00 timestamp can be constructed with an all 1's.. long shot..

Later,
Bill

[Marc Dijkstra]

Hi Bill,

The array on site is a HP VA7100, it was, however completely unaffected.... all the VG's, including the lock-disk.

Just the mirrored boot disk -- both sides, obviously.

I have attached a lvdisplay of the / lvol and the pvdisplay.

Oh, and on the binary --- man, it's friday, that seems like too much finking!

Just seems very strange that the files were zapped at 16h00 and then another bunch at 16h03 -- and the ftp was
at exactly those times....

As I stated earlier, it seem that all the files that were zapped had to do with disk and network, and the only common denominator for me, the thing that watches/uses *both* is MC ServiceGuard.....???????

MND

[Marc Dijkstra]

Back to top..... I am still one be-baffled techie.....