advfs salvage?

[lothar esser]

Hi,
we have an external Raid array with a advfs file system. When a disk broke it rebuilt itself and it seemed okay for a while. However, it started to be inaccessible out of the blue or more often when we copied files. When I checked the file system with sysman it was in a state of panic. It had worked flawlessly for too long so I did not have any experience with this error or any other for that matter. I used verify to see whether this would fix it but matters got worse.
In the /etc/fdmns/tera1_dmn we used to have dxlab_home but it was changed to dxlab_home_verify_BnIVIL and I can no longer mount the array.
advfscan could not fix it either.
"Can't open device rtera1_dmna" which is odd because it is tera1_dmn not rtera1_dmna ???

So the best I could do so far is to run salvage and got about 10-30 % of the data off the array. Of course not the data that I needed. I have restored the user accounts from a 1.5 month old tape so there is about a month worth of data on it(the array) that we need.
Is there any additional thing I could do to rebuild/repair the array ?

Thanks,

Lothar

[Rob Leadbeater]

Hi,

Without knowing what sort of external array you're using, it's difficult to say...

Can you post the output of "showfdmn tera1_dmn" and, "hwmgr view device" assuming that you're using v5.

Cheers,

Rob

[Venkatesh BL]

> "Can't open device rtera1_dmna" which is odd because it is tera1_dmn not rtera1_dmna ???

I think there is silly mistake here. Looks like the tool added 'r' for (raw device) and 'a' for partition for your input. Please check the man page and give the inputs correctly.

Also, you could try 'fixfdmn' utility. I think there is a '-n' option to check things out without making any sort of change to the disk.

[lothar esser]

Hi,

output from $ showfdmn tera1_dmn

# showfdmn tera1_dmn
showfdmn: unable to get info for domain 'tera1_dmn'
showfdmn: error = I/O error
# hwmgr view device
HWID: Device Name Mfg Model Location
------------------------------------------------------------------------------
3: /dev/dmapi/dmapi
4: /dev/scp_scsi
5: /dev/kevm
35: /dev/disk/dsk0c COMPAQ BD0366459B bus-0-targ-0-lun-0
36: /dev/disk/cdrom0c COMPAQ CRD-8402B bus-1-targ-0-lun-0
41: /dev/disk/floppy0c 3.5in floppy fdi0-unit-0
44: /dev/disk/dsk1c JetStor III bus-0-targ-1-lun-0
57: /dev/random
58: /dev/urandom
63: /dev/ntape/tape0 HP Ultrium 2-SCSI bus-3-targ-3-lun-0

The problem device is #44 JetStor III

L.

[Rob Leadbeater]

Hi,

I'm not familiar with the JetStor array, but things look fairly terminal.

Just to confirm, can you post the output of "ls -l /etc/fdmns/tera1_dmn"

Cheers,

Rob

[lothar esser]

Hi,

here is the output.

# ls -l /etc/fdmns/tera1_dmn/
total 8
lrwxr-xr-x 1 root system 15 May 13 2004 dsk1c@ -> /dev/disk/dsk1c*
drwxr-xr-x 2 root system 8192 Nov 14 13:22 dxlab_home_verify_BnIVIL/
#
L.

[Venkatesh BL · Accepted solution]

# ls -l /etc/fdmns/tera1_dmn/
total 8
lrwxr-xr-x 1 root system 15 May 13 2004 dsk1c@ -> /dev/disk/dsk1c*
drwxr-xr-x 2 root system 8192 Nov 14 13:22 dxlab_home_verify_BnIVIL/ <<<<
#

This is strange. This directory is used only for the device links used by the domain. Looks like some one accidentally put this directory on "Nov 14th". Can you move this directory to /tmp or some place and try the showfdmn command again?

[Rob Leadbeater]

Hi,

I also don't like the look of the @ and * symbols in the link to dsk1c.

I'd try recreating the link, after (re)moving the dxlab_home_verify_BnIVIL directory.

# cd /etc/fdmns/tera1_dmn
# ln -s /dev/disk/dsk1c

Cheers,

Rob

[Steven Schweda]

> I also don't like the look of the @ and *
> symbols in the link to dsk1c.

I don't either, which is why I don't make
"ls" an alias (or whatever) for "ls -F".

[Dennis Handly]

>Steven: which is why I don't make "ls" an alias for "ls -F".

Or when you don't want to use the alias (and using a real shell) you use: \ls

[lothar esser]

Hi Guys,


thanks for the help. Sorry for the delay but a cold knocked me out for 3 days.

Before I do something rash allow me to spell out what I think you are suggesting - because it is not entirely clear to me:

# Remove the strange entry and the other one:

cd /etc/fdmns/tera1_dmn

rm dxlab_home_verify_BnIVIL
rm dsk1c

ln -s /dev/disk/dsk1c .

#----------------------------------!

Is this dot the right interpretation of your suggetion? Is this all or do I need to create a link to dxlab_home ?

Sorry about the @* notation, I got used to it though. Next time I use \ls !

Thanks,

Lothar

[Venkatesh BL]

> ln -s /dev/disk/dsk1c .

#----------------------------------!

There is no 'dot'.

Give the command as:
# ln -s /dev/disk/dsk1c dsk1c

This will recreate the device link in the directory.

Did you try running fixfdmn -n?

[lothar esser]

Hi,

you might have noticed that Mr. Leadbeater's suggested command fell short of the 2nd argument to the ln command and I assumed that a dot as 2nd argument should do it.

So I did remove dxlab_home_verify_BnIVIL which apparently was a directory but I could do that of course with rmdir.
I also removed dsk1c (link) and recreated it as you suggested.
However, I still cannot mount the disk.
Mount returns an I/O error.

Hmm, did I have to recreate dxlab_home also and if so how??

You asked about fixdmn -n and it returned the following:

fixfdmn: Can't open device '/dev/disk/dsk1c'.
fixfdmn: Function not implemented
fixfdmn: No valid volumes found in domain 'tera1_dmn'.

fixfdmn found errors on disk it could not fix.
This domain should be restored from backup
tapes, or you can use the salvage tool to
pull what files it can find off of this domain.


Does this help?

Lothar

[Rob Leadbeater]

Hi,

I didn't put the second argument to the ln command, as you don't really need it.

Can you post the output of "ls -l /etc/fdmns/tera1_dmn" now that you've recreated things so that we can check that everything is as it should be.

Cheers,

Rob

[lothar esser]

Hi,

did not know that ln -s is okay without 2nd args. Thanks.

The out put of your ls command this time is:

$ ls -l /etc/fdmns/tera1_dmn

total 0
lrwxrwxrwx 1 root system 15 Nov 23 10:11 dsk1c@ -> /dev/disk/dsk1c*
#

[Venkatesh BL]

> fixfdmn: Can't open device '/dev/disk/dsk1c'.

Can you please post the o/p of 'disklabel dsk1c'?

[lothar esser]

The output of disklabel dsk1c is

# disklabel dsk1c
# /dev/rdisk/dsk1c:
type: SCSI
disk: III
label: tera1_dmn
flags:
bytes/sector: 512
sectors/track: 128
tracks/cylinder: 128
sectors/cylinder: 16384
cylinders: 11975
sectors/unit: 2343682048
rpm: 5410
interleave: 1
trackskew: 11
cylinderskew: 20
headswitch: 0 # milliseconds
track-to-track seek: 0 # milliseconds
drivedata: 0

8 partitions:
# size offset fstype fsize bsize cpg # ~Cyl values
a: 131072 0 unused 0 0 # 0 - 7
b: 262144 131072 unused 0 0 # 8 - 23
c: 2343682048 0 AdvFS # 0 - 143046
d: 0 0 unused 0 0 # 0 - 0
e: 0 0 unused 0 0 # 0 - 0
f: 0 0 unused 0 0 # 0 - 0
g: 1171644416 393216 unused 0 0 # 24 - 71535*
h: 1171644416 1172037632 unused 0 0 # 71535*- 143046
#

[lothar esser]

So, is it hopeless then ?

Thanks for all your replies. I do appreciate your helping me.

Lothar

[Rob Leadbeater]

Hi Lothar,

I'm not sure there's much else that can be done from the OS level.

What status are things in on the JetStor ?
Maybe you could ask the vendor of that. A quick look at their website suggests that the unit is supported on Tru64, so they may have some more options...

Cheers,

Rob

[Venkatesh BL]

> fixfdmn: Can't open device '/dev/disk/dsk1c'.
> fixfdmn: Function not implemented
> fixfdmn: No valid volumes found in domain 'tera1_dmn'.

Sorry, I don't have a tru64 box now to check the fixfdmn behaviour.

Could you try running 'advfsscan' on the device file (dsklc) instead on the domain?

If you have support contract, you could contact the support center for swift response.

[lothar esser]

Hi,

/sbin/advfs/advscan /dev/disk/dsk1c

Scanning devices /dev/disk/rdsk1c
Can't open device /dev/disk/rdsk1ca
#

I also tried /dev/rdisk/rdsk1c but to no avail. The message is the same.

No we no longer have a service contract with HP. Too expensive for a small lab and I had been planning to replace our last Alpha Server ES40 with a different server. Bad to see Murphy's law in operation.

Lothar

[Rob Leadbeater]

Hi,

> No we no longer have a service contract with HP

I doubt they'd be able to help much anyway, given that the storage is on a non-HP array... Did you ask JetStor ?

Cheers,

Rob

[lothar esser]

Hi,

Yes I did ask ACNC Support (JetStor III support staff) and they said that they have no (no longer) any clue about ADVFS. However, they recommended that I check the SCSI cable connection(s) or switch the host adapter. I did check the cable and it seems okay but I have not yet tried to switch the host adapter.

Lothar

[Venkatesh BL]

>/sbin/advfs/advscan /dev/disk/dsk1c

>Scanning devices /dev/disk/rdsk1c
>Can't open device /dev/disk/rdsk1ca

Use the command as '/sbin/advfs/advscan /dev/disk/dsk1' (without the 'c' character)

[lothar esser]

/sbin/advfs/advscan /dev/disk/dsk1

The result is:


Scanning devices /dev/rdisk/dsk1

Found domains:

tera1_dmn
Domain Id 40a35e44.000f10ea
Created Thu May 13 07:38:44 2004

Domain volumes 1
/etc/fdmns links 1

Actual partitions found:
dsk1c

Sorry for the glitch with the 'c'