Operating System - OpenVMS
1748183 Members
3422 Online
108759 Solutions
New Discussion

Re: Extremely slow backup to LTO Tape after replacement of hot plug (shadowed) internal SCSI disk

 
SOLVED
Go to solution
MarkOfAus
Valued Contributor

Re: Extremely slow backup to LTO Tape after replacement of hot plug (shadowed) internal SCSI disk

"No, I don't think so: the MONITOR data of EMU1 confirms, that the majority of the reads are from $3$DKA300: (the LOCAL member) and only SOME IOs are from $4$DKB300: (the REMOTE member). And the local disk $3$DKA300: is actually the one you've replaced."

 

I stand corrected, you are right. It's just the master member of the set because it fell into that role when the $3$DKA300 disk died. My apologies.

 

Could something as mundane as a io autoconfigure help here?

 

Or, the solution is to replace all the other disks and get wonderful throughput back again... :-)

 

Volker Halle
Honored Contributor

Re: Extremely slow backup to LTO Tape after replacement of hot plug (shadowed) internal SCSI disk

Mark,

 

you can change the Read Cost of the shadowset members on the fly.

 

Maybe start to experiment on EMU2:

 

EMU2 $  SET DEV/READ=1 $3$DKA200:

EMU2 $  SET DEV/READ=500 $4$DKB200:

 

Then try the BACKUP/PHY DSA2: test on EMU2. Most of the reads should now be from the EMU1 member and should be 'slow'. If that works as expected, apply the same setting on EMU1 for DSA1: and DSA2: by reducing the Read Cost for the (remote) EMU2 members and increasing it for the (local) EMU1 members.

 

To reset the read cost values back to their defaults, use $ SET DEV/READ DSA2:

 

And no, I don't think a SYSMAN IO AUTO will help here. Probably only a reboot or even power-off/on of EMU1. But maybe can can work around this problem by modifying the Read Cost...

 

Volker.

MarkOfAus
Valued Contributor

Re: Extremely slow backup to LTO Tape after replacement of hot plug (shadowed) internal SCSI disk

Volker,

 

I did as you requested:

 

SET DEV/READ=1 $3$DKA200:

SET DEV/READ=500 $4$DKB200:

 

and for good measure:

 

SET DEV/READ=1 $3$DKA100:

SET DEV/READ=500 $4$DKB100:

 

Results (as you expected):

 

                            OpenVMS Monitor Utility
                              DISK I/O STATISTICS
                                 on node EMU2
                             8-AUG-2014 21:50:17.29

I/O Operation Rate                         CUR        AVE        MIN        MAX

$4$DKB0:      (EMU2)   ALPHASYS1          0.00       0.12       0.00       5.00
$4$DKB100:    (EMU2)   DATA1              0.00       0.00       0.00       0.00
$4$DKB200:    (EMU2)   DATA2              0.00       0.00       0.00       0.00
$4$DKB300:    (EMU2)   DATA3              0.33       1.73       0.00       7.00
$4$DKB400:    (EMU2)   ADMIN1             0.00       0.00       0.00       0.00
DSA1:                  DATA1             59.00      37.16       0.00      76.00
DSA2:                  DATA2              0.00      18.73       0.00      76.66
DSA3:                  DATA3              0.33       1.17       0.00       3.66
$3$DKA100:    (EMU1)   DATA1        R    59.00      37.15       0.00      76.00
$3$DKA200:    (EMU1)   DATA2        R     0.00      18.73       0.00      76.66
$3$DKA300:    (EMU1)   DATA3        R     0.33       0.98       0.00       2.33

 Curiosity got the better of me:

SET DEV/READ=1 $3$DKA300:

SET DEV/READ=500 $4$DKB300:

 

Output:

                            OpenVMS Monitor Utility
                              DISK I/O STATISTICS
                                 on node EMU2
                             8-AUG-2014 21:51:47.31

I/O Operation Rate                         CUR        AVE        MIN        MAX

$4$DKB0:      (EMU2)   ALPHASYS1          0.00       0.15       0.00       5.00
$4$DKB100:    (EMU2)   DATA1              0.00       0.00       0.00       0.00
$4$DKB200:    (EMU2)   DATA2              0.00       0.00       0.00       0.00
$4$DKB300:    (EMU2)   DATA3             32.33       5.32       0.00      34.66
$4$DKB400:    (EMU2)   ADMIN1             0.00       0.00       0.00       0.00
DSA1:                  DATA1              0.00      24.77       0.00      76.00
DSA2:                  DATA2              0.00      12.48       0.00      76.66
DSA3:                  DATA3           2340.66     289.70       0.00    2361.00
$3$DKA100:    (EMU1)   DATA1        R     0.00      24.77       0.00      76.00
$3$DKA200:    (EMU1)   DATA2        R     0.00      12.48       0.00      76.66
$3$DKA300:    (EMU1)   DATA3        R  2341.00     289.57       0.00    2361.00

 

It's like the new disk has stolen all the bandwidth... :-(

 

Volker Halle
Honored Contributor

Re: Extremely slow backup to LTO Tape after replacement of hot plug (shadowed) internal SCSI disk

Mark,

 

It's like the new disk has stolen all the bandwidth... :-(

 

Or all the lower SCSI IDs have some problem ? How about the performance of $3$DKA400: ?

 

The experiments with changing Read Cost have shown, that this would be a valid, quick and safe workaround to decrease your backup runtime, until this problem can really be solved.

 

Volker.

MarkOfAus
Valued Contributor

Re: Extremely slow backup to LTO Tape after replacement of hot plug (shadowed) internal SCSI disk

Volker,

 

"Or all the lower SCSI IDs have some problem ? How about the performance of $3$DKA400: ?"

 

Nope. I recall I tested it. It came up as 81. A tad better than 76 (I'm an optimist).

 

                            OpenVMS Monitor Utility
                              DISK I/O STATISTICS
                                 on node EMU1
                             8-AUG-2014 19:52:06.38

I/O Operation Rate                         CUR        AVE        MIN        MAX

$3$DKA0:      (EMU1)   ALPHASYS           4.00       1.10       0.00      24.66
$3$DKA100:    (EMU1)   DATA1              0.00      36.45       0.00      76.00
$3$DKA200:    (EMU1)   DATA2              0.00       8.28       0.00      76.66
$3$DKA300:    (EMU1)   DATA3              0.66     551.49       0.00    2834.33
$3$DKA400:    (EMU1)   ADMIN             21.33       7.75       0.00      81.00
DSA1:                  DATA1              0.00      36.45       0.00      76.00
DSA2:                  DATA2              0.00       8.28       0.00      76.66
DSA3:                  DATA3              0.66     548.33       0.00    2834.33
$4$DKB100:    (EMU2)   DATA1        R     0.00       0.00       0.00       0.00
$4$DKB200:    (EMU2)   DATA2        R     0.00       0.00       0.00       0.00
$4$DKB300:    (EMU2)   DATA3        R     0.66      11.09       0.00      50.33

 

I really appreciate all your help, Volker. Thank you.

 

 

I will use your suggestion of setting the read cost appropriately to preference the remote disks. That is a great solution/workaround.

 

(If the moderators and yourself, Volker, don't mind, I will leave this as unsolved and await the resolution of the case from HP to hopefully resolve this conundrum).

 

Cheers,

Mark.

 

Just a note for anyone reading this, the command to reset the read cost is:

 

SET DEVICE/READ=1 DSA1:

 

You may specify any read cost because the O/S ignores it and resets the read costs for each device in the shadow set to its default.

 

Volker Halle
Honored Contributor

Re: Extremely slow backup to LTO Tape after replacement of hot plug (shadowed) internal SCSI disk

Mark,

 

now if ALL disks on that SCSI bus EXCEPT the newly replaced one perform badly after the replacement, you might also have to ask yourself: is the replacement disk $3$DKA300 itself causing the problems ? Is that disk of the SAME type as the previous one ?

 

Volker.

MarkOfAus
Valued Contributor

Re: Extremely slow backup to LTO Tape after replacement of hot plug (shadowed) internal SCSI disk

Hi Volker,

 

"now if ALL disks on that SCSI bus EXCEPT the newly replaced one perform badly after the replacement, you might also have to ask yourself: is the replacement disk $3$DKA300 itself causing the problems ?"
 
Indeed.
 
Perhaps, come Monday morning (it's Saturday morning here now), I could take that disk back out of the shadow set and see what happens running some backup tests?
 
 
"Is that disk of the SAME type as the previous one ?"
 
NO. When HP sent me the replacement the box contained a note stating this disk is a comparable replacement to the disk that has failed. It was a different disk. It had different sectors per track and tracks per cylinder but the general capacity listed as 36GB was the same.
 
Our contract specifies ours are 3R-A3079-AA, but the replacement was not that. I checked this even before I saw the note in the bottom of the box.
 
What's more the normal procedure for returning the bad disk was skipped; they didn't want it. To quote the person:
"The part is a non returnable therefore we no longer require the part to be returned."
 
Cheers,
Mark
Dennis Handly
Acclaimed Contributor

Re: Extremely slow backup to LTO Tape after replacement of hot plug (shadowed) internal SCSI disk

>I attempt to attach the .csv from T4 to which this junk software says:
>"The contents of the attachment doesn't match its file type."

 

I also get that attachment error and Excel likes that file fine.   Your .zip solution is a good workaround.

Forum problems should be reported here:

http://h30499.www3.hp.com/t5/Community-Feedback-Suggestions/bd-p/community-feedback-suggestions

 

 

>If ... Volker, don't mind, I will leave this as unsolved ...

 

You can still assign Kudos to each helpful answer by clicking on the Kudos star.

Hoff
Honored Contributor

Re: Extremely slow backup to LTO Tape after replacement of hot plug (shadowed) internal SCSI disk

What specific sort of disk is it?  SHOW DEVICE /FULL $3$DKA300: will usually give some sort of an identity for the device, otherwise there are some SCSI-level tools to gather information.

 

With V7.3-2, you have access to dissimilar device shadowing (DDS) and dynamic volume expansion (DVE), two features that are worth enabling when you next have some scheduled downtime or can otherwise dismount the shadowsets briefly.

 

36 GB disks are ancient, too.  I've been scrounging new-old-stock 146 GB SCSI disk drives for ~US$40 for a while now.  Amazon has available HP 72 GB 15K SCSI disks in what is likely the appropriate "Universal" disk sled for that AlphaServer DS25 for ~US$15, with free shipping.  Several different 146 GB 15K HP drives on offer for ~$37 to $40 over there, too.  

 

I'd look to patch to current V7.3-2 (or to upgrade), enable DDS/DVE on the disks, and start a migration to those 72 GB or 146 GB drives, and then let HP know you've upgraded your storage for purposes of the support contract.

 

MarkOfAus
Valued Contributor
Solution

Re: Extremely slow backup to LTO Tape after replacement of hot plug (shadowed) internal SCSI disk

The situation was finally resolved.

 

I placed a support call and after some procrastination by the people at HP that do such diagnosis they came to the conclusion that the tape drive controller needs replacing. What?!?!?!

 

This was because they saw errors logged in ERRLOG.SYS at the time I hot-swapped the drive. This was DESPITE the fact I told them, and showed them, the evidence I assembled above indicating all BUT the new disk are performing badly.

 

 

A HP Engineer came to the site bearing a replacement SCSI controller for the Tape drive.

 

Before he replaced the card he said the hot-swap was undoubtedly the problem. Other systems with RAID controllers we have are more forgiving of hot-swaps. Even though the hardware supports hot-swap, it seems OpenVMS doesn't necessarily behave well afterwards (I am paraphrasing the engineer).

 

So, a reboot solved the problem.

 

 

Here is the final output from running BACKUP/PHY/NOCRC/GROUP=0 disk: NLA0:x.x/SAVE  on all disks in the system after a system reboot:

 

                             OpenVMS Monitor Utility
                               DISK I/O STATISTICS
                                  on node EMU1
                             14-AUG-2014 00:03:54.70

I/O Operation Rate                         CUR        AVE         MIN        MAX

$3$DKA0:      (EMU1)   ALPHASYS        2403.66     272.33        0.00    2404.00
$3$DKA100:    (EMU1)   DATA1              0.00     429.30        0.00    2519.33
$3$DKA200:    (EMU1)   DATA2              0.00     297.22        0.00    2526.33
$3$DKA300: (EMU1) DATA3 34.00 341.23 0.00 2674.33 $3$DKA400: (EMU1) ADMIN 0.00 428.48 0.00 3497.00 DSA1: DATA1 0.00 429.30 0.00 2519.33 DSA2: DATA2 0.00 297.22 0.00 2526.33 DSA3: DATA3 34.00 341.14 0.00 2674.33 $4$DKB100: (EMU2) DATA1 R 0.00 0.15 0.00 12.00 $4$DKB200: (EMU2) DATA2 R 0.00 0.28 0.00 5.00 $4$DKB300: (EMU2) DATA3 R 34.00 24.63 0.00 50.33

 

So, basically, a disk can be hot-swapped into the DS25E (using the on-board motherboard controller) but a reboot needs to be scheduled for everything to get back to normality.

 

A partial solution, until the reboot can be scheduled, and to reduce the slowness of a backup to tape is to change the read_cost of the disks that make up the shadow set to bias them towards the remote server.

 

Thank you to Volker for his invaluable help and partial solution/interim work-around.