SOLVED
Volker,
what's slowing down the backup now ? - slower writes to tape ? - slower reads from disks ? - slower intersite link ?
My friend, if I knew that, the problem would be solved.
We back up from the main server (DS25) so I assume (correctly?) that all disk access is local if there are served disks in the shadow set that are local, which there are.
Therefore, I have only T4 data at hand for the DS25.
If you look at the T4 data for the disks (during backup), do you now see larger than usual queues '[MON.DISK]Qln' on the local or remote mbrs ? Or larger Response times?
The mon.disk qlen is abnormally high for the duration of 10 hours the backup is running. I will attach the 26th July T4 (before the disk replacement) and the T4 after the disk replacement, in this case I've picked 5-aug-2014.
You can see for yourself the stark difference. The backup normally runs from midnight till about 3am, as it did on the 26th July. Compare that to the backup on the 5th August, where the Qlen is high from midnight until 10:30am when it finally finishes (or I think it was killed off because of impact on the system - I have to find that out).
The disk oprate is normally a spike for 2-3 hours, but now it's a steady sprawl over 10 hours.
What I do to the test read throughput from disks is: $ SPAWN/NOWAIT BACKUP/PHY/NOCRC/GROUP=0 disk: NLA0:x.x/SAVE then $ MONI DISK and look at the Dirio-Rate You could do this for the shadowsets and also for individually mounted disks from the local and remote site.
I will get back to you after running this (I need to wait for the backup to finish before clobbering the system more...).
Then try a BACKUP/PHYSICAL from a local disk to tape. Any resource problems (nonpaged pool) on the node running backup ?
It would be odd (far too eerily coincidental) that a change in disk would correspond with that.
Anyway, sysgen reports:
Parameter Name Current Default Min. Max. Unit Dynamic -------------- ------- ------- ------- ------- ---- ------- NPAGEDYN 39993344 1048576 163840 -1 Bytes NPAGEVIR 156827648 8388608 163840 -1 Bytes NPAG_BAP_MIN 40960 0 0 -1 Bytes NPAG_BAP_MAX 131072 0 0 -1 Bytes NPAG_BAP_MIN_PA 0 0 0 -1 Mbytes NPAG_BAP_MAX_PA 1024 -1 0 -1 Mbytes NPAG_RING_SIZE 2048 2048 0 -1 Entries NPAGECALC 0 1 0 2 Coded-valu NPAGERAD 0 0 0 -1 Bytes NPAG_INTERVAL 30 30 0 -1 Seconds D NPAG_GENTLE 85 85 0 100 Percent D NPAG_AGGRESSIVE 50 50 0 100 Percent D SYSGEN>
Backup is running now so here tis:
...$COMMON:COMMON.ADMIN> SHOW MEMORY/POOL/FULL
System Memory Resources on 8-AUG-2014 10:20:22.33
Nonpaged Dynamic Memory (Lists + Variable)
Current Size (MB) 38.14 Current Size (Pagelets) 78112
Initial Size (MB) 38.14 Initial Size (Pagelets) 78112
Maximum Size (MB) 149.56 Maximum Size (Pagelets) 306304
Free Space (MB) 17.96 Space in Use (MB) 20.17
Largest Var Block (MB) 10.11 Smallest Var Block (bytes) 64
Number of Free Blocks 9941 Free Blocks LEQU 64 bytes 2017
Free Blocks on Lookasides 5123 Lookaside Space (MB) 1.85
Bus Addressable Memory (Lists + Variable)
Current Size (KB) 128.00 Current Size (Pagelets) 256
Initial Size (KB) 128.00 Initial Size (Pagelets) 256
Free Space (KB) 110.87 Space in Use (KB) 17.12
Largest Var Block (KB) 104.00 Smallest Var Block (KB) 6.87
Number of Free Blocks 2 Free Blocks LEQU 64 bytes 0
Free Blocks on Lookasides 0 Lookaside Space (bytes) 0
Paged Dynamic Memory
Current Size (MB) 7.68 Current Size (Pagelets) 15744
Free Space (MB) 4.17 Space in Use (MB) 3.50
Largest Var Block (MB) 4.12 Smallest Var Block (bytes) 16
Number of Free Blocks 227 Free Blocks LEQU 64 bytes 198
Lock Manager Dynamic Memory
Current Size (MB) 56.89 Current Size (Pages) 7283
Free Space (MB) 50.73 Hits 38995840
Space in Use (MB) 6.16 Misses 0
Number of Empty Pages 6228 Expansions 7283
Number of Free Packets 207799 Packet Size (bytes) 256
Write Bitmap (WBM) Memory Summary
Local bitmap count: 0 Local bitmap memory usage (bytes) 0.00
Master bitmap count: 0 Master bitmap memory usage (bytes) 0.00
...$COMMON:COMMON.ADMIN>
Some timings (this is just indicative - other things impact it, I know):
DSA1: 144GB used, takes 5:25 hours to backup (lots of small files, large files (300 MB each) as well)
DSA2: 73GB used, takes 4:51 hours to backup (lots of small user files on this disk)
DSA3: 16GB used, takes 16minutes - system shared software - UAF/security/rightslist/queues etc.
DKA400: 5GB used - ODS-5 (all the rest of the disks are ODS-2) - Oracle Client Disk. Takes 49 minutes to backup. Non-shadowed. Lots of small files and other detritus Oracle places there.
Thanks for your assistance.
(This forum software is rubbish. I set my included output to Courier new and it removes it!)
(I also attempt to attach the .csv from T4 to which this junk software says:
"The contents of the attachment doesn't match its file type.")