MSA1000 - How to tell if upgrading to a 512K cache will help?

[Carson Hovey]

How do I determine if adding the optional 256K cache module to my MSA1000 will help my performance?

Is there any info on the "show perf" and "show cacheinfo" commands? The manuals do not say much.

I've attached the output from "show perf" and "show cacheinfo" which was taken during my nightly build.

[Carson Hovey]

Hmmm... I was not able to view my attachment so I'll attempt to paste it to this window using the "retain format(spacing) feature:

SysMgmt4000> show perf

Sample interval: 17143.7s
Structure version: 101
% CPU utilization: 46%
Command count: 16245535
Avg Command Latency: 0.65ms
% Time Active Reqs: 32.7%
Max Outst Cmd Lists: 261
Logical Req Count: 16245535
Avg Free Phys Reqs: 1228
Avg Free Log. Reqs: 843
Avg Free Xbuf Sects: 0
Avg Lrgst Avail Xb S: 0
Avg Free CPU DRAM KB: 0
Max DMA Xfer Q Depth: 0
% Time DMA Xfr Stall: 0.0%
Avg Locked Stripes: 0
Avg Stripes Waiting: 0
Avg Completions/Sec: 947.6
Avg Log Req Comp/Sec: 947.6

Logical volume data: (All values in hex except Vol and AvQDep)
Vol LWrites LReads AvQDep CHits CMiss RACount LSectRd LSectWr
0 9579C F0ABF 0 DFC28 12865 F11F60 3812F6 11F5E4
1 9 2FF7 0 811 27E6 2580 81FE0 9
2 9 506 0 2FB 20B E0 50F 9
3 1ADC1B 1E6DFD 0 1CD97A 1A877 1F8C380 4D120D 213BCC7
4 9 506 0 2FB 20B E0 515 9
5 9 3E5 0 47 39E 60 254F6 9
6 0 0 0 0 0 0 0 0
7 0 0 0 0 0 0 0 0
14 11E4 463B 0 4125 516 34A0 CE10 19E6
15 0 0 0 0 0 0 0 0
16 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0
26 7506B6 30588A 0 2D3437 4A488 93DF20 9C6687 5102873
27 10F 1CD 0 1BB 12 0 747 18D
28 10B 66E 0 195 4DF 0 13413 189

SysMgmt4000> show cacheinfo
00014295 LINES: NOT SET
00000007 LINES: LINE_LOCK
00000002 LINES: LINE_DIRTY
00000010 LINES: FLUSH_LOCK LINE_DIRTY
Minimum DMA CDB count since last check: 622

[Jefferson Humber]

Carson,

I go by comparing the ratio of CHits to CMiss (Cache Hit to Cache Miss).

You will need to convert the values from hex though, so use a good calculator.

A high percentage of CHits shows a good utilisation of cache. You could argue that if this figure is low then more cache may help, but it really depends on the nature of your application.... is it random data accessing ?

Hope this helps,

Jeff

[John Kufrovich · Accepted solution]

Carson,
In that one cacheinfo snapshot, you are not using your cache.

This could be because your servers are not hitting the MSA.

Run it a couple of more times, try it during your peak MSA useage.

[Carson Hovey]

Why do you say I'm not using my cache? CHits is 6D9EE1 (on Vol 0). Or do you mean my CMiss is high 12886F (on Vol 0).

Here's another snapshot where perf has been running a little longer. My "application" is a C++ compiles, links and running some short/small regression tests. The same sources are built debug/nodebug on Alpha and IA64. One of my system disks is on the MSA as well.

Also, can someone explain the output from "cacheinfo"?

Sample interval: 75013.1s
Structure version: 101
% CPU utilization: 43%
Command count: 60434444
Avg Command Latency: 0.79ms
% Time Active Reqs: 31.6%
Max Outst Cmd Lists: 264
Logical Req Count: 60434444
Avg Free Phys Reqs: 1228
Avg Free Log. Reqs: 843
Avg Free Xbuf Sects: 0
Avg Lrgst Avail Xb S: 0
Avg Free CPU DRAM KB: 0
Max DMA Xfer Q Depth: 0
% Time DMA Xfr Stall: 0.0%
Avg Locked Stripes: 0
Avg Stripes Waiting: 0
Avg Completions/Sec: 233.0
Avg Log Req Comp/Sec: 233.0

Logical volume data: (All values in hex except Vol and AvQDep)
Vol LWrites LReads AvQDep CHits CMiss RACount LSectRd LSectWr
0 C52CF5 7B2577 0 6D9EE1 12886F A3A9720 1A407A2 8E2852F
1 538E 45339 0 38FCA C36F 3A680 247EFC 5B41
2 9 535 0 323 212 E0 542 9
3 8E9192 B0543C 0 A43EC9 ECB88 D8DAA40 1F5AECB 92DEC2D
4 9 535 0 323 212 E0 54C 9
5 9 40C 0 6D 39F 60 2551D 9
6 0 27 0 26 1 0 27 0
7 0 27 0 26 1 0 27 0
14 11E4 4662 0 414B 517 34A0 CE37 19E6
15 0 27 0 26 1 0 27 0
16 0 27 0 26 1 0 27 0
17 0 27 0 26 1 0 27 0
18 0 27 0 26 1 0 27 0
19 0 27 0 26 1 0 27 0
20 0 27 0 26 1 0 27 0
21 0 27 0 26 1 0 27 0
22 0 27 0 26 1 0 27 0
23 0 27 0 26 1 0 27 0
24 0 27 0 26 1 0 27 0
25 0 27 0 26 1 0 27 0
26 AD568C 67AD99 0 6000B5 999BE 29F45A0 12BFAA2 7B356AB
27 10F 1F4 0 1E1 13 0 76E 18D
28 10B 695 0 1BB 4E0 0 1343A 189

SysMgmt4000> show cacheinfo
00014300 LINES: NOT SET
00000007 LINES: LINE_LOCK
00000004 LINES: READ_AHEAD_LOCK
00000001 LINES: LINE_DIRTY
00000002 LINES: FLUSH_LOCK LINE_DIRTY
Minimum DMA CDB count since last check: 473

[John Kufrovich]

Perhaps my statement you are not using your cache is wrong. You are using the cache but your server is not sending enough data.

show cacheinfo
00014300 LINES: NOT SET
00000007 LINES: LINE_LOCK
00000004 LINES: READ_AHEAD_LOCK
00000001 LINES: LINE_DIRTY
00000002 LINES: FLUSH_LOCK LINE_DIRTY

A MSA cache line is 32 blocks or 16k in size. Look at your lines: NOT set. This represents the number of cache lines available. If you take 14300 * 16k, you get approximately your total cache size. The only thing I don't know is your cache ratio. The LINE: LINE_Dirty means lines waiting to be cached. Flush_Lock Line_Dirty represents lines that have been flushed and we are just waiting for a status good back from the drives indicating that the data has been written to the media.

[John Kufrovich]

Carson,
What type of applications are you running on your servers? eg. SQL, Exchange ?

[Carson Hovey]

My application is not SQL or exchange. I'm building a large application which means many small C++ compiles and links and running some small regression tests. The system disk (which has the compiler) is also out on the MSA. Builds for various operating systems run at the same time. Since this is not a database application maybe I'm not going to get a lot of cache hits. My hope was that directory headers and so forth would be cached to speed things up; but perhaps these get pushed out of the cache as the .cpp, .h files are accessed.

Perhaps for this "application" the read/write ratio should not be 50%?

As for the cacheinfo and my "server not sending enough data"... I ran the cacheinfo command in the morning after the nightly build had finished. Perhaps (unlike show perf) I need to run cacheinfo while the build is going on. Again, I don't have any documentation on cacheinfo so I don't know how to use it or what the output means.

Are you saying that to determine if I would benefit from the cache upgrade... I should not use "show perf" after the my application has been running for a while; but rather I should use "show cacheinfo" while my application is running and collect some snapshots of that information?

[Carson Hovey]

I forgot to answer your question about "cache ratio". Show Globals says it is 50% read/write.

Also, the OS is OpenVMS in case that matters.
===========================================
SysMgmt4000> show this_controller
Controller:
MSA1000(c) Compaq P56350HX3SM031 Software 4.48 Build 342 Hardware 7
Controller Identifer: 4000
NODE_ID = 500805F3-001A20C0
SCSI_VERSION = SCSI-3
Supported Redundancy Mode: Active/Standby
Current Redundancy Mode: Active/Standby
Current Role: Active
Device Port SCSI address 6
Terminal speed for the CLI is set to 19200.
Host Port_1:
REPORTED PORT_ID 500805F3-001A20C1
PORT_1_TOPOLOGY = F_Port
Cache:
128 megabyte read cache 128 megabyte write cache Version 2
Cache is GOOD, and Cache is enabled.
No unflushed data in cache
Battery:
Module #1 is fully charged and turned off.
Controller Up Time:
2 Days 00 Hours 39 Minutes 24 Seconds

SysMgmt4000> show globals

Global Parameters:
System Name: sysmgt4000
Rebuild Priority: medium
Expand Priority: medium

Total Cache: 256MB
50% Read Cache: 128MB
50% Write Cache: 128MB

Temperature:
EMU: 35 Celsius, 95 Fahrenheit
PS1: 40 Celsius, 104 Fahrenheit
PS2: 39 Celsius, 102 Fahrenheit
FIBRE BAY: 46 Celsius, 114 Fahrenheit

SysMgmt4000>

[Jefferson Humber]

Carson,

Am I right your running under OpenVMS ?

If so, do you have XFC enabled ?

Try a $SHOW MEMORY/CACHE , are you even hitting the MSA or has OpenVMS used it's own cache ?

Jeff

[Carson Hovey]

Yes, this is OpenVMS.

So it sounds like you are saying that I've got a software cache at the OS level (XFC) so I'm not making use of (do not need) a larger hardware cache on the MSA1000?

There are several systems accessing the MSA1000 and this command is from just one of them. And it doesn't show which disks are involved (some are on the MSA1000 and some are not).

$ sho mem/full/cache
System Memory Resources on 20-SEP-2006 15:20:18.32

Extended File Cache (Time of last reset: 18-SEP-2006 08:13:33.63)
Allocated (MBytes) 999.16 Maximum size (MBytes) 1023.81
Free (MBytes) 0.07 Minimum size (MBytes) 3.12
In use (MBytes) 999.08 Percentage Read I/Os 86%
Read hit rate 93% Write hit rate 0%
Read I/O count 68494204 Write I/O count 10240456
Read hit count 64334741 Write hit count 0
Reads bypassing cache 577 Writes bypassing cache 1097976
Files cached open 348 Files cached closed 4307
Vols in Full XFC mode 0 Vols in VIOC Compatible mode 59
Vols in No Caching mode 0 Vols in Perm. No Caching mode 0

I/O Statistics - Distributions (MAX_IO_SIZE: 127)
-------------------------------------------------
Transfer Size: Reads Read Hits Writes
1 Block IO: 14816418 14801252 590832
2 Block IO: 1812272 1807805 44779
3 Block IO: 198674 195748 52056
4 Block IO: 307083 306049 44631
5 Block IO: 1571912 1570172 31466
6 Block IO: 354652 353106 23236
7 Block IO: 425953 425087 21364
8 Block IO: 664549 663053 4009954
9 Block IO: 2618687 2617616 15692
10 Block IO: 583765 582820 664425
11 Block IO: 187469 186643 10235
12 Block IO: 340279 339379 11292
13 Block IO: 380036 378731 9639
14 Block IO: 617912 616965 8505
15 Block IO: 242732 242056 8215
16 Block IO: 1137744 1130424 11314
17 Block IO: 6902656 6889027 882030
18 Block IO: 300334 299052 7661
19 Block IO: 163766 162656 6744
20 Block IO: 591210 587623 8083
21 Block IO: 105844 105153 7283
22 Block IO: 116603 115834 6865
23 Block IO: 94560 93903 7184
24 Block IO: 127242 126229 6666
25 Block IO: 108165 107372 7857
26 Block IO: 1036868 1035995 7228
27 Block IO: 133607 132730 5932
28 Block IO: 463053 462251 6531
29 Block IO: 64145 63296 5699
30 Block IO: 113036 112245 133338
31 Block IO: 107227 106028 9802
32 Block IO: 10313766 6558330 2467244
33 Block IO: 31486 31259 1003
34 Block IO: 41598 41298 1023
35 Block IO: 40710 40389 1281
36 Block IO: 50325 49985 1480
37 Block IO: 35186 34859 1387
38 Block IO: 44356 43965 1341
39 Block IO: 43362 43132 1406
40 Block IO: 22785 22521 1266
41 Block IO: 41508 41073 918
42 Block IO: 45698 45216 947
43 Block IO: 41429 40964 889
44 Block IO: 26114 25768 837
45 Block IO: 34502 34110 817
46 Block IO: 3387 3174 917
47 Block IO: 43143 42822 1296
48 Block IO: 792054 782230 1035
49 Block IO: 54492 54225 937
50 Block IO: 180605 180396 1148
51 Block IO: 15681 15506 1174
52 Block IO: 12870 12700 976
53 Block IO: 49214 48830 1058
54 Block IO: 97844 97390 987
55 Block IO: 36643 36227 1244
56 Block IO: 138346 137990 1016
57 Block IO: 45717 45339 1381
58 Block IO: 42972 42584 1429
59 Block IO: 39836 39445 1012
60 Block IO: 40007 39583 960
61 Block IO: 34198 33708 1035
62 Block IO: 39431 38720 845
63 Block IO: 49701 48970 1419
64 Block IO: 9121063 9011350 34959
65 Block IO: 0 0 652
66 Block IO: 2 0 670
67 Block IO: 2 0 976
68 Block IO: 2 0 808
69 Block IO: 1 0 605
70 Block IO: 1 1 686
71 Block IO: 5 1 640
72 Block IO: 2 0 553
73 Block IO: 2 0 611
74 Block IO: 0 0 803
75 Block IO: 3 0 547
76 Block IO: 0 0 650
77 Block IO: 0 0 466
78 Block IO: 3 0 340
79 Block IO: 6 5 554
80 Block IO: 16 5 688
81 Block IO: 68 65 445
82 Block IO: 0 0 497
83 Block IO: 0 0 458
84 Block IO: 1 0 477
85 Block IO: 2 0 1007
86 Block IO: 0 0 428
87 Block IO: 0 0 564
88 Block IO: 2 0 517
89 Block IO: 1 0 414
90 Block IO: 1 0 316
91 Block IO: 8 6 277
92 Block IO: 5 4 579
93 Block IO: 0 0 691
94 Block IO: 1 0 293
95 Block IO: 0 0 289
96 Block IO: 14 6 373
97 Block IO: 1 0 345
98 Block IO: 0 0 235
99 Block IO: 0 0 257
100 Block IO: 10121342 9940116 469
101 Block IO: 0 0 644
102 Block IO: 1 0 723
103 Block IO: 2 0 313
104 Block IO: 0 0 355
105 Block IO: 1 0 514
106 Block IO: 1 0 632
107 Block IO: 0 0 401
108 Block IO: 0 0 532
109 Block IO: 8 7 548
110 Block IO: 1 0 493
111 Block IO: 0 0 643
112 Block IO: 14 5 467
113 Block IO: 0 0 411
114 Block IO: 1 0 430
115 Block IO: 0 0 573
116 Block IO: 2 0 615
117 Block IO: 6 3 566
118 Block IO: 0 0 541
119 Block IO: 1 0 724
120 Block IO: 1 0 964
121 Block IO: 0 0 533
122 Block IO: 1 0 796
123 Block IO: 0 0 1076
124 Block IO: 31649 12748 27800
125 Block IO: 0 0 698
126 Block IO: 0 0 682
127 Block IO: 2112 1411 1088

>127<256 Block IO: 113 974309
>255 Block IO: 317 0
-------------------------------------------------
Totals: 68494204 64334741 10240456

[Jefferson Humber]

A 'Read Hit' ratio of 93% in XFC is pretty high. ;-)

It's possible that this is what is happening, you are certainly making good use of the OpenVMS Extended File System cache. Of course this will only benefit this host, other boxes connected with the MSA will not benefit obviously.

Jeff

[John Kufrovich]

Carson,
It appears you are trying to do reads to the MSA and hoping the header files will be in the MSA cache.

To get really good read performance, the OS will need to send multiple commands to the MSA. Sending a single read request will not help. While you are compiling, issue the command >show taskstats

Like the show cacheinfo this command takes a snapshot of how many commands the MSA is currently working. If the commands is progress is low, then your performance could be low.