MSA30-SB Sudden Slowdown

[cam9269]

Hi Gurus,

I'd like to seek assistance in diagnosing an issue which happend recently. Disk i/o's began to drastically slowdown while we were doing backup/restore tests.

Config info:
- MSA30-SB, with 6x146GB 15K (RAID 1+0)+ 1x146GB 15K hotspare
- Smart Array 6402 SCSI card, 256MB cache
- RP4440-8, 16GB, 4-core CPU 1GHz

When I did some testings, I found that 'avwait' on 'sar -d' peaked to 18000, averaging to ~11000. CPU was at a 95% ave. idle value.

I tried to look for defunct processes and killed them which did not actually help.

The last thing I did was reboot the server, which seem to have 'cleaned' whatever it was which gave me high i/o. This was confirmed when I did the tests that I did before the reboot - true enough 'avwait' was nowhere near
the values above.

My problem is, how do I properly diagnose which might have caused this - if ever it does happen again. What are the things that I need to keep tags on so that i could better understand what was happening and prevent it.

Btw, there were no error logs on 'syslog' nor from 'sautil', cache batteries are OK also, which could refer to a HW failure

Regards

[John Kufrovich]

Cam,

Have you tried to update the HDD FW?

jk

[cam9269]

Hi JK,

I haven't, do you have procedures that would tell me:

a. how to check current FW version
b. how to update FW version

Thanks for replying.

[cam9269]

Hi Guys, would really appreciate a bit of help here, the client's are after recommendations and root cause which I am not able to provide at the moment.

thanks!

[Shweta Verma · Accepted solution]

Hello cam9269 ,

Disk firmware for an MSA30 enclosure can be checked by the following commands:

1. Check the device file associated with the smart array controller using the following command:

ioscan -kfnd ciss

2. Enter the sautil -s , to get more information about the smart array controller and the scsi disks attached to that Smart Array Controller and also the disk current firmware levels.

Disk firmwares can be updated using STM or sautil command. It is easier to use STM , if the disk firmware files have a .frm extension.

Thanks,
Shweta

[cam9269]

Thanks for the info Shweta, at this time, I not yet thinking of updating HDD FWs bec.
1. I have no data yet telling me that this will be the solution to this problem we have
2. any other diagnosis that I can use as support to tell my customers which can fix this

As of this time, i/o is back to normanl, but i still dread the day when we will encounter the same problem and Go-live is approaching.

[cam9269]

Hi Guys,

I still need to explain how this happened, any ideas? I'm not even sure if I would go for an HDD FW upgrade, the fact that I do not know what caused it.

really need this one sorted out.

thanks for the help.