"Unwritable cache data exists for a volume" - this means new writes came to Controller cache but not yet written to backend drives because Vdisk is not available at that point. It could be vdisk was quarantined or offline or QTOF state.
Somewhere you have mentioned that you have cleared Cache data which is a big mistake because this data you lost.
Coming to why vdisk reconstruction failed. This could be due to other drives in same vdisk having hardware errors which is not allowing the vdisk rebuild process to complete. This means MSA controller not able to read from some other drives part of same vdisk to reconstruct RAID data. In RAID 5 more than one drive having issue or failed means it's RAID failed and data recovery difficult.
In future I would always suggest to keep your MSA up to date in terms of firmware with respect to all components like Controllers, IO modules and drives. Also this type of case you should always take help from HPE support.
Right now I would suggest to figure out other faulty drive and delete the vdisk. Then replace all faulty drives together. Create new vdisk and volumes as per your requirement. Then restore data from backup.
Hope this helps!
Regards
Subhajit
I am an HPE employee
If you feel this was helpful please click the KUDOS! thumb below!
***********************************************************************************
I work for HPE
I am not sure how much you can decode the store.logs file. You can try unzip the file and open with Notepad++ then take a look for all drives serial number which are currently present in the system. You can also get the drive serial number from SMU or GUI or CLI as well. Take each drive serial number and search. In results window look for medium error or hard error of the drive. To be more specific look for sense key. To understand more about sense you can refer the below guide,
https://en.wikipedia.org/wiki/Key_Code_Qualifier
You can also log a case with HPE support and review the logs to find drive errors because we have Internal tool to decode the logs.
We can also look for vdisk level error as well which may get captured as part of scrub process.
If you don't have further query then kindly mark this as resolved so that others can get the update who are following this topic.
Hope this helps!
Regards
Subhajit
I am an HPE employee
If you feel this was helpful please click the KUDOS! thumb below!
***********************************************************************************
I work for HPE
Hi Adam,
One thing you might consider is to create the new vdisk using offline init.
mode online|offline
Optional. Specifies whether the vdisk is initialized online or offline.
• offline: You must wait for the vdisk initialization process to finish before using the vdisk;
however, offline takes less time to complete initializing than online. At the time of creation, a
vdisk using offline initialization can have either one volume or none. If you want the vdisk to have
more than one volume, create the vdisk with no volumes and then add volumes after initialization
is complete.
This will move any bad blocks on the drives to the "do not use" list. Additionally, if there is an additional drive that is marginal for available block it will flag the drive as bad and remove it from use. It will take a little longer to lay down the parity and stripes for the vdisk, but might help ensure that once the init is completed you are not using a drive with potential problems.
Cheers,
Shawn
I work for Hewlett Packard Enterprise. The comments in this post are my own and do not represent an official reply from HPE. No warranty or guarantees of any kind are expressed in my reply.
I work for HPE