topic Re: Read Only File Systems in Operating System - Linux

Read Only File Systems

Carlos A. Munoz Lopez — Fri, 26 Mar 2010 17:18:25 GMT

Hi guys. I have configured an EVA 4400 (FW Ver. 09522000) connected to five BL460 running SuSE Enterprise 11 and two BL680 running Windows 2003 R2. Since I've been having problems with Command View (can't detect the attached EVA) I decided to restart both controllers (one at a time) based on some suggestions found in different threads in the forum. First, I restarted controller B and suddenly all file systems mounted on Linux servers reported a Read-Only mode, obviously this caused problems in databases running on the systems. Once the controller finished restarting, the file systems kept reporting this state, I had to reboot all Linux servers so they could restore the Read-Write mode.

This is the messaged displayed in system log:

Mar 25 18:32:26 dbserver kernel: sd 0:0:0:3: [sdc] Attached SCSI disk
Mar 25 18:32:26 dbserver kernel: sd 0:0:0:3: Attached scsi generic sg9 type 0
Mar 25 18:32:26 dbserver kernel: sd 1:0:0:2: [sdh] Write cache: disabled, read cache: enabled, supports DPO and FUA
Mar 25 18:32:26 dbserver kernel: sd 1:0:0:2: [sdh] 524288000 512-byte hardware sectors: (268 GB/250 GiB)
Mar 25 18:32:26 dbserver kernel: sd 1:0:0:2: [sdh] Write Protect is off
Mar 25 18:32:26 dbserver kernel: sd 1:0:0:2: [sdh] Mode Sense: 97 00 10 08
Mar 25 18:32:26 dbserver kernel: sd 1:0:0:2: [sdh] Write cache: disabled, read cache: enabled, supports DPO and FUA

Multipath software is enabled in all operating systems. Actually, multipath reported a loss in one of the paths and continued accesing the file systems through the remaining active path, but in read-only mode. I consider this is not a normal condition, since I've been working with EVA arrays (but attached to UNIX systems) and I never experienced this kind of scenario.

One interesting remark, the Windows servers didn't suffer any interference on the disks, the OS only reported a loss in one of the path and then acknowledged the restored path.

I would like your help on this issue and tell me what caused this condition. Thanks in advanced.

Re: Read Only File Systems

John Poff — Fri, 26 Mar 2010 17:57:31 GMT

Hi,

Are the Linux filesystems in LVM? If so, are the volume groups using the multipath devices or the /dev/sd* devices?

JP

Re: Read Only File Systems

Carlos A. Munoz Lopez — Fri, 26 Mar 2010 18:32:48 GMT

Yes. I'm using LVM and VGs are configured with multipath devices.

Re: Read Only File Systems

Michael Steele_2 — Sat, 27 Mar 2010 21:09:31 GMT

HI

In Red Hat, when a file system goes read only its because of a bad disk.

Since multipath reported only one active path, I would assume the other has a HW problem. Probably an HBA on the Unix server since none of the windows servers didn't suffer.

I'm guessing that in Suse you'll find HBA status as you would for Red Hat, under /pro/scsi/qla#### for qlogic FC HBA. Let me know if you have Emmulex HBA's or something else.

Re: Read Only File Systems

Kapil Jha — Mon, 29 Mar 2010 01:54:50 GMT

I faced same problem smtime back and as far as I remember, ext3 would put the file system to read-only mode when there is any further chances to damage the file system.

this include busy I/O retry or SAN path failover errors.
and Redhat suggested to upgrade the kernel.
What kernel you running.
This problem is in most of the LINUX flavours I suppose.

BR,
Kapil+

Re: Read Only File Systems

Vipulinux — Mon, 29 Mar 2010 08:56:13 GMT

Hi

When the filesyetem is not confugred properly to be accessed form mor ethan 1 location; there are bound to be issues wherein the file system becomes read only. Make sure that you configure it properly if it is to be used from 2 locations.

Cheers
Vipul

Re: Read Only File Systems

Carlos A. Munoz Lopez — Mon, 29 Mar 2010 14:48:24 GMT

When you say file system not properly configured, I think this is the job of multipath software and its configuration file. Following is the multipath configuration for EVA 4400 (HSV 300):

vendor "HPâ
product "HSV2[01]0|HSV300|HSV4[05]0"
getuid_callout â /lib/udev/scsi_id -g -u /dev/%nâ
prio_callout alua
hardware_handler "0"
path_selector "round-robin 0"
path_grouping_policy group_by_prio
failback immediate
rr_weight uniform
no_path_retry 18
rr_min_io 100
path_checker tur

As far as I know, when you have multipath implemented, if one the path fails the file systems should continue to be accesible (through the remaining active path), this is one the purpose of multipath, at least this is the case in UNIX environments (HP-UX, Solaris, AIX, etc.).

Once you customize the multipath configuration template, there is nothing else to do in the file systems.

Re: Read Only File Systems

D Block 2 — Tue, 30 Mar 2010 02:54:08 GMT

sounds like a path problem to command-view. both eva controllers look good ? is this the first concurrence of this problem ? if you focus on getting fiber from command-view to controllers, this too would solve the linux patthing issue. sounds like linux is not running multi-path or is the daemon for dm running -- check the listings from the dm status to see both paths in linux.

are the windows severs running the mpio product or not? if not, then they are single path to the one eva controller -- which one seems to have the link lights on the ports flashing when database is running ?

maybe a visual view might help. maybe you have lost command-view from seeing the array, so its not just linux..

Re: Read Only File Systems

Carlos A. Munoz Lopez — Tue, 30 Mar 2010 14:00:56 GMT

Multipath and dm daemon are running. Multipath shows both active paths, when failure occurred it only showed one path. Windows servers are running also MPIO.

The Command View problem is, according to some threads in the forum, related to a bug in the controller firmware and it should be solved with and upgrade. I even presented a virtual disk to the Command View server, and it worked, so the problem is not the path or zoning, the problem is just with the Linux servers.

Re: Read Only File Systems

skt_skt — Wed, 31 Mar 2010 08:51:25 GMT

here is how we tuned scsiid_config, lvm.conf and multipath.conf, but with EMC Clariion.

~]# cat /etc/scsi_id.config |grep -v '#'
options=-g
vendor="DGC",options=-p 0x83

Also tuned lvm filter and multipath.conf

#cat /etc/lvm/lvm.conf |grep -v '#' |grep -e 'filter' -e 'types'
filter = [ "a|/dev/mpath/.*/|","a|/dev/mapper/.*|", "a|/dev/sda|","r|/dev/sd[b-z]|", "r/.*/" ]
types = [ "device-mapper", 1]

#cat /etc/multipath.conf |grep -v '#'
devnode_blacklist {
devnode "^sda"
}

defaults {
user_friendly_names yes
}

multipaths {
multipath {
wwid 360060160f1731100c642e85121b2dd11
alias mpath0
}
multipath {
wwid 360060160f173110012f68b1e56b3dd11
alias mpath1
}
multipath {
wwid 360060160f17311000c55033921b2dd11
alias mpath2
}
multipath {
wwid 360060160f1731100b254eb42b9b4dd11
alias mpath3
}
}