vmunix: DIAGNOSTIC SYSTEM WARNING:

[pfwhufc]

I have the following alerts in syslog: -

Feb 25 12:49:42 ukxxxx vmunix: DIAGNOSTIC SYSTEM WARNING:
Feb 25 12:49:42 ukxxxx vmunix: The diagnostic logging facility is no longer r
eceiving excessive
Feb 25 12:49:42 ukxxxx vmunix: errors from the I/O subsystem. 1 I/O error en
tries were lost.
Feb 25 12:49:42 ukxxxx vmunix: LVM: Performed a switch for Lun ID = 0 (pv = 0x00
000000426ba000), from raw device 0x1f0a6400 (with priority: 0, and current flags
: 0x40) to raw device 0x1f0c6400 (with priority: 1, and current flags: 0x0).
Feb 25 12:49:42 ukxxxx vmunix: LVM: Restored PV 44 to VG 2.
Feb 25 12:49:52 ukxxxx vmunix: LVM: Performed a switch for Lun ID = 0 (pv = 0x00
000000425eb000), from raw device 0x1f0a4000 (with priority: 0, and current flags
: 0x40) to raw device 0x1f0c4000 (with priority: 1, and current flags: 0x0).
Feb 25 12:49:56 ukxxxx vmunix: LVM: Performed a switch for Lun ID = 0 (pv = 0x00
0000004258a000), from raw device 0x1f0a1600 (with priority: 0, and current flags
: 0x40) to raw device 0x1f0c1600 (with priority: 1, and current flags: 0x0).
Feb 25 12:49:56 ukxxxx vmunix: LVM: Performed a switch for Lun ID = 0 (pv = 0x00
000000425e2000), from raw device 0x1f0a3600 (with priority: 0, and current flags
: 0x40) to raw device 0x1f0c3600 (with priority: 1, and current flags: 0x0).
Feb 25 12:49:56 ukxxxx vmunix: LVM: Performed a switch for Lun ID = 0 (pv = 0x00
000000426c2000), from raw device 0x1f0a6600 (with priority: 0, and current flags
: 0x40) to raw device 0x1f0c6600 (with priority: 1, and current flags: 0x0).
Feb 25 12:50:10 ukxxxx vmunix: LVM: Path (device 0x1f0a1000) to PV 0 in VG 2 Fai
led!
Feb 25 12:50:10 ukxxxx vmunix: LVM: Path (device 0x1f0a1100) to PV 1 in VG 2 Fai
led!
Feb 25 12:50:10 ukxxxx vmunix: LVM: Path (device 0x1f0a1200) to PV 2 in VG 2 Fai
led!
Feb 25 12:50:10 ukxxxx vmunix: LVM: Path (device 0x1f0a1300) to PV 3 in VG 2 Fai
led!
Feb 25 12:50:10 ukxxxx vmunix: LVM: Path (device 0x1f0a1400) to PV 4 in VG 2 Fai
led!
Feb 25 12:50:10 ukxxxx vmunix: LVM: Path (device 0x1f0a1500) to PV 5 in VG 2 Fai
led!
Feb 25 12:50:10 ukxxxx vmunix: LVM: Path (device 0x1f0a1700) to PV 7 in VG 2 Fai
led!
Feb 25 12:50:10 ukxxxx vmunix: LVM: Path (device 0x1f0a2000) to PV 8 in VG 2 Fai
led!
Feb 25 12:50:10 ukxxxx vmunix: LVM: Performed a switch for Lun ID = 0 (pv = 0x00
000000425ba000), from raw device 0x1f0a2000 (with priority: 0, and current flags
: 0x40) to raw device 0x1f0c2000 (with priority: 1, and current flags: 0x0).
Feb 25 12:50:10 ukxxxx vmunix: LVM: Path (device 0x1f0a2100) to PV 9 in VG 2 Fai
led!
Feb 25 12:50:10 ukxxxx vmunix: LVM: Path (device 0x1f0a2200) to PV 10 in VG 2 Fa
iled!
Feb 25 12:50:10 ukxxxx vmunix: LVM: Performed a switch for Lun ID = 0 (pv = 0x00
000000420d2000), from raw device 0x1f0a2200 (with priority: 0, and current flags
: 0x40) to raw device 0x1f0c2200 (with priority: 1, and current flags: 0x0).
Feb 25 12:50:10 ukxxxx vmunix: LVM: Path (device 0x1f0a2300) to PV 11 in VG 2 Fa
iled!
Feb 25 12:50:10 ukxxxx vmunix: LVM: Performed a switch for Lun ID = 0 (pv = 0x00
000000425ba000), from raw device 0x1f0a2000 (with priority: 0, and current flags
: 0x40) to raw device 0x1f0c2000 (with priority: 1, and current flags: 0x0).
Feb 25 12:50:10 ukxxxx vmunix: LVM: Path (device 0x1f0a2100) to PV 9 in VG 2 Fai
led!
Feb 25 12:50:10 ukxxxx vmunix: LVM: Path (device 0x1f0a2200) to PV 10 in VG 2 Fa
iled!
Feb 25 12:50:10 ukxxxx vmunix: LVM: Performed a switch for Lun ID = 0 (pv = 0x00
000000420d2000), from raw device 0x1f0a2200 (with priority: 0, and current flags
: 0x40) to raw device 0x1f0c2200 (with priority: 1, and current flags: 0x0).
Feb 25 12:50:10 ukxxxx vmunix: LVM: Path (device 0x1f0a2300) to PV 11 in VG 2 Fa
iled!
Feb 25 12:50:10 ukxxxx vmunix: LVM: Path (device 0x1f0a2400) to PV 12 in VG 2 Fa
iled!
Feb 25 12:50:10 ukxxxx vmunix: LVM: Performed a switch for Lun ID = 0 (pv = 0x00
000000420db000), from raw device 0x1f0a2400 (with priority: 0, and current flags
: 0x40) to raw device 0x1f0c2400 (with priority: 1, and current flags: 0x0).
Feb 25 12:50:10 ukxxxx vmunix: LVM: Path (device 0x1f0a2500) to PV 13 in VG 2 Fa
iled!
Feb 25 12:50:10 ukxxxx vmunix: LVM: Path (device 0x1f0a2600) to PV 14 in VG 2 Fa
iled!
Feb 25 12:50:10 ukxxxx vmunix: LVM: Performed a switch for Lun ID = 0 (pv = 0x00
000000425c2000), from raw device 0x1f0a2600 (with priority: 0, and current flags
: 0x40) to raw device 0x1f0c2600 (with priority: 1, and current flags: 0x0).
Feb 25 12:50:10 ukxxxx vmunix: LVM: Path (device 0x1f0a2700) to PV 15 in VG 2 Fa
iled!
Feb 25 12:50:10 ukxxxx vmunix: LVM: Path (device 0x1f0a3000) to PV 16 in VG 2 Fa
iled!
Feb 25 12:50:10 ukxxxx vmunix: LVM: Performed a switch for Lun ID = 0 (pv = 0x00
000000425ca000), from raw device 0x1f0a3000 (with priority: 0, and current flags
: 0x40) to raw device 0x1f0c3000 (with priority: 1, and current flags: 0x0).
Feb 25 12:50:10 ukxxxx vmunix: LVM: Path (device 0x1f0a3100) to PV 17 in VG 2 Fa
iled!
Feb 25 12:50:10 ukxxxx vmunix: LVM: Path (device 0x1f0a3200) to PV 18 in VG 2 Fa
iled!
Feb 25 12:50:10 ukxxxx vmunix: LVM: Performed a switch for Lun ID = 0 (pv = 0x00
000000425d1000), from raw device 0x1f0a3200 (with priority: 0, and current flags
: 0x40) to raw device 0x1f0c3200 (with priority: 1, and current flags: 0x0).
Feb 25 12:50:10 ukxxxx vmunix: LVM: Path (device 0x1f0a3300) to PV 19 in VG 2 Fa
iled!
Feb 25 12:50:10 ukxxxx vmunix: LVM: Path (device 0x1f0a3400) to PV 20 in VG 2 Fa
iled!
Feb 25 12:50:10 ukxxxx vmunix: LVM: Performed a switch for Lun ID = 0 (pv = 0x00
000000425da000), from raw device 0x1f0a3400 (with priority: 0, and current flags
: 0x40) to raw device 0x1f0c3400 (with priority: 1, and current flags: 0x0).
Feb 25 12:50:10 ukxxxx vmunix: LVM: Path (device 0x1f0a3500) to PV 21 in VG 2 Fa
iled!
Feb 25 12:50:10 ukxxxx vmunix: LVM: Path (device 0x1f0a3700) to PV 23 in VG 2 Fa
iled!
Feb 25 12:50:10 ukxxxx vmunix: LVM: Path (device 0x1f0a4100) to PV 25 in VG 2 Fa
iled!
Feb 25 12:50:10 ukxxxx vmunix: DIAGNOSTIC SYSTEM WARNING:
Feb 25 12:50:10 ukxxxx vmunix: The diagnostic logging facility has started re
ceiving excessive
Feb 25 12:50:10 ukxxxx vmunix: errors from the I/O subsystem. I/O error entr
ies will be lost
Feb 25 12:50:10 ukxxxx vmunix: until the cause of the excessive I/O logging i
s corrected.
Feb 25 12:50:10 ukxxxx vmunix: If the diaglogd daemon is not active, use the
Daemon Startup command
Feb 25 12:50:10 ukxxxx vmunix: in stm to start it.
Feb 25 12:50:10 ukxxxx vmunix: If the diaglogd daemon is active, use the logt
ool utility in stm
Feb 25 12:50:10 ukxxxx vmunix: to determine which I/O subsystem is logging ex
cessive errors.
Feb 25 12:50:10 ukxxxx vmunix: LVM: Path (device 0x1f0a4200) to PV 26 in VG 2 Fa
iled!
Feb 25 12:50:10 ukxxxx vmunix: LVM: Performed a switch for Lun ID = 0 (pv = 0x00
000000425f2000), from raw device 0x1f0a4200 (with priority: 0, and current flags
: 0x40) to raw device 0x1f0c4200 (with priority: 1, and current flags: 0x0).
Feb 25 12:50:10 ukxxxx vmunix: LVM: Path (device 0x1f0a4300) to PV 27 in VG 2 Fa
iled!
Feb 25 12:50:10 ukxxxx vmunix: LVM: Path (device 0x1f0a4400) to PV 28 in VG 2 Fa
iled!
Feb 25 12:50:10 ukxxxx vmunix: LVM: Performed a switch for Lun ID = 0 (pv = 0x00
000000425fb000), from raw device 0x1f0a4400 (with priority: 0, and current flags
: 0x40) to raw device 0x1f0c4400 (with priority: 1, and current flags: 0x0).
Feb 25 12:50:10 ukxxxx vmunix: LVM: Path (device 0x1f0a4500) to PV 29 in VG 2 Fa
iled!
Feb 25 12:50:10 ukxxxx vmunix: LVM: Path (device 0x1f0a4600) to PV 30 in VG 2 Fa
iled!
Feb 25 12:50:10 ukxxxx vmunix: LVM: Performed a switch for Lun ID = 0 (pv = 0x00
00000042681000), from raw device 0x1f0a4600 (with priority: 0, and current flags
: 0x40) to raw device 0x1f0c4600 (with priority: 1, and current flags: 0x0).
Feb 25 12:50:10 ukxxxx vmunix: LVM: Path (device 0x1f0a4700) to PV
Feb 25 12:50:11 ukxxxx vmunix: 31 in VG 2 Failed!
Feb 25 12:50:11 ukxxxx vmunix: LVM: Path (device 0x1f0a5000) to PV 32 in VG 2 Fa
iled!
Feb 25 12:50:11 ukxxxx vmunix: LVM: Performed a switch for Lun ID = 0 (pv = 0x00
0000004268a000), from raw device 0x1f0a5000 (with priority: 0, and current flags
: 0x40) to raw device 0x1f0c5000 (with priority: 1, and current flags: 0x0).
Feb 25 12:50:11 ukxxxx vmunix: LVM: Path (device 0x1f0a5100) to PV 33 in VG 2 Fa
iled!
Feb 25 12:50:11 ukxxxx vmunix: LVM: Path (device 0x1f0a5200) to PV 34 in VG 2 Fa
iled!
00000042691000), from raw device 0x1f0a5200 (with priority: 0, and current flags
: 0x40) to raw device 0x1f0c5200 (with priority: 1, and current flags: 0x0).
Feb 25 12:50:11 ukxxxx vmunix: LVM: Path (device 0x1f0a5300) to PV 35 in VG 2 Fa
iled!
Feb 25 12:50:11 ukxxxx vmunix: LVM: Path (device 0x1f0a5400) to PV 36 in VG 2 Fa
iled!
Feb 25 12:50:11 ukxxxx vmunix: LVM: Performed a switch for Lun ID = 0 (pv = 0x00
0000004269a000), from raw device 0x1f0a5400 (with priority: 0, and current flags
: 0x40) to raw device 0x1f0c5400 (with priority: 1, and current flags: 0x0).
Feb 25 12:50:11 ukxxxx vmunix: LVM: Path (device 0x1f0a5500) to PV 37 in VG 2 Fa
iled!
Feb 25 12:50:11 ukxxxx vmunix: LVM: Path (device 0x1f0a5600) to PV 38 in VG 2 Fa
iled!
Feb 25 12:50:11 ukxxxx vmunix: LVM: Performed a switch for Lun ID = 0 (pv = 0x00
000000426a2000), from raw device 0x1f0a5600 (with priority: 0, and current flags
: 0x40) to raw device 0x1f0c5600 (with priority: 1, and current flags: 0x0).
Feb 25 12:50:11 ukxxxx vmunix: LVM: Path (device 0x1f0a5700) to PV 39 in VG 2 Fa
iled!
Feb 25 12:50:11 ukxxxx vmunix: LVM: Path (device 0x1f0a6000) to PV 40 in VG 2 Fa
iled!
Feb 25 12:50:11 ukxxxx vmunix: LVM: Performed a switch for Lun ID = 0 (pv = 0x00
000000426ab000), from raw device 0x1f0a6000 (with priority: 0, and current flags
: 0x40) to raw device 0x1f0c6000 (with priority: 1, and current flags: 0x0).
Feb 25 12:50:11 ukxxxx vmunix: LVM: Path (device 0x1f0a6100) to PV 41 in VG 2 Fa
iled!
Feb 25 12:50:11 ukxxxx vmunix: LVM: Path (device 0x1f0a6200) to PV 42 in VG 2 Fa
iled!
Feb 25 12:50:11 ukxxxx vmunix: LVM: Performed a switch for Lun ID = 0 (pv = 0x00
000000426b3000), from raw device 0x1f0a6200 (with priority: 0, and current flags
: 0x40) to raw device 0x1f0c6200 (with priority: 1, and current flags: 0x0).
Feb 25 12:50:11 ukxxxx vmunix: LVM: Path (device 0x1f0a6300) to PV 43 in VG 2 Fa
iled!
Feb 25 12:50:11 ukxxxx vmunix: LVM: Path (device 0x1f0a6500) to PV 45 in VG 2 Fa
iled!
Feb 25 12:50:11 ukxxxx vmunix: LVM: Path (device 0x1f0a6700) to PV 47 in VG 2 Fa
iled!
Feb 25 12:50:11 ukxxxx vmunix: LVM: Path (device 0x1f0a7000) to PV 48 in VG 2 Fa
iled!
Feb 25 12:50:11 ukxxxx vmunix: LVM: Performed a switch for Lun ID = 0 (pv = 0x00
000000426cb000), from raw device 0x1f0a7000 (with priority: 0, and current flags
: 0x40) to raw device 0x1f0c7000 (with priority: 1, and current flags: 0x0).
Feb 25 12:50:11 ukxxxx vmunix: LVM: Path (device 0x1f0a7100) to PV 49 in VG 2 Fa
iled!
Feb 25 12:50:11 ukxxxx vmunix: LVM: Path (device 0x1f0a7200) to PV 50 in VG 2 Fa
iled!
Feb 25 12:50:11 ukxxxx vmunix: LVM: Performed a switch for Lun ID = 0 (pv = 0x00
000000426d3000), from raw device 0x1f0a7200 (with priority: 0, and current flags
: 0x40) to raw device 0x1f0c7200 (with priority: 1, and current flags: 0x0).
Feb 25 12:50:11 ukxxxx vmunix: LVM: Path (device 0x1f0a7300) to PV 51 in VG 2 Fa
iled!
Feb 25 12:50:11 ukxxxx vmunix: LVM: Path (device 0x1f0a7400) to PV 52 in VG 2 Fa
iled!
Feb 25 12:50:11 ukxxxx vmunix: LVM: Performed a switch for Lun ID = 0 (pv = 0x00
000000426db000), from raw device 0x1f0a7400 (with priority: 0, and current flags
: 0x40) to raw device 0x1f0c7400 (with priority: 1, and current flags: 0x0).
Feb 25 12:50:11 ukxxxx vmunix: LVM: Path (device 0x1f0a7500) to PV 53 in VG 2 Fa
iled!
Feb 25 12:50:11 ukxxxx vmunix: LVM: Path (device 0x1f0a7600) to PV 54 in VG 2 Fa
iled!
Feb 25 12:50:11 ukxxxx vmunix: LVM: Performed a switch for Lun ID = 0 (pv = 0x00
000000426e3000), from raw device 0x1f0a7600 (with priority: 0, and current flags
: 0x40) to raw device 0x1f0c7600 (with priority: 1, and current flags: 0x0).
Feb 25 12:50:11 ukxxxx vmunix: LVM: Path (device 0x1f0a7700) to PV 55 in VG 2 Fa
iled!
Feb 25 12:50:11 ukxxxx vmunix: LVM: Path (device 0x1f0a8000) to PV 56 in VG 2 Fa
iled!
Feb 25 12:50:11 ukxxxx vmunix: LVM: Path (device 0x1f0a8100) to PV 57 in VG 2 Fa
iled!
Feb 25 12:50:11 ukxxxx vmunix: LVM: Performed a switch for Lun ID = 0 (pv = 0x00
000000426ef000), from raw device 0x1f0a8100 (with priority: 0, and current flags
: 0x40) to raw device 0x1f0c8100 (with priority: 1, and current flags: 0x0).
Feb 25 12:50:11 ukxxxx vmunix: LVM: Path (device 0x1f0a8200) to PV 58 in VG 2 Fa
iled!
Feb 25 12:50:11 ukxxxx vmunix: LVM: Path (device 0x1f0a8300) to PV 59 in VG 2 Fa
iled!
Feb 25 12:50:11 ukxxxx vmunix: LVM: Performed a switch for Lun ID = 0 (pv = 0x00
000000426f7000), from raw device 0x1f0a8300 (with priority: 0, and current flags
: 0x40) to raw device 0x1f0c8300 (with priority: 1, and current flags: 0x0).
Feb 25 12:50:11 ukxxxx vmunix: LVM: Path (device 0x1f0a8400) to PV 60 in VG 2 Fa
iled!
Feb 25 12:50:11 ukxxxx vmunix: LVM: Path (device 0x1f0a8500) to PV 61 in VG 2 Fa
iled!
Feb 25 12:50:11 ukxxxx vmunix: LVM: Performed a switch for Lun ID = 0 (pv =
Feb 25 12:50:12 ukxxxx vmunix: 0x00000000426ff000), from raw device 0x1f0a8500
(with priority: 0, and current flags: 0x40) to raw device 0x1f0c8500 (with prior
ity: 1, and current flags: 0x0).
Feb 25 12:50:12 ukxxxx vmunix: LVM: Path (device 0x1f0a8600) to PV 62 in VG 2 Fa
iled!
Feb 25 12:50:12 ukxxxx vmunix: LVM: Path (device 0x1f0a8700) to PV 63 in VG 2 Fa
iled!
Feb 25 12:50:12 ukxxxx vmunix: LVM: Performed a switch for Lun ID = 0 (pv = 0x00
00000042707000), from raw device 0x1f0a8700 (with priority: 0, and current flags
: 0x40) to raw device 0x1f0c8700 (with priority: 1, and current flags: 0x0).
Feb 25 12:50:12 ukxxxx vmunix: LVM: Path (device 0x1f0a9000) to PV 64 in VG 2 Fa
iled!
Feb 25 12:50:12 ukxxxx vmunix: LVM: Path (device 0x1f0a9100) to PV 65 in VG 2 Fa
iled!
Feb 25 12:50:12 ukxxxx vmunix: LVM: Performed a switch for Lun ID = 0 (pv = 0x00
0000004270e000), from raw device 0x1f0a9100 (with priority: 0, and current flags
: 0x40) to raw device 0x1f0c9100 (with priority: 1, and current flags: 0x0).
Feb 25 12:50:12 ukxxxx vmunix: LVM: Path (device 0x1f0a9200) to PV 66 in VG 2 Fa
iled!
Feb 25 12:50:12 ukxxxx vmunix: LVM: Path (device 0x1f0a9300) to PV 67 in VG 2 Fa
iled!
Feb 25 12:50:12 ukxxxx vmunix: LVM: Performed a switch for Lun ID = 0 (pv = 0x00
00000042716000), from raw device 0x1f0a9300 (with priority: 0, and current flags
: 0x40) to raw device 0x1f0c9300 (with priority: 1, and current flags: 0x0).
Feb 25 12:50:12 ukxxxx vmunix: LVM: Path (device 0x1f0a9400) to PV 68 in VG 2 Fa
iled!
Feb 25 12:50:12 ukxxxx vmunix: LVM: Path (device 0x1f0a9500) to PV 69 in VG 2 Fa
iled!
Feb 25 12:50:12 ukxxxx vmunix: LVM: Performed a switch for Lun ID = 0 (pv = 0x00
0000004271f000), from raw device 0x1f0a9500 (with priority: 0, and current flags
: 0x40) to raw device 0x1f0c9500 (with priority: 1, and current flags: 0x0).
Feb 25 12:50:12 ukxxxx vmunix: LVM: Path (device 0x1f0a9600) to PV 70 in VG 2 Fa
iled!
Feb 25 12:50:12 ukxxxx vmunix: LVM: Path (device 0x1f0a9700) to PV 71 in VG 2 Fa
iled!
Feb 25 12:50:12 ukxxxx vmunix: LVM: Performed a switch for Lun ID = 0 (pv = 0x00
00000042727000), from raw device 0x1f0a9700 (with priority: 0, and current flags
: 0x40) to raw device 0x1f0c9700 (with priority: 1, and current flags: 0x0).
Feb 25 12:50:12 ukxxxx vmunix: LVM: vg[2]: pvnum=0 (dev_t=0x1f0a1000) is POWERFA
ILED
Feb 25 12:50:12 ukxxxx vmunix: LVM: vg[2]: pvnum=1 (dev_t=0x1f0c1100) is POWERFA
ILED
Feb 25 12:50:12 ukxxxx vmunix: LVM: vg[2]: pvnum=2 (dev_t=0x1f0a1200) is POWERFA
ILED
Feb 25 12:50:12 ukxxxx vmunix: LVM: vg[2]: pvnum=3 (dev_t=0x1f0c1300) is POWERFA
ILED
Feb 25 12:50:12 ukxxxx vmunix: LVM: vg[2]: pvnum=4 (dev_t=0x1f0a1400) is POWERFA
ILED
Feb 25 12:50:12 ukxxxx vmunix: LVM: vg[2]: pvnum=5 (dev_t=0x1f0c1500) is POWERFA
ILED
Feb 25 12:50:12 ukxxxx vmunix: LVM: vg[2]: pvnum=6 (dev_t=0x1f0c1600) is POWERFA
ILED
Feb 25 12:50:12 ukxxxx vmunix: LVM: vg[2]: pvnum=7 (dev_t=0x1f0c1700) is POWERFA
ILED
Feb 25 12:50:12 ukxxxx vmunix: LVM: vg[2]: pvnum=8 (dev_t=0x1f0c2000) is POWERFA
ILED
Feb 25 12:50:12 ukxxxx vmunix: LVM: vg[2]: pvnum=9 (dev_t=0x1f0c2100) is POWERFA
ILED
Feb 25 12:50:12 ukxxxx vmunix: LVM: vg[2]: pvnum=10 (dev_t=0x1f0c2200) is POWERF
AILED
Feb 25 12:50:12 ukxxxx vmunix: LVM: vg[2]: pvnum=11 (dev_t=0x1f0c2300) is POWERF
AILED
Feb 25 12:50:12 ukxxxx vmunix: LVM: vg[2]: pvnum=12 (dev_t=0x1f0c2400) is POWERF
AILED
Feb 25 12:50:12 ukxxxx vmunix: LVM: vg[2]: pvnum=13 (dev_t=0x1f0c2500) is POWERF
AILED
Feb 25 12:50:12 ukxxxx vmunix: LVM: vg[2]: pvnum=14 (dev_t=0x1f0c2600) is POWERF
AILED
Feb 25 12:50:12 ukxxxx vmunix: LVM: vg[2]: pvnum=15 (dev_t=0x1f0c2700) is POWERF
AILED
Feb 25 12:50:12 ukxxxx vmunix: LVM: vg[2]: pvnum=16 (dev_t=0x1f0c3000) is POWERF
AILED
Feb 25 12:50:12 ukxxxx vmunix: LVM: vg[2]: pvnum=17 (dev_t=0x1f0c3100) is POWERF
AILED
Feb 25 12:50:12 ukxxxx vmunix: LVM: vg[2]: pvnum=18 (dev_t=0x1f0c3200) is POWERF
AILED
Feb 25 12:50:12 ukxxxx vmunix: LVM: vg[2]: pvnum=19 (dev_t=0x1f0c3300) is POWERF
AILED
Feb 25 12:50:12 ukxxxx vmunix: LVM: vg[2]: pvnum=20 (dev_t=0x1f0c3400) is POWERF
AILED
Feb 25 12:50:12 ukxxxx vmunix: LVM: vg[2]: pvnum=21 (dev_t=0x1f0c3500) is POWERF
AILED
Feb 25 12:50:12 ukxxxx vmunix: LVM: vg[2]: pvnum=22 (dev_t=0x1f0c3600) is POWERF
AILED
Feb 25 12:50:12 ukxxxx vmunix: LVM: vg[2]: pvnum=23 (dev_t=0x1f0c3700) is POWERF
AILED
Feb 25 12:50:12 ukxxxx vmunix: LVM: vg[2]: pvnum=24 (dev_t=0x1f0c4000) is POWERF
AILED
Feb 25 12:50:12 ukxxxx vmunix: LVM: vg[2]: pvnum=25 (dev_t=0x1f0c4100) is POWERF
AILED
Feb 25 12:50:12 ukxxxx vmunix: LVM: vg[2]: pvnum=26 (dev_t=0x1f0c4200) is POWERF
AILED
Feb 25 12:50:12 ukxxxx vmunix: LVM: vg[2]: pvnum=27 (dev_t=0x1f0c4300) is POWERF
AILED
Feb 25 12:50:12 ukxxxx vmunix: LVM: vg[2]: pvnum=28 (dev_t=0x1f0c4400) is POWERF
AILED
Feb 25 12:50:12 ukxxxx vmunix: LVM: vg[2]: pvnum=29 (dev_t=0x1f0c4500) is POWERF
AILED
Feb 25 12:50:12 ukxxxx vmunix: LVM: vg[2]: pvnum=30 (dev_t=0x1f0c4600) is POWERF
AILED
Feb 25 12:50:12 ukxxxx vmunix: LVM: vg[2]: pvnum=31 (dev_t=0x1f0c4700) is POWERF
AILED
Feb 25 12:50:12 ukxxxx vmunix: LVM: vg[2]: pvnum=32 (dev_t=0x1f0c5000) is POWERF
AILED
Feb 25 12:50:12 ukxxxx vmunix: LVM: vg[2]: pvnum=33 (dev_t=0x1f0c5100) is POWERF
AILED
Feb 25 12:50:12 ukxxxx vmunix: LVM: vg[2]: pvnum=34 (dev_t=0x1f0c5200) is POWERF
AILED
Feb 25 12:50:12 ukxxxx vmunix: LVM: vg[2]: pvnum=35 (dev_t=0x1f0c5300) is POWERF
AILED
Feb 25 12:50:12 ukxxxx vmunix: LVM: vg[2]: pvnum=36 (dev_t=0x1f0c5400) is POWERF
AILED
Feb 25 12:50:12 ukxxxx vmunix: LVM: vg[2]: pvnum=37 (dev_t=0x1f0c5500) is POWERF
AILED
Feb 25 12:50:12 ukxxxx vmunix: LVM: vg[2]: pvnum=38 (dev_t=0x1f0c5600) is POWERF
AILED
Feb 25 12:50:12 ukxxxx vmunix: LVM: vg[2]: pvnum=39 (dev_t=0x1f0c5700) is POWERF
AILED
Feb 25 12:50:12 ukxxxx vmunix: LVM: vg[2]: pvnum=40 (dev_t=0x1f0c6000) is POWERF
AILED
Feb 25 12:50:12 ukxxxx vmunix: LVM: vg[2]: pvnum=41 (dev_t=0x1f0c6100) is POWERF
AILED
Feb 25 12:50:12 ukxxxx vmunix: LVM: vg[2]: pvnum=42 (dev_t=0x1f0c
Feb 25 12:50:12 ukxxxx vmunix: 6200) is POWERFAILED
Feb 25 12:50:12 ukxxxx vmunix: LVM: vg[2]: pvnum=43 (dev_t=0x1f0c6300) is POWERF
AILED
Feb 25 12:50:12 ukxxxx vmunix: LVM: vg[2]: pvnum=44 (dev_t=0x1f0c6400) is POWERF
AILED
Feb 25 12:50:12 ukxxxx vmunix: LVM: vg[2]: pvnum=45 (dev_t=0x1f0c6500) is POWERF
AILED
Feb 25 12:50:12 ukxxxx vmunix: LVM: vg[2]: pvnum=46 (dev_t=0x1f0c6600) is POWERF
AILED
Feb 25 12:50:12 ukxxxx vmunix: LVM: vg[2]: pvnum=47 (dev_t=0x1f0c6700) is POWERF
AILED
Feb 25 12:50:12 ukxxxx vmunix: LVM: vg[2]: pvnum=48 (dev_t=0x1f0c7000) is POWERF
AILED
Feb 25 12:50:12 ukxxxx vmunix: LVM: vg[2]: pvnum=49 (dev_t=0x1f0c7100) is POWERF
AILED
Feb 25 12:50:12 ukxxxx vmunix: LVM: vg[2]: pvnum=50 (dev_t=0x1f0c7200) is POWERF
AILED
Feb 25 12:50:12 ukxxxx vmunix: LVM: vg[2]: pvnum=51 (dev_t=0x1f0c7300) is POWERF
AILED
Feb 25 12:50:12 ukxxxx vmunix: LVM: vg[2]: pvnum=52 (dev_t=0x1f0c7400) is POWERF
AILED
Feb 25 12:50:12 ukxxxx vmunix: LVM: vg[2]: pvnum=53 (dev_t=0x1f0c7500) is POWERF
AILED
Feb 25 12:50:12 ukxxxx vmunix: LVM: vg[2]: pvnum=54 (dev_t=0x1f0c7600) is POWERF
AILED
Feb 25 12:50:12 ukxxxx vmunix: LVM: vg[2]: pvnum=55 (dev_t=0x1f0c7700) is POWERF
AILED
Feb 25 12:50:12 ukxxxx vmunix: LVM: vg[2]: pvnum=56 (dev_t=0x1f0c8000) is POWERF
AILED
Feb 25 12:50:12 ukxxxx vmunix: LVM: vg[2]: pvnum=57 (dev_t=0x1f0c8100) is POWERF
AILED
Feb 25 12:50:12 ukxxxx vmunix: LVM: vg[2]: pvnum=58 (dev_t=0x1f0c8200) is POWERF
AILED
Feb 25 12:50:12 ukxxxx vmunix: LVM: vg[2]: pvnum=59 (dev_t=0x1f0c8300) is POWERF
AILED
Feb 25 12:50:12 ukxxxx vmunix: LVM: vg[2]: pvnum=60 (dev_t=0x1f0c8400) is POWERF
AILED
Feb 25 12:50:12 ukxxxx vmunix: LVM: vg[2]: pvnum=61 (dev_t=0x1f0c8500) is POWERF
AILED
Feb 25 12:50:12 ukxxxx vmunix: LVM: vg[2]: pvnum=62 (dev_t=0x1f0c8600) is POWERF
AILED
Feb 25 12:50:12 ukxxxx vmunix: LVM: vg[2]: pvnum=63 (dev_t=0x1f0c8700) is POWERF
AILED
Feb 25 12:50:12 ukxxxx vmunix: LVM: vg[2]: pvnum=64 (dev_t=0x1f0c9000) is POWERF
AILED
Feb 25 12:50:12 ukxxxx vmunix: LVM: vg[2]: pvnum=65 (dev_t=0x1f0c9100) is POWERF
AILED
Feb 25 12:50:12 ukxxxx vmunix: LVM: vg[2]: pvnum=66 (dev_t=0x1f0c9200) is POWERF
AILED
Feb 25 12:50:12 ukxxxx vmunix: LVM: vg[2]: pvnum=67 (dev_t=0x1f0c9300) is POWERF
AILED
Feb 25 12:50:12 ukxxxx vmunix: LVM: vg[2]: pvnum=68 (dev_t=0x1f0c9400) is POWERF
AILED
Feb 25 12:50:12 ukxxxx vmunix: LVM: vg[2]: pvnum=69 (dev_t=0x1f0c9500) is POWERF
AILED
Feb 25 12:50:12 ukxxxx vmunix: LVM: vg[2]: pvnum=70 (dev_t=0x1f0c9600) is POWERF
AILED
Feb 25 12:50:12 ukxxxx vmunix: LVM: vg[2]: pvnum=71 (dev_t=0x1f0c9700) is POWERF
AILED
Feb 25 12:50:12 ukxxxx vmunix: DIAGNOSTIC SYSTEM WARNING:
Feb 25 12:50:12 ukxxxx vmunix: The diagnostic logging facility is no longer r
eceiving excessive
Feb 25 12:50:12 ukxxxx vmunix: errors from the I/O subsystem. 44 I/O error e
ntries were lost.
Feb 25 12:50:12 ukxxxx vmunix: LVM: Performed a switch for Lun ID = 0 (pv = 0x00
000000425b7000), from raw device 0x1f0a1200 (with priority: 0, and current flags
: 0xc0) to raw device 0x1f0c1200 (with priority: 1, and current flags: 0x0).
Feb 25 12:50:12 ukxxxx vmunix: LVM: Performed a switch for Lun ID = 0 (pv = 0x00
00000042679000), from raw device 0x1f0a1400 (with priority: 0, and current flags
: 0xc0) to raw device 0x1f0c1400 (with priority: 1, and current flags: 0x0).
Feb 25 12:50:12 ukxxxx vmunix: LVM: Performed a switch for Lun ID = 0 (pv = 0x00
00000042647800), from raw device 0x1f0a1000 (with priority: 0, and current flags
: 0xc0) to raw device 0x1f0c1000 (with priority: 1, and current flags: 0x0).
Feb 25 12:50:12 ukxxxx vmunix: LVM: Restored PV 46 to VG 2.
Feb 25 12:50:13 ukxxxx vmunix: LVM: Restored PV 6 to VG 2.
Feb 25 12:50:13 ukxxxx vmunix: LVM: Restored PV 22 to VG 2.
Feb 25 12:50:13 ukxxxx vmunix: LVM: Restored PV 24 to VG 2.
Feb 25 12:50:15 ukxxxx vmunix: DIAGNOSTIC SYSTEM WARNING:
Feb 25 12:50:15 ukxxxx vmunix: The diagnostic logging facility has started re
ceiving excessive
Feb 25 12:50:15 ukxxxx vmunix: errors from the I/O subsystem. I/O error entr
ies will be lost
Feb 25 12:50:15 ukxxxx vmunix: until the cause of the excessive I/O logging i
s corrected.
Feb 25 12:50:15 ukxxxx vmunix: If the diaglogd daemon is not active, use the
Daemon Startup command
Feb 25 12:50:15 ukxxxx vmunix: in stm to start it.
Feb 25 12:50:15 ukxxxx vmunix: If the diaglogd daemon is active, use the logt
ool utility in stm
Feb 25 12:50:15 ukxxxx vmunix: to determine which I/O subsystem is logging ex
cessive errors.
Feb 25 12:50:15 ukxxxx vmunix: LVM: Recovered Path (device 0x1f0c1000) to PV 0 i
n VG 2.
Feb 25 12:50:15 ukxxxx vmunix: LVM: Recovered Path (device 0x1f0c1200) to PV 2 i
n VG 2.
Feb 25 12:50:15 ukxxxx vmunix: LVM: Recovered Path (device 0x1f0c1400) to PV 4 i
n VG 2.
Feb 25 12:50:15 ukxxxx vmunix: LVM: Restored PV 0 to VG 2.
Feb 25 12:50:16 ukxxxx vmunix: LVM: Restored PV 2 to VG 2.
Feb 25 12:50:16 ukxxxx vmunix: LVM: Restored PV 4 to VG 2.
Feb 25 12:50:20 ukxxxx vmunix: DIAGNOSTIC SYSTEM WARNING:
Feb 25 12:50:20 ukxxxx vmunix: The diagnostic logging facility is no longer r
eceiving excessive
Feb 25 12:50:20 ukxxxx vmunix: errors from the I/O subsystem. 49 I/O error e
ntries were lost.
Feb 25 12:50:20 ukxxxx vmunix: The diagnostic logging facility has started re
ceiving excessive
Feb 25 12:50:20 ukxxxx vmunix: errors from the I/O subsystem. I/O error entr
ies will be lost
Feb 25 12:50:20 ukxxxx vmunix: until the cause of the excessive I/O logging i
s corrected.
Feb 25 12:50:20 ukxxxx vmunix: If the diaglogd daemon is not active, use the
Daemon Startup command
Feb 25 12:50:20 ukxxxx vmunix: DIAGNOSTIC SYSTEM WARNING:
Feb 25 12:50:20 ukxxxx vmunix: in stm to start it.
Feb 25 12:50:20 ukxxxx vmunix: If the diaglogd daemon is active, use the logt
ool utility in stm
Feb 25 12:50:20 ukxxxx vmunix: to determine which I/O subsystem is logging ex
cessive errors.
Feb 25 12:50:25 ukxxxx vmunix: DIAGNOSTIC SYSTEM WARNING:
Feb 25 12:50:25 ukxxxx vmunix: The diagnostic logging facility is no longer r
eceiving excessive
Feb 25 12:50:25 ukxxxx vmunix: errors from the I/O subsystem. 49 I/O error e
ntries were lost.
Feb 25 12:50:25 ukxxxx vmunix: The diagnostic logging facility has started re
ceiving excessive
Feb 25 12:50:25 ukxxxx vmunix: errors from the I/O subsystem. I/O error entr
ies will be lost

Feb 25 12:54:16 ukxxxx vmunix: LVM: Recovered Path (device 0x1f0a1000) to PV 0 i
n VG 2.
Feb 25 12:54:16 ukxxxx vmunix: LVM: Performed a switch for Lun ID = 0 (pv = 0x00
00000042647800), from raw device 0x1f0c1000 (with priority: 1, and current flags
: 0x0) to raw device 0x1f0a1000 (with priority: 0, and current flags: 0x80).
Feb 25 12:54:16 ukxxxx vmunix: LVM: Recovered Path (device 0x1f0a1100) to PV 1 i
n VG 2.
Feb 25 12:54:16 ukxxxx vmunix: LVM: Recovered Path (device 0x1f0a1200) to PV 2 i
n VG 2.
Feb 25 12:54:16 ukxxxx vmunix: LVM: Performed a switch for Lun ID = 0 (pv = 0x00
000000425b7000), from raw device 0x1f0c1200 (with priority: 1, and current flags
: 0x0) to raw device 0x1f0a1200 (with priority: 0, and current flags: 0x80).
Feb 25 12:54:16 ukxxxx vmunix: LVM: Recovered Path (device 0x1f0a1300) to PV 3 i
n VG 2.
Feb 25 12:54:16 ukxxxx vmunix: LVM: Recovered Path (device 0x1f0a1400) to PV 4 i
n VG 2.
Feb 25 12:54:16 ukxxxx vmunix: LVM: Performed a switch for Lun ID = 0 (pv = 0x00
00000042679000), from raw device 0x1f0c1400 (with priority: 1, and current flags
: 0x0) to raw device 0x1f0a1400 (with priority: 0, and current flags: 0x80).
Feb 25 12:54:16 ukxxxx vmunix: LVM: Recovered Path (device 0x1f0a1500) to PV 5 i
n VG 2.
Feb 25 12:54:16 ukxxxx vmunix: LVM: Recovered Path (device 0x1f0a1600) to PV 6 i
n VG 2.
Feb 25 12:54:16 ukxxxx vmunix: LVM: Performed a switch for Lun ID = 0 (pv = 0x00
0000004258a000), from raw device 0x1f0c1600 (with priority: 1, and current flags
: 0x0) to raw device 0x1f0a1600 (with priority: 0, and current flags: 0x80).
Feb 25 12:54:16 ukxxxx vmunix: LVM: Recovered Path (device 0x1f0a1700) to PV 7 i
n VG 2.
Feb 25 12:54:16 ukxxxx vmunix: LVM: Recovered Path (device 0x1f0a2000) to PV 8 i
n VG 2.
Feb 25 12:54:16 ukxxxx vmunix: LVM: Performed a switch for Lun ID = 0 (pv = 0x00
000000425ba000), from raw device 0x1f0c2000 (with priority: 1, and current flags
: 0x0) to raw device 0x1f0a2000 (with priority: 0, and current flags: 0x80).
Feb 25 12:54:16 ukxxxx vmunix: LVM: Recovered Path (device 0x1f0a2100) to PV 9 i
n VG 2.
Feb 25 12:54:16 ukxxxx vmunix: LVM: Recovered Path (device 0x1f0a2200) to PV 10
in VG 2.
Feb 25 12:54:16 ukxxxx vmunix: LVM: Performed a switch for Lun ID = 0 (pv = 0x00
000000420d2000), from raw device 0x1f0c2200 (with priority: 1, and current flags
: 0x0) to raw device 0x1f0a2200 (with priority: 0, and current flags: 0x80).
Feb 25 12:54:16 ukxxxx vmunix: LVM: Recovered Path (device 0x1f0a2300) to PV 11
in VG 2.
Feb 25 12:54:16 ukxxxx vmunix: LVM: Recovered Path (device 0x1f0a2400) to PV 12
in VG 2.
Feb 25 12:54:16 ukxxxx vmunix: LVM: Performed a switch for Lun ID = 0 (pv = 0x00
000000420db000), from raw device 0x1f0c2400 (with priority: 1, and current flags
: 0x0) to raw device 0x1f0a2400 (with priority: 0, and current flags: 0x80).

This seems to have happened once before, but nothing more until this occurence, and it appears to happen according to the Application people whilst they are doing a heavy workload.

Therefore, is it most likely to be a timeout issue ?.

[Devender Khatana]

Hi,

Heavy usage should not cause this. This is related to some problem in I/O devices. It could be due to bad cable/terminator as well. What is the devices you are using for LVM?

I would suspect some hardware issue and would recommend to call vendor to diagnose this.

HTH,
Devender

[pfwhufc]

Thanks Devender.

[Andrew Rutter]

hi,

it could be a timeout issue as its a large system with many disks, especially if as you say it was exceptionally heavy, but I would expect lbolt errors aswell if this was the case.

I think its more likely to be an interface or cable problem though.

I would check the hardware out and run tests on the interfaces aswell, and also check for relevent patches, lvm and system.

Would aslo be worth running this by HP aswell, see what they ahve to say.

Andy

[Andrew Merritt_2]

You should also check the /var/opt/resmon/log/event.log file to see if any EMS events were logged at this time, and also in logtool in STM to see what the I/O errors being logged were. This should give some more clues as to which hardware is affected and how.

Andrew

[pfwhufc]

Thanks to all that have replied.
The event log in /var/opt/resmon/log has nothing relating to this.

I have also noticed that another server that accesses the same Symmetrix through a Cisco switch had the same error. 3 servers that are directly connected to the Symmetrix did not have the issue.

So, I am checking for switch issues, and/or timeouts. The timeouts on the PV's are all set to default which I understand is 30 seconds.