pvlink error 질문입니다.

[김학곤]

cwbksvr1:/> ioscan -fnC disk

Class I H/W Path Driver S/W State H/W Type Description

==========================================================================

disk 1 0/0/1/1.2.0 sdisk CLAIMED DEVICE HP 73.4GST373453LC

/dev/dsk/c1t2d0 /dev/rdsk/c1t2d0

disk 3 0/0/2/0.2.0 sdisk CLAIMED DEVICE HP 73.4GST373453LC

/dev/dsk/c2t2d0 /dev/rdsk/c2t2d0

disk 4 0/0/2/1.2.0 sdisk CLAIMED DEVICE HP DVD-ROM 305

/dev/dsk/c3t2d0 /dev/rdsk/c3t2d0

disk 27 0/2/0/0.97.26.19.0.0.0 sdisk CLAIMED DEVICE DGC CX500WDR5

/dev/dsk/c21t0d0 /dev/rdsk/c21t0d0

disk 28 0/2/0/0.97.26.19.0.0.1 sdisk CLAIMED DEVICE DGC CX500WDR5

/dev/dsk/c21t0d1 /dev/rdsk/c21t0d1

disk 29 0/2/0/0.97.26.19.0.0.2 sdisk CLAIMED DEVICE DGC CX500WDR5

/dev/dsk/c21t0d2 /dev/rdsk/c21t0d2

disk 30 0/2/0/0.97.26.19.0.0.3 sdisk CLAIMED DEVICE DGC CX500WDR5

/dev/dsk/c21t0d3 /dev/rdsk/c21t0d3

disk 31 0/2/0/0.97.26.19.0.0.4 sdisk CLAIMED DEVICE DGC CX500WDR5

/dev/dsk/c21t0d4 /dev/rdsk/c21t0d4

disk 32 0/2/0/0.97.26.19.0.0.5 sdisk CLAIMED DEVICE DGC CX500WDR5

/dev/dsk/c21t0d5 /dev/rdsk/c21t0d5

disk 33 0/2/0/0.97.26.19.0.0.6 sdisk CLAIMED DEVICE DGC CX500WDR5

/dev/dsk/c21t0d6 /dev/rdsk/c21t0d6

disk 34 0/2/0/0.97.26.19.0.0.7 sdisk CLAIMED DEVICE DGC CX500WDR5

/dev/dsk/c21t0d7 /dev/rdsk/c21t0d7

disk 5 0/4/2/0.97.4.19.0.0.0 sdisk NO_HW DEVICE DGC CX400WDR5

/dev/dsk/c10t0d0 /dev/rdsk/c10t0d0

disk 6 0/4/2/0.97.4.19.0.0.1 sdisk NO_HW DEVICE DGC CX400WDR5

/dev/dsk/c10t0d1 /dev/rdsk/c10t0d1

disk 9 0/4/2/0.97.4.19.0.0.2 sdisk NO_HW DEVICE DGC CX400WDR5

/dev/dsk/c10t0d2 /dev/rdsk/c10t0d2

disk 10 0/4/2/0.97.4.19.0.0.3 sdisk NO_HW DEVICE DGC CX400WDR5

/dev/dsk/c10t0d3 /dev/rdsk/c10t0d3

disk 11 0/4/2/0.97.4.19.0.0.4 sdisk NO_HW DEVICE DGC CX400WDR5

/dev/dsk/c10t0d4 /dev/rdsk/c10t0d4

disk 12 0/4/2/0.97.4.19.0.0.5 sdisk NO_HW DEVICE DGC CX400WDR5

/dev/dsk/c10t0d5 /dev/rdsk/c10t0d5

disk 13 0/4/2/0.97.4.19.0.0.6 sdisk NO_HW DEVICE DGC CX400WDR5

/dev/dsk/c10t0d6 /dev/rdsk/c10t0d6

disk 14 0/4/2/0.97.4.19.0.0.7 sdisk NO_HW DEVICE DGC CX400WDR5

/dev/dsk/c10t0d7 /dev/rdsk/c10t0d7

disk 7 0/4/2/0.97.5.19.0.0.0 sdisk NO_HW DEVICE DGC CX400WDR5

/dev/dsk/c11t0d0 /dev/rdsk/c11t0d0

disk 8 0/4/2/0.97.5.19.0.0.1 sdisk NO_HW DEVICE DGC CX400WDR5

/dev/dsk/c11t0d1 /dev/rdsk/c11t0d1

disk 15 0/4/2/0.97.5.19.0.0.2 sdisk NO_HW DEVICE DGC CX400WDR5

/dev/dsk/c11t0d2 /dev/rdsk/c11t0d2

disk 16 0/4/2/0.97.5.19.0.0.3 sdisk NO_HW DEVICE DGC CX400WDR5

/dev/dsk/c11t0d3 /dev/rdsk/c11t0d3

disk 17 0/4/2/0.97.5.19.0.0.4 sdisk NO_HW DEVICE DGC CX400WDR5

/dev/dsk/c11t0d4 /dev/rdsk/c11t0d4

disk 18 0/4/2/0.97.5.19.0.0.5 sdisk NO_HW DEVICE DGC CX400WDR5

/dev/dsk/c11t0d5 /dev/rdsk/c11t0d5

disk 19 0/4/2/0.97.5.19.0.0.6 sdisk NO_HW DEVICE DGC CX400WDR5

/dev/dsk/c11t0d6 /dev/rdsk/c11t0d6

disk 20 0/4/2/0.97.5.19.0.0.7 sdisk NO_HW DEVICE DGC CX400WDR5

/dev/dsk/c11t0d7 /dev/rdsk/c11t0d7

--------------------------------------------------------------------------



보시는 바와 같이, CX500 쪽 DISK는 멀쩡한데 CX400 쪽만 NO_HW 로 표시됩니다.

당시의 syslog에 나타난 에러 메시지는 아래와 같습니다



------------------------------------------------------------------------

Jan 24 15:08:53 cwbksvr1 vmunix: LVM: VG 64 0x020000: Lost quorum.

Jan 24 15:08:53 cwbksvr1 vmunix: This may block configuration changes and I/Os. In order to reestablish quorum at least 1 of the following PVs (represented by current link) must become available:

Jan 24 15:08:53 cwbksvr1 vmunix: <31 0x0a0100>

Jan 24 15:08:53 cwbksvr1 vmunix: LVM: VG 64 0x020000: PVLink 31 0x0a0100 Failed! The PV is not accessible.

Jan 24 15:08:53 cwbksvr1 vmunix: LVM: VG 64 0x020000: PVLink 31 0x0b0100 Failed! The PV is not accessible.

Jan 24 15:10:05 cwbksvr1 vmunix: LVM: VG 64 0x030000: Lost quorum.

Jan 24 15:10:05 cwbksvr1 vmunix: This may block configuration changes and I/Os. In order to reestablish quorum at least 1 of the following PVs (represented by current link) must become available:

Jan 24 15:10:05 cwbksvr1 vmunix: <31 0x0a0000>

Jan 24 15:10:05 cwbksvr1 vmunix: LVM: VG 64 0x030000: PVLink 31 0x0a0000 Failed! The PV is not accessible.

--------------------------------------------------------------------------



위와 같이 몇몇 Physical Disk가 끊어진 것 처럼 나옵니다.

리부팅 하면 정상적으로 다 붙고요.



그 외에 오늘 데이터 I/O를 테스트했는데,

특이한 점은 I/O를 수행하는 시작 시점에 꽤 많은 Delay가 발생하는데,

추측하기로는 disk cache 쪽 문제가 아닌가 싶기도 하네요.



또 한가지 특이한 점은 같은 CX400 디스크를 사용하고 있는 Active Directory 서버나, RP3004 SFC 서버쪽에서는 별다른 이상 증상이 보이지 않고 있다는 겁니다.

현재로서는 remote에 있는 서버이다 보니 정확한 원인을 찾아내기가 아주 힘이 드네요.

마지막으로 아래는 lvmtab 내용입니다.

-------------------------------------------

cwbksvr1:/tsmbackup> strings /etc/lvmtab

/dev/vg00

/dev/dsk/c1t2d0

/dev/dsk/c2t2d0

/dev/vg03

/dev/dsk/c10t0d0

/dev/dsk/c11t0d0

/dev/vg02

/dev/dsk/c10t0d1

/dev/dsk/c11t0d1

--------------------------------------------

위와 같은상황인데요

hba card 교체를 하였는데도 같은증상입니다.

insf -e 등으로 해도 같은증상이구요

emc 측에서는 firmware 나 drive 등을 얘기하는데요

처음구성단계도 아니고 잘사용하던단계에서

나오는 얘기들이라 어떤식으로 접근을 해야하는지

또어떤것을 더체크해 볼수 있는지 궁금합니다.

답변부탁드립니다.

[김병수]

SAN으로 연결이 되어 있는지 , 아니면 FC direct인지..



그러나 rebooting하면 정상이라는것이 이상합니다.



여러가지 부분을 check하여야 될것으로 보입니다.



FC card에 관련된 patch라던가....



emc disk의 powerpath가 설치되어 있다면..



# powermt display 하여 보시고 optimal로 되어 있는지..



그럼~~~

[최이석]

훔...제가 보기엔 PVlink를 잘못거신거 같은데요?

/dev/vg03

/dev/dsk/c10t0d0

/dev/dsk/c11t0d0

/dev/vg02

/dev/dsk/c10t0d1

/dev/dsk/c11t0d1 의 디바이스를 보면

0/4/2/0.97.4.19.0.0.4로 HBA Card의 물리적 path가 동일합니다.

이렇게 되면 PVlink가 아니라 VG extend될거 같은데..

1. CX500 Navisphere보셔서 RaidGroup에 LUN잘 할당되었는지 보시고

2. SAN환경이면 조닝잘 되었는지 보시고..

3. 보기에는 0/2/0/0.97.26.19.0.0.0쪽과 0/4/2/0.97.4.19.0.0.4쪽이

PVlink가 되야할거 같은데 LUN개수가 좀 틀려서리..

한번 확인해 보셔야 할듯...

4. fw나 드라이버는 문제 찾기 어려울때 뭐 항상 나오는말이고, 그렇게

버전이 낮지 않음 위에같은 문제는 안생기더라구요..ㅋ