Operating System - HP-UX
1748170 Members
4063 Online
108758 Solutions
New Discussion

Re: (/dev/fclp1) : detected that device id 0xffffff, PWWN 0x8004070000000040 is offline.

 
SanjeevRao
Frequent Visitor

(/dev/fclp1) : detected that device id 0xffffff, PWWN 0x8004070000000040 is offline.

Dear team

 

Recent days, one of the Rx2800 Server started performing really slow, CPU utilisation (sar -u) shows

16:09:31    %usr    %sys    %wio   %idle

16:09:33       4       0       0      96
16:09:35      44       0       0      56


(though this is slave app machine in active-active) network, other servers are really performing well,

 

pinging from one machine to this machine shows me 10% packet loss,  completely went through the network switch/firewall logs in terms of any network latency, but no error msg found so far,

 

ping result from neighbouring machine in the same VLAN

 

ping 1921.68.0.4
PING 1921.68.0.4: 64 byte packets
64 bytes from 1921.68.0.4: icmp_seq=0. time=0. ms
64 bytes from 1921.68.0.4: icmp_seq=1. time=0. ms
64 bytes from 1921.68.0.4: icmp_seq=2. time=1028. ms
64 bytes from 1921.68.0.4: icmp_seq=3. time=18. ms
64 bytes from 1921.68.0.4: icmp_seq=4. time=0. ms
64 bytes from 1921.68.0.4: icmp_seq=5. time=0. ms
64 bytes from 1921.68.0.4: icmp_seq=6. time=1226. ms
64 bytes from 1921.68.0.4: icmp_seq=7. time=216. ms
64 bytes from 1921.68.0.4: icmp_seq=8. time=0. ms
64 bytes from 1921.68.0.4: icmp_seq=23. time=1600. ms
64 bytes from 1921.68.0.4: icmp_seq=24. time=590. ms
64 bytes from 1921.68.0.4: icmp_seq=25. time=0. ms
64 bytes from 1921.68.0.4: icmp_seq=26. time=0. ms
64 bytes from 1921.68.0.4: icmp_seq=27. time=1534. ms
64 bytes from 1921.68.0.4: icmp_seq=28. time=524. ms
64 bytes from 1921.68.0.4: icmp_seq=29. time=0. ms
64 bytes from 1921.68.0.4: icmp_seq=30. time=0. ms
64 bytes from 1921.68.0.4: icmp_seq=31. time=1674. ms
64 bytes from 1921.68.0.4: icmp_seq=32. time=664. ms
64 bytes from 1921.68.0.4: icmp_seq=33. time=0. ms
----1921.68.0.4 PING Statistics----
packets transmitted,  packets received, 0% packet loss
round-trip (ms)  min/avg/max = 0/276/122

 

Now one of the HP friend says login to MP and check any I/O related issue in this machine? unfortunately, i dont have the MP ip with me

 

any clues/guidance will be greatly appreciated.

 

 

P.S. this thread has been moved from Servers > Integrity Servers to HP-UX > sysadmin - HP Forums Moderator

San
8 REPLIES 8
Hiren N Dave
Valued Contributor

Re: (/dev/fclp1) : detected that device id 0xffffff, PWWN 0x8004070000000040 is offline.

Hi,

 

The %wio is the percentage of time the CPU spent on blocked I/O request.

 

Here, the %wio is very high which indicates a disk bottleneck.

 

You may want to use sar -d and iostat commands to find out if any disk problem is causing this issue.

 

Cheers.

I am an HP Employee

Was this post useful? - You may click the KUDOS! star to say thank you.
SanjeevRao
Frequent Visitor

Re: (/dev/fclp1) : detected that device id 0xffffff, PWWN 0x8004070000000040 is offline.

thanks Mr. Hiren

 

yes you are right, i was suspecting I/O as well, but didnt have details about MP connectivity,

 

below are sar -d result, when i tried sar -d 2 10, it doesnt show any result

 

19:06:23   device   %busy   avque   r+w/s  blks/s  avwait  avserv
19:06:27
        
19:06:29
19:06:31
19:06:33
19:06:36
19:06:37
19:06:40
19:06:41
19:06:43

so i tried  #sar -d 2 2

 

19:06:56   device   %busy   avque   r+w/s  blks/s  avwait  avserv
19:06:58
19:07:00   disk13    0.46    0.50      10     304    0.00    0.91

Average    disk13    0.24    0.50       5     159    0.00    0.91

 

is there any ways i can obtain the mp lan ip details from the servers itself? or the SAN (HP P6000)  will give any clues about MP Lan,

 

--

 

btw, i am searching for KUDO's button :(

San
Dennis Handly
Acclaimed Contributor

Re: (/dev/fclp1) : detected that device id 0xffffff, PWWN 0x8004070000000040 is offline.

>I am searching for KUDOs button

 

The Kudos buttons are those white stars to the left of each post.

Steven Schweda
Honored Contributor

Re: (/dev/fclp1) : detected that device id 0xffffff, PWWN 0x8004070000000040 is offline.

 
Hiren N Dave
Valued Contributor

Re: (/dev/fclp1) : detected that device id 0xffffff, PWWN 0x8004070000000040 is offline.

Yes, the sar output was misinterpreted.

 

It certainly looks like network problem.

I am an HP Employee

Was this post useful? - You may click the KUDOS! star to say thank you.
SanjeevRao
Frequent Visitor

Re: (/dev/fclp1) : detected that device id 0xffffff, PWWN 0x8004070000000040 is offline.

Hmm, past couple of days had a tough time with Network Operations team, Changed network cable, gathered the switch/fw logs, verified them to find any issues along with the Network team members, verified the cable/port connecting it to another standby linux (HP Proliant servers {tried with 2 servers}) got a good response from those two servers,  after that network team says firmly its an issue with HP server :-(,

With the above commands you referred, i tried most of them and attached here in .txt format,

 

 

San
Hiren N Dave
Valued Contributor

Re: (/dev/fclp1) : detected that device id 0xffffff, PWWN 0x8004070000000040 is offline.

 

Well, seems you have got your MP IP now.

 

Why this output shows SCAN instead of CLAIMED? Out of four paths, two paths might have problem here.

 

disk      6  0/0/0/9/0/0/0/4/0/0/0.1.22.0.0.0.1  sdisk   SCAN        DEVICE       HP      HSV300
                            /dev/dsk/c8t0d1   /dev/rdsk/c8t0d1
disk     11  0/0/0/9/0/0/0/4/0/0/0.1.22.0.0.0.2  sdisk   SCAN        DEVICE       HP      HSV300
                            /dev/dsk/c8t0d2   /dev/rdsk/c8t0d2
disk      7  0/0/0/9/0/0/0/4/0/0/0.1.23.0.0.0.1  sdisk   SCAN        DEVICE       HP      HSV300
                            /dev/dsk/c10t0d1   /dev/rdsk/c10t0d1
disk     12  0/0/0/9/0/0/0/4/0/0/0.1.23.0.0.0.2  sdisk   SCAN        DEVICE       HP      HSV300
                            /dev/dsk/c10t0d2   /dev/rdsk/c10t0d2


Anyway, it should not affect the network packet loss as the CPU usage is also normal so you might want to check network parameters like autonegotiation settings, duplicate IP, collision, any known patch level issue etc.

 

I recommend to log a case with HP solution center for detailed log verification.

 

I am an HP Employee

Was this post useful? - You may click the KUDOS! star to say thank you.
SanjeevRao
Frequent Visitor

Re: (/dev/fclp1) : detected that device id 0xffffff, PWWN 0x8004070000000040 is offline.

Still that doesnt look like MP ip,

 

Not able to ping them, not able to access it

 

yes will log a ticket with HP Soln centre,

San