(/dev/fclp1) : detected that device id 0xffffff, PWWN 0x8004070000000040 is offline.

[SanjeevRao]

Dear team

Recent days, one of the Rx2800 Server started performing really slow, CPU utilisation (sar -u) shows

16:09:31    %usr    %sys    %wio   %idle

16:09:33       4       0       0      96
16:09:35      44       0       0      56

(though this is slave app machine in active-active) network, other servers are really performing well,

pinging from one machine to this machine shows me 10% packet loss,  completely went through the network switch/firewall logs in terms of any network latency, but no error msg found so far,

ping result from neighbouring machine in the same VLAN

ping 1921.68.0.4
PING 1921.68.0.4: 64 byte packets
64 bytes from 1921.68.0.4: icmp_seq=0. time=0. ms
64 bytes from 1921.68.0.4: icmp_seq=1. time=0. ms
64 bytes from 1921.68.0.4: icmp_seq=2. time=1028. ms
64 bytes from 1921.68.0.4: icmp_seq=3. time=18. ms
64 bytes from 1921.68.0.4: icmp_seq=4. time=0. ms
64 bytes from 1921.68.0.4: icmp_seq=5. time=0. ms
64 bytes from 1921.68.0.4: icmp_seq=6. time=1226. ms
64 bytes from 1921.68.0.4: icmp_seq=7. time=216. ms
64 bytes from 1921.68.0.4: icmp_seq=8. time=0. ms
64 bytes from 1921.68.0.4: icmp_seq=23. time=1600. ms
64 bytes from 1921.68.0.4: icmp_seq=24. time=590. ms
64 bytes from 1921.68.0.4: icmp_seq=25. time=0. ms
64 bytes from 1921.68.0.4: icmp_seq=26. time=0. ms
64 bytes from 1921.68.0.4: icmp_seq=27. time=1534. ms
64 bytes from 1921.68.0.4: icmp_seq=28. time=524. ms
64 bytes from 1921.68.0.4: icmp_seq=29. time=0. ms
64 bytes from 1921.68.0.4: icmp_seq=30. time=0. ms
64 bytes from 1921.68.0.4: icmp_seq=31. time=1674. ms
64 bytes from 1921.68.0.4: icmp_seq=32. time=664. ms
64 bytes from 1921.68.0.4: icmp_seq=33. time=0. ms
----1921.68.0.4 PING Statistics----
packets transmitted,  packets received, 0% packet loss
round-trip (ms)  min/avg/max = 0/276/122

Now one of the HP friend says login to MP and check any I/O related issue in this machine? unfortunately, i dont have the MP ip with me

any clues/guidance will be greatly appreciated.

P.S. this thread has been moved from Servers > Integrity Servers to HP-UX > sysadmin - HP Forums Moderator

[Hiren N Dave]

Hi,

The %wio is the percentage of time the CPU spent on blocked I/O request.

Here, the %wio is very high which indicates a disk bottleneck.

You may want to use sar -d and iostat commands to find out if any disk problem is causing this issue.

Cheers.

[SanjeevRao]

thanks Mr. Hiren

yes you are right, i was suspecting I/O as well, but didnt have details about MP connectivity,

below are sar -d result, when i tried sar -d 2 10, it doesnt show any result

19:06:23   device   %busy   avque   r+w/s  blks/s  avwait  avserv
19:06:27
        
19:06:29
19:06:31
19:06:33
19:06:36
19:06:37
19:06:40
19:06:41
19:06:43

so i tried  #sar -d 2 2

19:06:56   device   %busy   avque   r+w/s  blks/s  avwait  avserv
19:06:58
19:07:00   disk13    0.46    0.50      10     304    0.00    0.91

Average    disk13    0.24    0.50       5     159    0.00    0.91

is there any ways i can obtain the mp lan ip details from the servers itself? or the SAN (HP P6000)  will give any clues about MP Lan,

--

btw, i am searching for KUDO's button :(

[Dennis Handly]

>I am searching for KUDOs button

The Kudos buttons are those white stars to the left of each post.

[Hiren N Dave]

Yes, the sar output was misinterpreted.

It certainly looks like network problem.

[SanjeevRao]

Hmm, past couple of days had a tough time with Network Operations team, Changed network cable, gathered the switch/fw logs, verified them to find any issues along with the Network team members, verified the cable/port connecting it to another standby linux (HP Proliant servers {tried with 2 servers}) got a good response from those two servers,  after that network team says firmly its an issue with HP server :-(,

With the above commands you referred, i tried most of them and attached here in .txt format,

[Hiren N Dave]

Well, seems you have got your MP IP now.

Why this output shows SCAN instead of CLAIMED? Out of four paths, two paths might have problem here.

disk      6  0/0/0/9/0/0/0/4/0/0/0.1.22.0.0.0.1  sdisk   SCAN        DEVICE       HP      HSV300
                            /dev/dsk/c8t0d1   /dev/rdsk/c8t0d1
disk     11  0/0/0/9/0/0/0/4/0/0/0.1.22.0.0.0.2  sdisk   SCAN        DEVICE       HP      HSV300
                            /dev/dsk/c8t0d2   /dev/rdsk/c8t0d2
disk      7  0/0/0/9/0/0/0/4/0/0/0.1.23.0.0.0.1  sdisk   SCAN        DEVICE       HP      HSV300
                            /dev/dsk/c10t0d1   /dev/rdsk/c10t0d1
disk     12  0/0/0/9/0/0/0/4/0/0/0.1.23.0.0.0.2  sdisk   SCAN        DEVICE       HP      HSV300
                            /dev/dsk/c10t0d2   /dev/rdsk/c10t0d2

Anyway, it should not affect the network packet loss as the CPU usage is also normal so you might want to check network parameters like autonegotiation settings, duplicate IP, collision, any known patch level issue etc.

I recommend to log a case with HP solution center for detailed log verification.

[SanjeevRao]

Still that doesnt look like MP ip,

Not able to ping them, not able to access it

yes will log a ticket with HP Soln centre,