HP8212zl J9091A - PD Over Current indication - FortiAP issue

[tmasazo]

HP8212zl  J9091A - IMAGE: K.15.18.0019 Boot Rom: K.15.30
HP J9534A 24p Gig-T PoE+ v2 zl Module

We have a HP8212zl that I moved a FortiAP 222B to module D (HP J9534A 24p Gig-T PoE+ v2 zl)  last week and since then have had two "PD Over Current indication" events.

W 01/02/19 14:45:08 00562 ports: AM1: port D23 PD Over Current indication.                                                            W 12/31/18 14:37:30 00562 ports: AM1: port D23 PD Over Current indication.

I brought the AP online on the 28th from Module I  (HP J9534A 24p Gig-T PoE+ v2 zl)

I 12/28/18 12:05:18 00076 ports: AM1: port D23 is now on-line
I 12/28/18 12:05:11 00561 ports: AM1: port D23 Applying Power to PD.
I 12/28/18 12:05:11 00560 ports: AM1: port D23 PD Detected.

And the unit was previously on a different module with no issues. The module D I have it on now has 2 FortiAP 321c, 1 FortiAP 222B and 13 YeaLInk PoE Phones.

We should have Plenty of power available so right now my working "theory" is that I have too many PoE devices on one module?!! (That just doesn't make sense)

# show power-over-ethernet

Status and Counters - System Power Status

System Power Status : No redundancy
PoE Power Status : No redundancy

Chassis power-over-ethernet:

Total Available Power : 1200 W
Total Failover Power : 900 W
Total Redundancy Power : 0 W
Total Used Power : 186 W +/- 6W
Total Remaining Power : 1014 W

Internal Power

Main Power
PS (Watts) Status
----- ------------- ---------------------
1 300 POE+ Connected
2 300 POE+ Connected
3 300 POE+ Connected
4 300 POE+ Connected

External Power
EPS1 /Not Connected.
EPS2 /Not Connected.

So, I am trying to reach out for some help in troubleshooting this as I am finding interface output, PoE Commcands. LLDP and Logs not really giving me enough details to diag the issue. 

The FortiAP CLI doesn't help much either as it is limited to pretty much just showing me the config output so before I go and engage them I thought I would come reach out to the community first and see if I could gain some knowlege on HPE\ARUBA PoE troubleshooting as my main back ground has been in cisco and it only carries me so far.

[parnassus]

What's about to verify with show power-over-ethernet brief and show power-over-ethernet slot D commands?

[tmasazo]

Here is the output of those commands. (Just module D not sure if you are wanting to see all of the modules)

Module D Power
Available: 180 W Used: 52 W Remaining: 128 W

PoE | Power Power Alloc Alloc Actual Config Detection Power Pre-std
Port | Enable Priority By Power Power Type Status Class Detect
------ + ------ -------- ----- ----- ------ -------- ----------- ----- ------
D1 | Yes low usage 17 W 2.5 W Delivering 2 off
D2 | Yes low usage 17 W 2.2 W Delivering 2 off
D3 | Yes low usage 17 W 2.1 W Delivering 2 off
D4 | Yes low usage 17 W 2.1 W Delivering 2 off
D5 | Yes low usage 17 W 2.0 W Delivering 2 off
D6 | Yes low usage 17 W 2.0 W Delivering 2 off
D7 | Yes low usage 17 W 2.1 W Delivering 2 off
D8 | Yes low usage 17 W 2.1 W Delivering 2 off
D9 | Yes low usage 17 W 0.0 W Searching 0 off
D10 | Yes low usage 17 W 2.1 W Delivering 2 off
D11 | Yes low usage 17 W 2.2 W Delivering 2 off
D12 | Yes low usage 17 W 0.0 W Searching 0 off
D13 | Yes low usage 17 W 2.0 W Delivering 2 off
D14 | Yes low usage 17 W 0.0 W Searching 0 off
D15 | No low usage 17 W 0.0 W Disabled 0 off
D16 | Yes low usage 17 W 2.1 W Delivering 2 off
D17 | Yes low usage 17 W 7.3 W Delivering 4 off
D18 | Yes low usage 17 W 7.2 W Delivering 4 off
D19 | Yes low usage 17 W 0.0 W Searching 0 off
D20 | Yes low usage 17 W 0.0 W Searching 0 off
D21 | Yes low usage 17 W 0.0 W Searching 0 off
D22 | Yes low usage 17 W 0.0 W Searching 0 off
D23 | Yes low usage 17 W 8.6 W Delivering 4 off
D24 | Yes low usage 17 W 2.2 W Delivering 2 off

# show power-over-ethernet slot D

Status and Counters - System Power Status for slot D

Maximum Power : 179 W Operational Status : On
Power In Use : 52 W +/- 6 W Usage Threshold (%) : 80

What I am trying to articulate is that I have not seen anything in the output that shows anything out of the ordinary that is causing a "over Current"  issue. Seems to have happen close to the same time on the two different days so I thought about going into the telco closet today around 14:30 and watching to see if maybe something is kicking on that is not connected to the switch and somehow interfering with it but I just don't see that being the cause.

[tmasazo]

So I see that you can view "Current" level output in "mA" from the GUI but have NOT seen ANY output from the CLI that shows this info. 

My question now is, Are the "Over Current" errors due to "Current\mA" being drawn from the AP or is being caused by the "Power" output in "x.x watts"?

I have watched port D23 in the GUI and seen the "Current" go as low as "130 mA" and as high as the picture "169 mA"

[tmasazo]

Here is the interface getting up to power = 12.7 watts / Current = 229 mA and interface remains up with no new "Current Overload"

[tmasazo]

Dropped Again today:

I 01/04/19 10:58:14 00561 ports: AM1: port D23 Applying Power to PD.
I 01/04/19 10:58:14 00560 ports: AM1: port D23 PD Detected.
I 01/04/19 10:58:12 00077 ports: AM1: port D23 is now off-line
W 01/04/19 10:58:11 00562 ports: AM1: port D23 PD Over Current indication.

I Enabled LLDP-MED yesterday hoping that the AP would communicate its PoE requirements with LLDP instead of "usage"

D23 | Yes low lldp 17 W 13.0 W Delivering 4 off
D24 | Yes low usage 17 W 2.1 W Delivering 2 off

I find it fustrating that I can only obtain the "mA" output from the GUI and not from any CLI commands.

My Working Theory now is that the AP is requesting to much "WATTS"  which in turn is pushing the "mA" too high triggering the "Over Current", But the port is negotiating with a class 4 (12.95–25.50)  so is it hitting over 25.5? no way to tell as there are no outputs or logging that will indicate this unless you sit there and refresh the screen\CLI. 

Class 4 should be "High" priority and we have all our PoE set for the default of "Low" so perhaps that is my issue?
That doesn't explain why I had no issues when this AP was on Module I previously and only now has issue being on Module D.

[tmasazo]

Just got back and found that the AP has dropped 4 more times since my earlier posting

# show log -r | i D23
I 01/04/19 13:04:48 00076 ports: AM1: port D23 is now on-line
I 01/04/19 13:04:45 00077 ports: AM1: port D23 is now off-line
I 01/04/19 13:04:33 00076 ports: AM1: port D23 is now on-line
I 01/04/19 13:04:27 00077 ports: AM1: port D23 is now off-line
I 01/04/19 13:04:18 00076 ports: AM1: port D23 is now on-line
I 01/04/19 13:04:11 00561 ports: AM1: port D23 Applying Power to PD.
I 01/04/19 13:04:11 00560 ports: AM1: port D23 PD Detected.
I 01/04/19 13:04:08 00077 ports: AM1: port D23 is now off-line
W 01/04/19 13:04:08 00562 ports: AM1: port D23 PD Over Current indication.
I 01/04/19 13:02:11 00076 ports: AM1: port D23 is now on-line
I 01/04/19 13:02:07 00077 ports: AM1: port D23 is now off-line
I 01/04/19 13:01:56 00076 ports: AM1: port D23 is now on-line
I 01/04/19 13:01:50 00077 ports: AM1: port D23 is now off-line
I 01/04/19 13:01:40 00076 ports: AM1: port D23 is now on-line
I 01/04/19 13:01:33 00561 ports: AM1: port D23 Applying Power to PD.
I 01/04/19 13:01:33 00560 ports: AM1: port D23 PD Detected.
I 01/04/19 13:01:31 00077 ports: AM1: port D23 is now off-line
W 01/04/19 13:01:30 00562 ports: AM1: port D23 PD Over Current indication.
I 01/04/19 12:49:29 00076 ports: AM1: port D23 is now on-line
I 01/04/19 12:49:26 00077 ports: AM1: port D23 is now off-line
I 01/04/19 12:49:18 00076 ports: AM1: port D23 is now on-line
I 01/04/19 12:49:07 00077 ports: AM1: port D23 is now off-line
I 01/04/19 12:48:58 00076 ports: AM1: port D23 is now on-line
I 01/04/19 12:48:51 00561 ports: AM1: port D23 Applying Power to PD.
I 01/04/19 12:48:51 00560 ports: AM1: port D23 PD Detected.
W 01/04/19 12:48:49 00562 ports: AM1: port D23 PD Over Current indication.
I 01/04/19 12:48:48 00077 ports: AM1: port D23 is now off-line
I 01/04/19 12:26:07 00076 ports: AM1: port D23 is now on-line
I 01/04/19 12:26:04 00077 ports: AM1: port D23 is now off-line
I 01/04/19 12:25:57 00076 ports: AM1: port D23 is now on-line
I 01/04/19 12:25:46 00077 ports: AM1: port D23 is now off-line
I 01/04/19 12:25:36 00076 ports: AM1: port D23 is now on-line
I 01/04/19 12:25:29 00561 ports: AM1: port D23 Applying Power to PD.
I 01/04/19 12:25:29 00560 ports: AM1: port D23 PD Detected.
I 01/04/19 12:25:26 00077 ports: AM1: port D23 is now off-line
W 01/04/19 12:25:26 00562 ports: AM1: port D23 PD Over Current indication.

I had ran a script to output "show power-over-ethernet br D23" over and over into a text file that should have caught all 4 of these events.  I found only 1 "Fault" event

# show power-over-ethernet br D23

Status and Counters - Port Power Status

PoE | Power Power Alloc Alloc Actual Config Detection Power Pre-std
Port | Enable Priority By Power Power Type Status Class Detect
------ + ------ -------- ----- ----- ------ -------- ----------- ----- ------
D23 | Yes low lldp 17 W 10.2 W Delivering 4 off

# show power-over-ethernet br D23

Status and Counters - Port Power Status

PoE | Power Power Alloc Alloc Actual Config Detection Power Pre-std
Port | Enable Priority By Power Power Type Status Class Detect
------ + ------ -------- ----- ----- ------ -------- ----------- ----- ------
D23 | Yes low lldp 17 W 0.0 W Fault 4 off  <- AP went "Over Current" ?

# show power-over-ethernet br D23

Status and Counters - Port Power Status

PoE | Power Power Alloc Alloc Actual Config Detection Power Pre-std
Port | Enable Priority By Power Power Type Status Class Detect
------ + ------ -------- ----- ----- ------ -------- ----------- ----- ------
D23 | Yes low lldp 17 W 13.0 W Delivering 4 off <- AP Back Online 

[tmasazo]

Continuing my scrub through the log file created by the script and being that PoE is either "Delievering" or "Searching" searched for "Searching"

# show power-over-ethernet br D23

Status and Counters - Port Power Status

PoE | Power Power Alloc Alloc Actual Config Detection Power Pre-std
Port | Enable Priority By Power Power Type Status Class Detect
------ + ------ -------- ----- ----- ------ -------- ----------- ----- ------
D23 | Yes low lldp 17 W 8.5 W Delivering 4 off

# show power-over-ethernet br D23

Status and Counters - Port Power Status

PoE | Power Power Alloc Alloc Actual Config Detection Power Pre-std
Port | Enable Priority By Power Power Type Status Class Detect
------ + ------ -------- ----- ----- ------ -------- ----------- ----- ------
D23 | Yes low lldp 17 W 12.2 W Delivering 4 off  < - Ramps up to 12.2 Watts

# show power-over-ethernet br D23

Status and Counters - Port Power Status

PoE | Power Power Alloc Alloc Actual Config Detection Power Pre-std
Port | Enable Priority By Power Power Type Status Class Detect
------ + ------ -------- ----- ----- ------ -------- ----------- ----- ------
D23 | Yes low lldp 17 W 12.2 W Delivering 4 off <- Still hanging at 12.2 Watts

# show power-over-ethernet br D23

Status and Counters - Port Power Status

PoE | Power Power Alloc Alloc Actual Config Detection Power Pre-std
Port | Enable Priority By Power Power Type Status Class Detect
------ + ------ -------- ----- ----- ------ -------- ----------- ----- ------
D23 | Yes low lldp 17 W 0.0 W Searching 0 off  <- BOOM, the slow refresh of the CLI might have missed the "Fault" but caught the "Searching" status of the PoE

# show power-over-ethernet br D23

Status and Counters - Port Power Status

PoE | Power Power Alloc Alloc Actual Config Detection Power Pre-std
Port | Enable Priority By Power Power Type Status Class Detect
------ + ------ -------- ----- ----- ------ -------- ----------- ----- ------
D23 | Yes low lldp 17 W 4.1 W Delivering 4 off <- AP powering back up 

So, Do I need to change the AP to priority "High"? and, why would this change as it was on Module I previously with Default settings for geez..... a year? and, no power issues. 

This then goes back to my original question, Do I have too many PoE devices connected on my PoE blade? or is it that I have too many Class 4 PoE devices now on Module D?

[tmasazo]

Ok, I really don't know what to do about this so I have moved the AP to Module J  (HP J9534A 24p Gig-T PoE+ v2 zl) as it only has 3 Class 2 PoE devices on it (Yea Link Phones)

Module J Power
Available: 144 W Used: 16 W Remaining: 128 W

PoE | Power Power Alloc Alloc Actual Config Detection Power Pre-std
Port | Enable Priority By Power Power Type Status Class Detect
------ + ------ -------- ----- ----- ------ -------- ----------- ----- ------
J1 | Yes low usage 17 W 2.2 W Delivering 2 off
J2 | Yes low usage 17 W 2.8 W Delivering 2 off
J3 | Yes low usage 17 W 0.0 W Searching 0 off
J4 | Yes low usage 17 W 0.0 W Searching 0 off
J5 | Yes low usage 17 W 0.0 W Searching 0 off
J6 | Yes low usage 17 W 0.0 W Searching 0 off
J7 | Yes low usage 17 W 0.0 W Searching 0 off
J8 | Yes low usage 17 W 0.0 W Searching 0 off
J9 | Yes low usage 17 W 0.0 W Searching 0 off
J10 | Yes low usage 17 W 2.7 W Delivering 2 off
J11 | Yes low usage 17 W 0.0 W Searching 0 off
J12 | Yes low usage 17 W 0.0 W Searching 0 off
J13 | Yes low usage 17 W 0.0 W Searching 0 off
J14 | Yes low usage 17 W 0.0 W Searching 0 off
J15 | Yes low usage 17 W 0.0 W Searching 0 off
J16 | Yes low usage 17 W 0.0 W Searching 0 off
J17 | Yes low usage 17 W 0.0 W Searching 0 off
J18 | Yes low usage 17 W 0.0 W Searching 0 off
J19 | Yes low usage 17 W 0.0 W Searching 0 off
J20 | Yes low usage 17 W 0.0 W Searching 0 off
J21 | Yes low usage 17 W 0.0 W Searching 0 off
J22 | Yes low usage 17 W 0.0 W Searching 0 off
J23 | Yes low lldp 17 W 7.6 W Delivering 4 off  <- The AP 
J24 | Yes low usage 17 W 0.0 W Searching 0 off

Going to monitor now and see it I continue to have issues now on this module 

[tmasazo]

1- I was wrong on not being able to see "mA" in the CLI ( show power-over-ethernet [port]) 

# show power-over-ethernet j23

Status and Counters - Port Power Status for port J23

Power Enable : Yes
LLDP Detect : enabled
Priority : low Configured Type :
AllocateBy : usage Value : 17 W
Power Class : 4 Detection Status : Delivering

Over Current Cnt : 0 MPS Absent Cnt : 0
Power Denied Cnt : 0 Short Cnt : 0

Voltage : 55.5 V Current : 141 mA
Power : 7.8 W Pre-std Detect : off

Since moving the AP to Module " J " I have not have any "over current"  issues.

So, Is the end result that i had too many PoE devices on Module D?