HPE ILO integration -> NO interfaces retrieved

[RPapaux]

Hello,

I have added HPE Proliant ILOcard into IMC (0705P10).

I receive the correct MIB-II information but no interfaces, even though the ILO sees all server interfaces.

The MIB browser sees all of them as well.

This is crucial for me because I want IMC to draw the connection between all the server interfaces (there are actually 6) and the edge network switches.

The ILO is configure as SNMPv3 node but I could not find any ILO field to configure the V3 “context” so I left it blank in the IMC SNMP template.  

Any idea of what can be wrong and in which logfiles I shall look at? Or any ILO integration hints is welcome.

Thanks a lot

Ray

[jguse]

Hello Ray,

I'm not sure which iLO version you are using, but I checked this in my lab with an iLO 5 using SNMPv2c / v3 and had the same symptoms. Looking at this in more detail using a packet capture, I could see that it appears to be caused by lacking responses from the iLO in regards to the SNMP GetBulkRequest sent by iMC to retrieve the interface data in IF-MIB.

During the device add/sync process, iMC is sending a bulk request for the ifTable values under 1.3.6.1.2.1.2.2.1.x (x = instance), and the iLO is responding with a packet that contains none of the values requested, nor any error.

That's quite odd, as I can see other GetBulkRequests from iMC to iLO that receive an appropriate response. Either way though, it's not an issue on iMC side, as it won't be able to show an Interface List when it can't be retrieved via SNMP.

I think you need to open an HPE Support case with the Server team to take this further, as the iLO's SNMP agent response is the cause of this issue.

[RPapaux]

Hello Justin,

Sorry for the delay, and thank you for the investigation.

ILO F/W version is 2.31

I'm however concerned about your trace.

I have browsed the MIB-II using the IMC MIB brower, and the ILO did reply with the correct interfaces (see below).

So what is the difference between manual browsing and IMC browsing? is ther a specifc logfile for this.

Thanks again

Ray

***** SNMP WALK REQUEST STARTED *****
1) 1.3.6.1.2.1.2.1.0 (ifNumber.0) 6
2) 1.3.6.1.2.1.2.2.1.1.1 (ifIndex.1) 1
3) 1.3.6.1.2.1.2.2.1.1.2 (ifIndex.2) 2
4) 1.3.6.1.2.1.2.2.1.1.3 (ifIndex.3) 3
5) 1.3.6.1.2.1.2.2.1.1.4 (ifIndex.4) 4
6) 1.3.6.1.2.1.2.2.1.1.5 (ifIndex.5) 5
7) 1.3.6.1.2.1.2.2.1.1.6 (ifIndex.6) 6
1.3.6.1.2.1.2.2.1.2.1 (ifDescr.1) HPE Ethernet 1Gb 4-port 331i Adapter - NIC
9) 1.3.6.1.2.1.2.2.1.2.2 (ifDescr.2) HPE Ethernet 1Gb 4-port 331i Adapter - NIC
10) 1.3.6.1.2.1.2.2.1.2.3 (ifDescr.3) HPE Ethernet 1Gb 4-port 331i Adapter - NIC
11) 1.3.6.1.2.1.2.2.1.2.4 (ifDescr.4) HPE Ethernet 1Gb 4-port 331i Adapter - NIC
12) 1.3.6.1.2.1.2.2.1.2.5 (ifDescr.5) HPE Ethernet 10Gb 2-port 562FLR-SFP+ Adapter - NIC
13) 1.3.6.1.2.1.2.2.1.2.6 (ifDescr.6) HPE Ethernet 10Gb 2-port 562FLR-SFP+ Adapter - NIC
14) 1.3.6.1.2.1.2.2.1.3.1 (ifType.1) ethernet-csmacd(6)
15) 1.3.6.1.2.1.2.2.1.3.2 (ifType.2) ethernet-csmacd(6)
16) 1.3.6.1.2.1.2.2.1.3.3 (ifType.3) ethernet-csmacd(6)
17) 1.3.6.1.2.1.2.2.1.3.4 (ifType.4) ethernet-csmacd(6)
18) 1.3.6.1.2.1.2.2.1.3.5 (ifType.5) ethernet-csmacd(6)
19) 1.3.6.1.2.1.2.2.1.3.6 (ifType.6) ethernet-csmacd(6)
20) 1.3.6.1.2.1.2.2.1.4.1 (ifMtu.1) 1500
21) 1.3.6.1.2.1.2.2.1.4.2 (ifMtu.2) 1500
22) 1.3.6.1.2.1.2.2.1.4.3 (ifMtu.3) 1500
23) 1.3.6.1.2.1.2.2.1.4.4 (ifMtu.4) 1500
24) 1.3.6.1.2.1.2.2.1.4.5 (ifMtu.5) 1500
25) 1.3.6.1.2.1.2.2.1.4.6 (ifMtu.6) 1500
26) 1.3.6.1.2.1.2.2.1.5.1 (ifSpeed.1) 0
27) 1.3.6.1.2.1.2.2.1.5.2 (ifSpeed.2) 0
28) 1.3.6.1.2.1.2.2.1.5.3 (ifSpeed.3) 0
29) 1.3.6.1.2.1.2.2.1.5.4 (ifSpeed.4) 0
30) 1.3.6.1.2.1.2.2.1.5.5 (ifSpeed.5) 4294967295
31) 1.3.6.1.2.1.2.2.1.5.6 (ifSpeed.6) 4294967295
32) 1.3.6.1.2.1.2.2.1.6.1 (ifPhysAddress.1) 94:40:C9:30:EF:48
33) 1.3.6.1.2.1.2.2.1.6.2 (ifPhysAddress.2) 94:40:C9:30:EF:49
34) 1.3.6.1.2.1.2.2.1.6.3 (ifPhysAddress.3) 94:40:C9:30:EF:4A
35) 1.3.6.1.2.1.2.2.1.6.4 (ifPhysAddress.4) 94:40:C9:30:EF:4B
36) 1.3.6.1.2.1.2.2.1.6.5 (ifPhysAddress.5) D4:F5:EF:0E:DB:B0
37) 1.3.6.1.2.1.2.2.1.6.6 (ifPhysAddress.6) D4:F5:EF:0E:DB:B8
38) 1.3.6.1.2.1.2.2.1.7.1 (ifAdminStatus.1) down(2)
39) 1.3.6.1.2.1.2.2.1.7.2 (ifAdminStatus.2) down(2)
40) 1.3.6.1.2.1.2.2.1.7.3 (ifAdminStatus.3) down(2)
41) 1.3.6.1.2.1.2.2.1.7.4 (ifAdminStatus.4) down(2)
42) 1.3.6.1.2.1.2.2.1.7.5 (ifAdminStatus.5) up(1)
43) 1.3.6.1.2.1.2.2.1.7.6 (ifAdminStatus.6) up(1)
44) 1.3.6.1.2.1.2.2.1.8.1 (ifOperStatus.1) down(2)
45) 1.3.6.1.2.1.2.2.1.8.2 (ifOperStatus.2) down(2)
46) 1.3.6.1.2.1.2.2.1.8.3 (ifOperStatus.3) down(2)
47) 1.3.6.1.2.1.2.2.1.8.4 (ifOperStatus.4) down(2)
48) 1.3.6.1.2.1.2.2.1.8.5 (ifOperStatus.5) up(1)
49) 1.3.6.1.2.1.2.2.1.8.6 (ifOperStatus.6) up(1)
50) 1.3.6.1.2.1.2.2.1.9.1 (ifLastChange.1) 0:00:00.00
51) 1.3.6.1.2.1.2.2.1.9.2 (ifLastChange.2) 0:00:00.00
52) 1.3.6.1.2.1.2.2.1.9.3 (ifLastChange.3) 0:00:00.00
53) 1.3.6.1.2.1.2.2.1.9.4 (ifLastChange.4) 0:00:00.00
54) 1.3.6.1.2.1.2.2.1.9.5 (ifLastChange.5) 0:00:00.00
55) 1.3.6.1.2.1.2.2.1.9.6 (ifLastChange.6) 0:00:00.00
56) 1.3.6.1.2.1.2.2.1.10.1 (ifInOctets.1) 0
57) 1.3.6.1.2.1.2.2.1.10.2 (ifInOctets.2) 0
58) 1.3.6.1.2.1.2.2.1.10.3 (ifInOctets.3) 0
59) 1.3.6.1.2.1.2.2.1.10.4 (ifInOctets.4) 0
60) 1.3.6.1.2.1.2.2.1.10.5 (ifInOctets.5) 3843319330
61) 1.3.6.1.2.1.2.2.1.10.6 (ifInOctets.6) 3744716799
62) 1.3.6.1.2.1.2.2.1.11.1 (ifInUcastPkts.1) 0
63) 1.3.6.1.2.1.2.2.1.11.2 (ifInUcastPkts.2) 0
64) 1.3.6.1.2.1.2.2.1.11.3 (ifInUcastPkts.3) 0
65) 1.3.6.1.2.1.2.2.1.11.4 (ifInUcastPkts.4) 0
66) 1.3.6.1.2.1.2.2.1.11.5 (ifInUcastPkts.5) 88556609
67) 1.3.6.1.2.1.2.2.1.11.6 (ifInUcastPkts.6) 88554620
68) 1.3.6.1.2.1.2.2.1.12.1 (ifInNUcastPkts.1) 0
69) 1.3.6.1.2.1.2.2.1.12.2 (ifInNUcastPkts.2) 0
70) 1.3.6.1.2.1.2.2.1.12.3 (ifInNUcastPkts.3) 0
71) 1.3.6.1.2.1.2.2.1.12.4 (ifInNUcastPkts.4) 0
72) 1.3.6.1.2.1.2.2.1.12.5 (ifInNUcastPkts.5) 88556609
73) 1.3.6.1.2.1.2.2.1.12.6 (ifInNUcastPkts.6) 88554620
74) 1.3.6.1.2.1.2.2.1.13.1 (ifInDiscards.1) 0
75) 1.3.6.1.2.1.2.2.1.13.2 (ifInDiscards.2) 0
76) 1.3.6.1.2.1.2.2.1.13.3 (ifInDiscards.3) 0
77) 1.3.6.1.2.1.2.2.1.13.4 (ifInDiscards.4) 0
78) 1.3.6.1.2.1.2.2.1.13.5 (ifInDiscards.5) 0
79) 1.3.6.1.2.1.2.2.1.13.6 (ifInDiscards.6) 0
80) 1.3.6.1.2.1.2.2.1.14.1 (ifInErrors.1) 0
81) 1.3.6.1.2.1.2.2.1.14.2 (ifInErrors.2) 0
82) 1.3.6.1.2.1.2.2.1.14.3 (ifInErrors.3) 0
83) 1.3.6.1.2.1.2.2.1.14.4 (ifInErrors.4) 0
84) 1.3.6.1.2.1.2.2.1.14.5 (ifInErrors.5) 232480
85) 1.3.6.1.2.1.2.2.1.14.6 (ifInErrors.6) 232480
86) 1.3.6.1.2.1.2.2.1.15.1 (ifInUnknownProtos.1) 0
87) 1.3.6.1.2.1.2.2.1.15.2 (ifInUnknownProtos.2) 0
88) 1.3.6.1.2.1.2.2.1.15.3 (ifInUnknownProtos.3) 0
89) 1.3.6.1.2.1.2.2.1.15.4 (ifInUnknownProtos.4) 0
90) 1.3.6.1.2.1.2.2.1.15.5 (ifInUnknownProtos.5) 0
91) 1.3.6.1.2.1.2.2.1.15.6 (ifInUnknownProtos.6) 0
92) 1.3.6.1.2.1.2.2.1.16.1 (ifOutOctets.1) 0
93) 1.3.6.1.2.1.2.2.1.16.2 (ifOutOctets.2) 0
94) 1.3.6.1.2.1.2.2.1.16.3 (ifOutOctets.3) 0
95) 1.3.6.1.2.1.2.2.1.16.4 (ifOutOctets.4) 0
96) 1.3.6.1.2.1.2.2.1.16.5 (ifOutOctets.5) 24088620
97) 1.3.6.1.2.1.2.2.1.16.6 (ifOutOctets.6) 1080030452
98) 1.3.6.1.2.1.2.2.1.17.1 (ifOutUcastPkts.1) 0
99) 1.3.6.1.2.1.2.2.1.17.2 (ifOutUcastPkts.2) 0
100) 1.3.6.1.2.1.2.2.1.17.3 (ifOutUcastPkts.3) 0
101) 1.3.6.1.2.1.2.2.1.17.4 (ifOutUcastPkts.4) 0
102) 1.3.6.1.2.1.2.2.1.17.5 (ifOutUcastPkts.5) 0
103) 1.3.6.1.2.1.2.2.1.17.6 (ifOutUcastPkts.6) 0
104) 1.3.6.1.2.1.2.2.1.18.1 (ifOutNUcastPkts.1) 0
105) 1.3.6.1.2.1.2.2.1.18.2 (ifOutNUcastPkts.2) 0
106) 1.3.6.1.2.1.2.2.1.18.3 (ifOutNUcastPkts.3) 0
107) 1.3.6.1.2.1.2.2.1.18.4 (ifOutNUcastPkts.4) 0
108) 1.3.6.1.2.1.2.2.1.18.5 (ifOutNUcastPkts.5) 0
109) 1.3.6.1.2.1.2.2.1.18.6 (ifOutNUcastPkts.6) 0
110) 1.3.6.1.2.1.2.2.1.19.1 (ifOutDiscards.1) 0
111) 1.3.6.1.2.1.2.2.1.19.2 (ifOutDiscards.2) 0
112) 1.3.6.1.2.1.2.2.1.19.3 (ifOutDiscards.3) 0
113) 1.3.6.1.2.1.2.2.1.19.4 (ifOutDiscards.4) 0
114) 1.3.6.1.2.1.2.2.1.19.5 (ifOutDiscards.5) 0
115) 1.3.6.1.2.1.2.2.1.19.6 (ifOutDiscards.6) 0
116) 1.3.6.1.2.1.2.2.1.20.1 (ifOutErrors.1) 0
117) 1.3.6.1.2.1.2.2.1.20.2 (ifOutErrors.2) 0
118) 1.3.6.1.2.1.2.2.1.20.3 (ifOutErrors.3) 0
119) 1.3.6.1.2.1.2.2.1.20.4 (ifOutErrors.4) 0
120) 1.3.6.1.2.1.2.2.1.20.5 (ifOutErrors.5) 0
121) 1.3.6.1.2.1.2.2.1.20.6 (ifOutErrors.6) 0
122) 1.3.6.1.2.1.2.2.1.21.1 (ifOutQLen.1) 0
123) 1.3.6.1.2.1.2.2.1.21.2 (ifOutQLen.2) 0
124) 1.3.6.1.2.1.2.2.1.21.3 (ifOutQLen.3) 0
125) 1.3.6.1.2.1.2.2.1.21.4 (ifOutQLen.4) 0
126) 1.3.6.1.2.1.2.2.1.21.5 (ifOutQLen.5) 0
127) 1.3.6.1.2.1.2.2.1.21.6 (ifOutQLen.6) 0
***** SNMP WALK REQUEST FINISHED *****

[jguse]

Hello Ray,

The difference between my trace and your MIB Browse, is that the MIB Browse is individually requesting the values with SNMP Get / GetNext requests, while the device synchronization uses GetBulkRequest. It seems the iLO SNMP Agent has issues with the GetBulkRequest that iMC sends it for the interface indexes. The bulk request is a feature of SNMP v2c/v3 that makes it more efficient for the NMS to get a lot of values at once, as iMC needs to do here.

I've seem similar issues on switches which have too many interfaces with values to return for a GetBulkRequest, and that would cause the SNMP Response packet to be above 1500 bytes large (or however large the device SNMP Agent packet size is by default). That would result in the device returning an SNMP 'tooBig' error message.

However the Response packet I was looking at in the capture does not contain any error. I would expect a tooBig error if the issue above applied here, or at least a 'genErr' (general error if all else does not apply). Yet it also doesn't contain any OIDs/values that were requested by iMC. I also see other GetBulkRequests in the capture from iMC that iLO responds to correctly - so I don't think it's a general lack of GetBulk support.

That all points to a potential bug on the iLO side. Can you run a packet trace on iMC during the device add or synchronization of your iLO and have a look if you see the same packets I mentioned previously?

[RPapaux]

Hello,

Just to give you an update I have open a case to HPE <CASE:5355772375>.

After very painful explanation of the case through extensive mail exchange, I finally got an answer form the L2 support.

Needless to say that this answer is confusing and does not help at all. If I read correctly between the lines, they say that ILO SNMP is certified to work in conjonction with HPE SIM and OneView. No word about "SnmpBulkWalk" NOT being fully implemented or things like that. They seem to suggest to load the HPE SIM MIB, but I don't think it will help. (see case extract below)

<----------------------------------------------------------------------------------------------------------------------------------

Please note that we checked further regarding the issue . after collaboration with L2 team please find below their feedback :

As far as I know Snmpbulkwalk needs OID to be present and if there is no explicit OID then it uses MIBs. In regards to server monitoring we do not work with individual OIDs and are using MIBs, our HPE MIBs are dedicated to usage with HPE SIM.
We officially support snmp communication with the officially supported HPE applications for server monitoring which are HPE SIM and HPE Oneview. In the iLo quickspecs only HPE SIM and Oneview are mentioned:
https://h20195.www2.hpe.com/v2/getdocument.aspx?docname=c04154343#

Some customers try to use the HPE SIM MIBs for other application of their choice for snmp monitoring or collecting information via snmp and sometimes it works with these MIBs. This is not official neither supported but sometimes it works. Here is a link to HPE SIM MIB list: https://support.hpe.com/hpesc/public/docDisplay?docId=emr_na-c04272529

kindly note that your request is not a break and fix support and out of the support scope for the product. 
If theoretically what you want is achievable you need an HPE consultant.

---------------------------------------------------------------------------------------------------------------------------------------->

At least I give it a try.

Best

Ray

[jguse]

Hello Ray,

Sorry to hear that. I am checking this internally and will try to get some clarification for you.

Thinking about the issue here some more, I realize there may be a simple workaround. Could you please try adding the iLO with SNMPv1 in IMC instead? v1 does not implement getBulkRequest packets at all, so the interface information will be queried with individual GetRequests. That would allow you to avoid hitting the issue with getBulkRequest.

I've also found an interesting post on the server forums that explains some things about how the iLO v1-v3 SNMP 'agent' is working: "iLO, iLO2 and iLO3 don't have a complete SNMP stack. They only have a SNMP Passthrough functionality, i.e. SNMP queries sent to iLO are passed to whatever SNMP agent is running on the main server OS." 

https://community.hpe.com/t5/Server-Management-Remote-Server/ILO-SNMP-Configuration-and-MIB-information/td-p/5943551#.YLiK2agzZaQ

No word on whether this is still the same in iLO v4-v5, but it could provide an explanation as to why interfaces may not be retrieved via the agent. It actually depends on the agent running on the server OS.

[RPapaux]

Thank you for the further investigation.

I'll try with SNMPv1, which seems to be a good idea., at the next opportunity and give you feedback.

I have also look if there is a kind "PassThrough" flag somewhere in oder to go back to previous (ILO1-3) behavior (but I doubt)..

The server is running VMware/ESX version 7, and has its SNMP agent enable.. in IMC I see all its ESX interfaces, but I cannot get them mapped onto the edge switches in the topology, beside that I have enable CDP on both vSwitches and Edge switches. 

At the end what I want to achieve is a IMCdynamic topoloy where I can see what server's card is connected to what switch port.

Regards

Ray

[RPapaux]

Hello,

I have configured the ILO with SNMPv1 and it is now able to retreive all the interfaces correctly thanks.

However I'm still unable to draw (automatically) the link between the ILO or the Proliant interfaces and the switch on the network map.

CDP is ENABLE on the ESX side and I see the CDP information in my ComWare switch, but they are not drawn on the topology map. Just beware that VMware uses CDP for link discovery and advertizement (LLDP is available at the big expense of an enterprise license !?!)

As IMC is using LLDP per default, it there something more to do to enable CDP on the IMC side?

Thanks for advise

Ray

[jguse]

Hello,

You might have enabled CDP on ESX side, but iMC reads that via the CDP MIB using SNMP - and I'm not sure if the SNMP Agent on ESX will actually provide that information?

No additional configuration is needed on the iMC side for CDP. It reads both LLDP and CDP information (and more, such as ARP table and MAC addresses) automatically from your devices to detect the links. However it's generally recommended to use only one of LLDP or CDP protocols across your network, as inconsistencies between LLDP and CDP data on the same device can result in issues drawing links.

Have you considered using the VRM component that comes with the iMC platform to integrate your hypervisors? See https://support.hpe.com/hpesc/public/docDisplay?docId=c05228262&docLocale=en_US

That should help to determine where the vmnics are connected, as you get a detailed virtual resource topology, though it won't show you the iLO interface. In the end, though it isn't ideal, you could always add the link to iLO manually on the topology.