iLO4 on Dl380p Gen8 - Connection issues over VPN

[disti]

Hi,

We have a brand new DL380p Gen8 with iLO4.

I configured iLO to use a static address (192.168.100.31/24) and I connected it to the local network.

If I try to ping/open web page/start console from the LAN everything is fine, and a continuous ping doesn't show lost packets.

However, if I try to connect through VPN (from 192.168.2.15/24) I experience a very strange behaviour: my server seems "dead" for long periods of time (from 15 to 30 minutes): no ping, no web access.

Then, suddenly, it starts responding for 2 to 5 minutes and then it dies again, with average ping times of 50ms.

During blackouts, I can connect via remote desktop to the server itself, using the IP assigned to the "normal" NICs.

Of course I tested my VPN connection: during blackouts every other client on the LAN is reachable from my remote pc: I can ping them and I can access services such as remote desktop, web pages, etc.

I had a look at my firewall logs (btw it's a Watchguard Firebox XTM525) but I couldn't find anything relevant.

iLO firmware version is 2.02 Sep 05 2014.

Any ideas?

Thank you!

Roberto Reale

[hpkasabbagh]

Hello,

ILO Firmeware 2.03 was available. Check and try.

Sincerly,

[Jimmy Vance]

do you have the correct defalt gateway set within iLO?

[hpkasabbagh]

Hello,

And what is result if test you network configuration on ILO: Test Settings on the Security→Directory during the blackout.

sincerly,

[disti]

Thank you for your answers!

I updated to 2.03. It didn't help.

Default gateway is correct. In fact, I can sometimes connect to iLO; I think that with a wrong default gateway it would never connect. I double checked it anyway.

Directory network tests fail beacuse I don't need active directory, so I did not configure directory parameters.

I'm really confused...

[disti]

I found out that if I reset iLO I can ping it a couple of minutes, then it goes down.

[Johan Guldmyr]

Hi, sometimes firewalls are mean to VPN traffic. Have you looked into the firewall more?

Doesn't seem like an iLO issue if there is no problem when connecting from the local network.

[disti]

I don't think this is a firewall issue, for two reasons:

1. Above all, every time I restart iLO, it does respond for a couple of minutes. I can't imagine how iLO restart could affect firewall inspection.

2. Ping works for each and every IP in the lan (included the IP assigned to Windows on the same machine), except iLO one.

Firewall is set up to log all blocked traffic, however it doesn't report anything blocked from/to the iLO IP!

Thank you!

Roberto

[Jimmy Vance]

---

@disti wrote:

I don't think this is a firewall issue, for two reasons:

1. Above all, every time I restart iLO, it does respond for a couple of minutes. I can't imagine how iLO restart could affect firewall inspection.

2. Ping works for each and every IP in the lan (included the IP assigned to Windows on the same machine), except iLO one.

Firewall is set up to log all blocked traffic, however it doesn't report anything blocked from/to the iLO IP!

Thank you!

Roberto

---

One of your other posts says that you have no issue conencting to iLO when attached to the same network. This pretty much rules out any issues with iLO itself. You also say you can get to other systems on the iLO network from the VPN.  The issue would appear to be with routing over the two networks, or maybe another system on your VPN has the same IP as your client?

[disti]

I did additional tests:

1. I tried to change IP address assigned to iLO. I have written documentation on IP addresses assignment and I was sure that the used IPs were not assigned to other devices, but I tried anyway. Same results.

2. I picked up one of our public addresses and I NATted icmp and tcp:80 to the iLO interface. Same results: whenever I restore iLO it responds for a minute or two, then it stops responding to pings from external, while internal pings are fine.

3. I enabled additional logging on the firewall. These logs show that incoming ping requests from my remote pc to iLO, both through vpn and nat, are welcome.

4. I logged in to iLO via ssh and I tried oemhp_ping command with external addresses (vpn and internet public addresses). No external address is reachable!

It seems that some sort of service that starts inside iLO, and that takes about one minute to start, interfere with external connections.

[Jimmy Vance]

---

@disti wrote:

I did additional tests:

4. I logged in to iLO via ssh and I tried oemhp_ping command with external addresses (vpn and internet public addresses). No external address is reachable! 

---

That would suggest a routing issue.  can you oemhp_ping systems on the same network is iLO?

[disti]

---

@Jimmy Vance wrote:

That would suggest a routing issue.  can you oemhp_ping systems on the same network is iLO?

---

Yes I can always ping to/from internal addesses (including default gateway).

The weird thing is that every time I reset iLO I can also ping external/VPN addresses for a minute or two!

As I said, it seems that some kind of service is started inside iLO, that introduces routing problems.

Unfortunately, I'm not aware of networking related commands in the ssh console (to trace routes, show routes, show nic configuration) to further investigate this issue!

[waaronb]

Do you have a "reactive" firewall handling the VPN traffic?  Is it seeing traffic to the ILO over some of those other ports used for things like virtual media, remote console, etc. and if there aren't rules setup for those it starts to deny traffic thinking you're attacking it?

Just a guess there.  Since it sounds like it works fine locally and it's only remotely that you lose access after a while, I don't think the ILO itself has a problem unless, as others mentioned, your default gateway is bad.

To make sure, you should double-check the default gateway setting and also the subnet mask.  If your network has VLANs or anything, make sure all of that is correct for the port the ILO is plugged into, etc.

Otherwise it sure seems like the VPN/firewall is doing something funny.  If it has any logging, you could look there and see what's happening to traffic to the ILO when it's working, and then when it's not working...see what's changing.

[disti]

Thank you waaronb,

I don't think this issue is firewall related: I can continuously ping every other IP through VPN and also NATted IPs for hours.

I have no VLANs, the LAN setup is quite easy, indeed: subnet 192.168.100.0/24, gateweay on 192.168.100.1, no VLANs, iLO NICs on 192.168.100.154 and 192.168.100.155 (I have two servers).

As I stated in my previous posts, firewall logs have been throughly analyzed and showed that all traffic to iLO IP is authorized. Outbound traffic is always authorized from anyone to anyone. For testing purposes I also created ad hoc policies to explicitly allow traffic to/from iLO on the interested server.

As a side note I'd like to stress that iLO on the other server (same subnet, same gateway, same switch) is working perfectly.

[Jimmy Vance]

---

@disti wrote:

As a side note I'd like to stress that iLO on the other server (same subnet, same gateway, same switch) is working perfectly.
 

---

In this case I'd call support and see about getting a new system board

[waaronb]

Ditto that... if another server with the same config is fine, then it's probably the system board (ILO in particular) misbehaving.

I have one server where the ILO remote console is unusable... everything is garbled.  I used to be able to see my mouse moving around and the screen was only a little mixed up, so I could kind of do things remotely.  But it's degraded to the point where it's nothing but static.

Point being, the ILO is a totally separate function on the mainboard and it can (and does) do weird things even when everything else is okay on that system.

[Mema]

Hi,
what was the solution to fix this problem?

[RossASL]

Hi Roberto,

Did you ever get to the bottom of this?

Thanks

Ross

[disti]

YES!

afrer months I found out that the switch the server was connected to had a wrong gateway address: it was set to the default 192.168.1.1 while in our network we use 192.168.100.1.

I corrected it and iLO started working!

That really confused me, because:

1. The switch itself had no routing functionality enabled.

2. Every other device on the network always worked properly (including other iLO devices)

3. The "malfunctioning" server used to work sometimes.

I still have to find an explanation to this...

BTW: the switch is a Cisco SG500.

Bye!

Roberto