ilo2 (2.15) PMTU discovery broken

[Ruben_Herold]

hi,

I was trying to connect to some ilo card via ipsec vpn. The connections are very slow, so I did an tcpdump on the gateway to the ilo card and it seems that the ilo2 ignores pmtu discovery:

20:27:49.899451 IP 192.168.101.112.443 > 192.168.56.93.50366: tcp 1460
20:27:49.899489 IP 192.168.101.65 > 192.168.101.112: ICMP 192.168.56.93 unreachable - need to frag (mtu 1430), length 556
20:27:49.922517 IP 192.168.101.112.443 > 192.168.56.93.50366: tcp 1460
20:27:49.922553 IP 192.168.101.65 > 192.168.101.112: ICMP 192.168.56.93 unreachable - need to frag (mtu 1430), length 556
20:27:49.927440 IP 192.168.56.93.50366 > 192.168.101.112.443: tcp 0
20:27:49.933959 IP 192.168.101.112.443 > 192.168.56.93.50366: tcp 1460
20:27:49.933990 IP 192.168.101.65 > 192.168.101.112: ICMP 192.168.56.93 unreachable - need to frag (mtu 1430), length 556
20:27:49.945002 IP 192.168.101.112.443 > 192.168.56.93.50366: tcp 1460
20:27:49.945032 IP 192.168.101.65 > 192.168.101.112: ICMP 192.168.56.93 unreachable - need to frag (mtu 1430), length 556
20:27:49.955965 IP 192.168.101.112.443 > 192.168.56.93.50366: tcp 1460
20:27:49.955998 IP 192.168.101.65 > 192.168.101.112: ICMP 192.168.56.93 unreachable - need to frag (mtu 1430), length 556
20:27:50.956805 IP 192.168.101.112.443 > 192.168.56.93.50366: tcp 1460
20:27:50.958898 IP 192.168.56.93.50366 > 192.168.101.112.443: tcp 0
20:27:50.959495 IP 192.168.101.112.443 > 192.168.56.93.50366: tcp 1460
20:27:50.961385 IP 192.168.56.93.50366 > 192.168.101.112.443: tcp 0
20:27:50.962036 IP 192.168.101.112.443 > 192.168.56.93.50366: tcp 1460
20:27:50.963814 IP 192.168.56.93.50366 > 192.168.101.112.443: tcp 0
20:27:50.964363 IP 192.168.101.112.443 > 192.168.56.93.50366: tcp 1460
20:27:50.966148 IP 192.168.56.93.50366 > 192.168.101.112.443: tcp 0
20:27:50.976079 IP 192.168.101.112.443 > 192.168.56.93.50366: tcp 1460

Can I manual lower the mtu on the ilo? Cause this behavior is complete broken and violates RFC 1191.

[Oscar A. Perez]

iLO2 TCP/IP stack is defined as a Host, not a Relay. The TCP/IP standard requires relaying to be Off as default.

[Ruben_Herold]

This has nothing to to with Host, Router, Relay and so on. PMTU discovery should to be activ on all devices connected to the internet, or must be reachable via links with an smaller mtu than 1500. Please take a look to:

http://en.wikipedia.org/wiki/Path_MTU_Discovery

"For IPv4 packets, Path MTU Discovery works by setting the Don't Fragment (DF) option bit in the IP headers of outgoing packets. Then, any device along the path whose MTU is smaller than the packet will drop it, and send back an Internet Control Message Protocol (ICMP) Fragmentation Needed (Type 3, Code 4) message containing its MTU, allowing the source host to reduce its Path MTU appropriately. The process is repeated until the MTU is small enough to traverse the entire path without fragmentation."

[Oscar A. Perez]

Sorry, I misread your post.

Does this issue happen with iLO2 v2.09?  

I'm asking because in version 2.12 we added few countermeasures for ICMP blind reset attacks. In the ICMP unreachable message that we get back, we now check the Internet Header + the 64 bits of Data Datagram portion. If it doesn't match with the IP message that we send, we discard the ICMP.

[Ruben_Herold]

With 2.09 it seems too run fine. I  could do some more network dumps the next day.

If you send me your e-mail Address  I can send you the pcap file with the problem in 2.15.

[Oscar A. Perez]

I sent you a PM

[Oscar A. Perez]

We found the bug that broke Path MTU discovery in 2.12 and later and we are fixing it in the next iLO2 v2.20 that will be available on the FTP on early May 2013.

[Ivan_Kuznetsov]

Hello!

It seems that PMTU discovery problem exists in iLO4 firmware too.

We have a customer far from us with two brand new DL360p Gen8 servers we should set up remotely. The servers has iLO4 Advanced I can reach through an IPsec VPN tunnel. I can ping iLO and ssh to it but can't reach the web interface. Tcp connection to https service is established but freezes quickly. I take tcpdump and see that some "big" packets (~1400 bytes) from iLO do not reach browser. We still investigate the problem but it looks similar the discussed above

iLO4 firmware version is 2.13

[Oscar A. Perez]

Please PM me a network trace if you can.

[TD-NorNet]

The problem also occurs in my setup, with ilo2 firmware 2.15.

I have captured a pcap packet trace (stored here: http://www.exp-math.uni-essen.de/~dreibh/temp/ilo.pcap) on a router connecting a local Ethernet (MTU: 1500) with a VPN tunnel (MTU: 1472). The iLO machine is 10.1.2.241, the web browser is 10.1.1.50. When the iLO machine sends a 1500-byte packet, with Don't Fragment flag set in the IP header, the router correctly responds with an ICMP Fragmentation Needed. However, the iLO machine just ignoes that and keeps sending 1500-byte-packets as retransmissions -- which are dropped. The HTTPS connection therefore hangs.

When will there be a fix for this problem? I think trying to use iLO over a VPN is not that uncommon.