dead gateway detection and firewalls

[Ben Dehner]

We have a Netscreen firewall which does not respond to ICMP packets. While this firewall is not my default router, it is on the same network as the default router and my (HPUX 11.0) system. Dead gateway detection listed this as a 'dead' gateway and I could not get packets through the firewall. I have disabled dead gateway detection, and I have also cleared and re-added my default route. I have even reset the network interface with lanadmin, but I still cannot get past the firewall. We have reviewed the firewall configuration, and it seems to be correct.

'ndd -get /dev/ip ip_ire_status' does not list any dead gateways. I strongly suspect there is still some residual table entry somewhere that prevents going through the firewall. Is there something else I need to do -- short of rebooting the system -- to route through the gateway?

[Michael Steele_2]

I think your confusing an issue of the server LAN downing itself and getting through a DMZ firewall that is overseen by the network administrator. For without his assistance in indicating what ports are open to you will have to try each one like a hacker.

Use 'tcpdump' to see what traffic is being let through the router:

tcpdump udp/tcp/icmp
tcpdump router_ip
etc.

http://hpux.cict.fr/hppd/hpux/Networking/Admin/tcpdump-3.6.2/

[Kevin Wright]

Do you have a static route through the Netscreen? Otherwise, dead gateway detection would never come into play. can you ping your default route? do a traceroute to ensure your using the default route to get to other networks.
Disable dead gateway detection, re-add your static route throught the Netscreen (if you have one)

[Steven E. Protter]

This is probably not an HP-UX admin issue. If the firewall is unreliable, or not configured to let traffic through from that IP address, the behavior you note is possible, if not probable.

The fix is to deal with netscreen.

Dead gateway detection is a new aspect of HP-UX adminsitration to me. How do you detect a dead gateway? netstat ???

SEP

SEP

[Michael Steele_2]

Dead gateway is often seen with a DMZ router when the DMZ router is configured to not respond or answer icmp traffic, i.e., the 3 second ping automatically put out in HP-UX.

Consequently the server's LAN goes into a down state.

Refer to 'ip_ire_gw_probe' in ndd.

Here's a thread:

http://bizforums.itrc.hp.com/cm/QuestionAnswer/0,,0xcd529c196a4bd71190080090279cd0f9,00.html

[Ben Dehner]

Thanks for the replies, but just to clarify a few things:

1) I have reviewed the firewall configuration with the network engineer. The firewall rules do allow for my Unix host to have access outside into our DMZ.

2) I have also reviewed the threads on ndd and dead gateways, and I have disabled dead gateway detection. I don't need a specific route through the firewall, since my default router should handle it. However, since the firewall and the default router are on the same network as my system, arp resolution has figured out the firewall is a gateway, and dead gateway detection said that it was dead.

[Michael Steele_2]

I believe you may have to reboot after the lan goes down. This is what I recollect anyway. But for clarification:

Regarding "... arp resolution has figured out the firewall is a gateway...", then there is no connection issue at layer two?

linkloop MAC (* router MAC and node on other side? *)

Regarding "...We have a Netscreen firewall which does not respond to ICMP packets..."

So you are not able to test for layer three connectivity with 'ping', which is icmp based?

ping ip (* router ip - ip on other side *)

Regarding "...and I have disabled dead gateway...", so 'ip_ire_gw_probe' is set to 0?

ndd -get /dev/ip ip_ire_gw_probe

Then you will have to test for layer four connectivity using 'telnet socket':

telnet ip port#

for example

telnet 192.1.1.13 21
Connected to yourserver.
Escape character is '^]'.

Or use 'lsof' or 'netstat -an' to observe the sockets.

[Ben Dehner]

Thanks for the quick reply.

1) there is no layer 2 problem that I know of. I can get an arp resolution on the firewall address.

2) linkloop times out when trying to test the firewall interface.

3) The firewall will route ICMP packets, but its interface will not respond to them. Specifically, I can ping an address in our DMZ (past the firewall) from any Windows system here, or from any HP-UX system on a different network than the firewall. There is a firewall rule that specifically allows any internal address to ping a DMZ address.

4) ip_ire_gw_probe is set to 0. Also, 'ndd -get /dev/ip ip_ire_status | grep DEAD' now shows nothing as a dead gateway.

5) We have also set up a rule for port 80 allowing the HPUX system into the DMZ, but 'telnet

80' just times out.

As an additional note, I cannot test by enabling ICMP on the firewall interface. It simply does not allow this option.

[Ron Kinner]

icmp redirect from the default gateway is telling your box to use the netscreen which is how your box figured out that the netscreen was a gateway. It figured out that it was dead because it didn't reply to a ping. So that's how it got killed.

Your network guy could probably tell the router not to do icmp redirect which would mask the problem but cause a bit more traffic on the local LAN.

From other posts I believe there was a patch on this option which is needed before the 0 really takes effect. PHNE 26125 and PHNE 28538 are the latest ndd and ARPA patches so I would try them.

Another thing you might want to check:

ip_pmtu_strategy

It is usually set to 2 or so our help says (we do not have any of the above patches on our system) tho on our system it is set to 1 and we didn't change it. Try setting it to 0 to see if there is a problem with the mtu discovery. Firewalls are know for screwing this up.

Ron

[Michael Steele_2]

REGARDING "...1) there is no layer 2 problem that I know of. I can get an arp resolution on the firewall address. -AND- 2) linkloop times out when trying to test the firewall interface....."

HOW CAN LINKLOOP FAIL BUT ARP WORK? YOU CAN LINKLOOP THE FIRST SWITCH OUT OF THE SERVER?

(* ARP IS A TABLE OF STORED IP'S AND MAC'S. SO THE PROBLEM COULD HAVE OCCURRED AFTER THE ENTRY INTO THE TABLE WAS MADE. *)

MATCHED SPEED AND DUPLEX OF SERVER NIC AND SWITCH NIC?

lanadmin -x NMID/PPA (* current speed and duplex *)
lanadmin -X 100FD (* 100 Mbps @ Full duplex *)
Also use 100HD, 10FD, and 10HD. Match the NIC to the switch.

##############################################

REGARDING "...3) ...I can ping an address in our DMZ (past the firewall) from any Windows system here, or from any HP-UX system on a different network than the firewall...."

WHAT IS THE CURRENT LAN STATUS?

lanscan or lanadmin (* status up/down? *)

ifconfig lan4 unplumb
ifconfig lan4 plumb
ifconfig lan4 up

##############################################

4) ip_ire_gw_probe is set to 0. Also, 'ndd -get /dev/ip ip_ire_status | grep DEAD' now shows nothing as a dead gateway.

##############################################

YOU MAY HAVE TO REBOOT. MAKE SURE YOUR 'ndd.conf' UPDATES ARE IN.

TRANSPORT_NAME[2]=icmp
NDD_NAME[2]=ip_ire_gw_probe
NDD_VALUE[2]=0

##############################################

REGARDING "...5) "...but 'telnet

80' just times out...."

THIS HAS TO WORK.

[Ben Dehner]

Maybe a reboot will fix the problem. However, I will install the latest patches for ndd and the lan drivers this weekend, which will neccessitate a reboot anyway, and then see where we are. I have already set the nddconf to disable dead gateway connection on startup.

[rick jones]

A couple things for various folks:

Dead gateway detection does _NOT_ down an entire LAN. It marks as dead a _route_ which is a different beast entirely. Now, when this is the default route it can look like the entire LAN interface is dead, but the LAN interface is still just fine and dandy. The joys of layering and all that...

Linkloop does its work with either XID or TEST 802.mumble frames. Not all equipment out there will respond to those frames (though I would encourage folks to ask their supliers to add that to their kit since linkloop is a very useful (IMO) thing.

IMO a firewall that will not respond to ICMP Echo Requests (aka pings) at least on its "inside" side is a broken pile of otherwise unmanageable junk. Regardless of one's opinion of the dead gateway detection implementation in HP-UX 11.

[Ron Kinner]

linkloop MAC will also fail if the PC has more than one NIC and the target is not on lan0. You have to tell it which ppa to use. Think it's the -i option but check the man to be sure.

Isn't the transport IP in nddconf instead of ICMP? It's true it uses a ping which is ICMP but it effects IP routing and you set it in ndd with ndd -set /dev/ip.

Ron

[Ben Dehner]

Thanks to all for replying, but here's the current status:

I have disabled dead gateway detection (DGD).
I have modified nddconf to keep DGD disabled.

I have added the latest patches for LAN, ARPA and ndd to the system. This required reboot. (This is a production system, so I cannot arbitrarily reboot.)

On reboot DGD was properly disabled:
ndd -get /dev/ip ip_ire_gw_probe = 0

I still cannot communicate past our firewall. Also, if our firewall was mis-configured so that I could not get past, my attempts to do so would be logged as a security violation. There are no such violations being logged.

Something seems to have changes, but using linkloop I get:
error: expected primitive 0x30, got DL_ERROR_ACK
dl_error_primitive = 0x2d
dl_error = 0x04
dl_unix_error = 57

Help?
Thanks.

[rick jones]

Grep and a bit of knowledge about Unix include files and errno are your friends:

The dl_unix_error is the "errno"

$ grep 57 /usr/include/sys/errno.h
#define ENOLINK 57 /* the link has been severed */
#define ECONFIG 157 /* Configured kernle resource exhausted */

this suggests (apart from the owner of errno.h not being able to spell well...) that the link-layer is not functioning - perhaps no or a bad cable or something, or a failure on the NIC itself that precludes the link being brought-up.

Dead gateway detection would not be an issue for link-loop. Dead gateway detection will only "shut-off" IP traffic and above. What precludes linkloop from working is either the link not being good, or the remote MAC not responding to XID/test frames. In this case it looks like the former - that the link itself is down. When the remote is down or does not respond to the linkloop frame you get the timeout message.

[Ben Dehner]

The link CAN'T be down, because that interface on the HPUX box is the primary interface for our applications, and the firewall is our company's link to the internet. Every other bleedin' system (including HPUX on a different network) can see past the firewall. But something else must be happening at this layer.

[John Dvorchak]

I am really curious about your statement that the default router is the router not the firewall, and the router should take care of it? The other hint is everything else works that is on another subnet. The third hint is that there were no security violations logged. Indicating to me that the packets never got to the firewall. I don't think your "router is handling it" for you.

I think I would break this problem down and first of all put the default gateway as the firewall on the HPUX. I'll bet it starts going to the internet through the firewall. Then if need be add either another default router pointing to the router. Or use static routes for internal addresses pointing to the router.

Good luck,

[Kevin Wright]

As I mentioned days ago now, are you positive your are getting to the firewall?? I would enable icmp on the firewall from the inside and ping it. Then do a traceroute from your HP box, through the FW to the DMZ. Ensure your going through the default route, then to the FW, to the DMZ and back again.

do you have the FW in your arp cache?

and BTW, icmp redirect or routing protocols like rip or gated advertise and modify routing tables, not arp.

[Ben Dehner]

Thanks for the replies. Just to make things a little clearer, here is what my IP scheme looks like:

Unix host: 162.15.228.44
Default gateway ('router'): 162.15.228.1
Firewall: 162.15.228.3
DMZ network: 185.143.18.0 x 255.255.255.0

I cannot enable icmp on the firewall interface; this is a default configuration and there is no way to change it.

We have multiple internal networks that I can reach via the router. We have a complicated internal network topology, and in general I am not worried about getting through the firewall to the internet. I am trying to enable communication to the DMZ network, which must go through the firewall. (Technically, the firewall is also a router, but in this context I use 'router' to refer to the gateway device at 228.1.)

If I just 'ping' into the DMZ, then I see a route appear on my unix system -- probably an ICMP redirect from the router -- that goes through the firewall. I also see the firewall MAC in my Unix arp cache. But the ping does not succeed, and there is no log on the firewall that it actually received any packets. If I manually add a static route to the DMZ, either using the firewall or the router, same thing. I am not running RIP or gated, or any other dynamic routing protocols, on the Unix host.

I can try adding another default route, but this is a production system so that will have to wait. I don't want to make changes like that during working hours.

[rick jones]

If as you say ICMP is not enabled on your firewall, then it does not seem surprising that attempts to ping the firewall or DMZ would be unsuccessful. A "ping" is more formally known as an ICMP Echo Request, triggering an ICMP Echo Reply. And if the firewall is filtering ICMP traffic...

That you see the firewall MAC in your ARP tables suggests that there is indeed connectivity at the link-level to the firewall, despite the errors you encountered with linkloop.

That things did not improve with a dedicated static route suggests that trying to add a second default route would not work. And, HP-UX is not setup to have multiple default routes out the same interface. There are ways to get most of the effect of per-interface default routes by setting ip_strong_es_model, but that is not quite the same thing.

[Michael Steele_2]

YOU HAVE TO 'TELNET' INTO THE PORT OPENED BY THE FIREWALL TO TEST CONNECTIVITY.

telnet ip port#

for example

telnet 192.1.1.13 53
Connected to yourserver.
Escape character is '^]'.

IN THIS EXAMPLE 53 IS THE PORT NUMBER!

[Ron Kinner]

I'm not clear on your exact architecture. Does the firewall have three interfaces (Internet, DMZ, internal) or two (Internet + DMZ, internal).

Don't forget the other end needs a route back to you. Since you are able to get out to the internet the firewall knows about you OK but do the DMZ boxes? Normally the firewall would NAT your IP to something in the Internet range before sending it out. If it did that then the DMZ boxes should be able to get back to you. (subject to the firewall letting them which is also sometimes a problem.) However, could it be that the Firewall realizes you want to go just to the DMZ so it lets you through without NAT so that your return address is your real address which the DMZ box might not know can be reached via the firewall?

Ask the DMZ box what its
netstat -rn
looks like and see if there is a route back to your stuff or if it just has the default route and no idea where your address is? If there is no route try adding a route via the firewall and see if it suddenly starts working.

The firewall could also be doing policy routing which said that anything from the DMZ has to go out.

Ron

PS I think it's past time to connect up a sniffer and see what is really happening. If you don't have one you can get free software which will let a PC act as one. tcpdump for unix / snort or windump for Windows. You will need to tell your switch that you want to monitor the traffic on the port connected to your HPUX box since otherwise you will only see broadcasts. I do not trust tcpdump running on the target to report what is really happening.

[Ben Dehner]

Never did quite resolve thie issue. However, we have since change our network configuration so the firewall is beyond the default router, so there is no ICMP redirects or dead gateway detection, and I don't worry about any routing outside of my LAN interface. Just throw packets to the default router and let figure out what its suppose to do. My favorite way to do host-based routing -- not at all.