ping to external IP address - IP address is changed to my firewall IP

[Jim Scott_2]

hpux 11.0

If I issue a ping to any IP outside of the lan, the IP address is changed to the address of my firewall. I see this from the firewall logs as well as a sniffer. Routing table looks good, nothing funny about it. The REALLY strange thing is that after a reboot pings will get out, then after some amount of time it is always changed to the firewall address. pings to any internal address work fine. This behavior started a week ago, I am not aware of any change that took place at the time.

[Tim D Fulford]

Hi

do netstat -in & ifconfig for all configured interfaces.

I suspect that you may be suffering from the problem/feature I outlined at the bottom of the following thread

http://forums.itrc.hp.com/cm/QuestionAnswer/1,,0x2cba3a7b3682d611abdb0090277a778c,00.html

Tim

[Jim Scott_2]

I only have one interface configured at the moment. Below are the vitals.

# netstat -rn
Routing tables
Dest/Netmask Gateway Flags Refs Use Interface Pmtu
127.0.0.1 127.0.0.1 UH 0 1242 lo0 4136
172.21.192.132 172.21.192.132 UH 0 1616 lan1 4136
172.21.0.0 172.21.192.132 U 2 0 lan1 1500
127.0.0.0 127.0.0.1 U 0 0 lo0 4136
default 172.21.191.121 UG 0 0 lan1 1500
# netstat -in
Name Mtu Network Address Ipkts Opkts
lan1 1500 172.21.0.0 172.21.192.132 1809255 3183336
lo0 4136 127.0.0.0 127.0.0.1 1242 1242

# ifconfig lan1
lan1: flags=843
inet 172.21.192.132 netmask ffff0000 broadcast 172.21.255.255

[Tim D Fulford]

Must say I'm stumped, I suspected you had two cards & the card logic was contary to the routing logic.

"ip_strong_es_model" should do nothing as you only have one card. ditto "ip_forwarding"...

I'm going to bow out now to think....

Tim

[Tim D Fulford]

Jim, sorry to say I've thought & really no no practicle suggestions. I SUSPECT it is the firewall that is doing this.

Sorry

Tim

[Paula J Frazer-Campbell]

Hi

This sounds like either the firewall or a router gathering information on route and translating the external ip addres to the internal firewall address.

After a reboot how many seperate ping attempts does it take before the firewall address is presented?

It is not good security to have ping/traceroure enabled through your firewall, just enable the services that are required.

If you require ping/traceroute then turn it on, use it and then turn it off.


HTH

Paula

[U.SivaKumar_2]

Hi,
Can you ping to your firewall from internal machines ?.
If you have disabled ping in the firewall. Then
give this command in your HP-UX server.
#ndd -set /dev/ip ip_ire_gw_probe 0
After this if the problem persists , check the
NAT rules in firewall and also anti-spoofing rules in the firewall. Which firewall are you using ?.
I suggest not to open ICMP in firewall for ping to external hosts in internet. Dangerous !!

regards,
U.SivaKumar

regards

[Jim Scott_2]

I agree, I don't allow ICMP traffic to the firewall, it's droppped and logged. Only the usual suspects are allowed through. I've never counted the pings that make it happen. My feeling is it's not happening external to the box, like it being NAT'd somehow. The sniffer logs show its changed leaving the card. To reiterate its not just a ping thats changed. Any external traffic is altered, such as telnet or http. Any thing internal works just dandy.

[Christopher Caldwell]

Without knowing more detail, I'd say you have a NAT/PAT issue with your firewall.

Here's some things you can check:

-see if your firewall is running at or near capacity in terms of translations (a number of vendors restrict the number of connections in software - in other cases there aren't enough translations in terms of IP addresses to perform NAT so the firewall reverts to PAT (overloading) which may exhibit some of the symptoms you are having).

-look to see if any network device (VPN concentrator, switch, router) has been added to the network - if so, make sure this device isn't proxy-arping for your host

-check the arp table on your host and make sure it looks reasonable

[Jim Scott_2]

I already tried ndd, did it again just to make sure :) Same behavior. NAT and spoofing rules haven't changed for this system and are the same for other systems that don't show the problem.

The firewall is not running near capacity. No new devices added to the network that I can discover. The arp table looks good and is similar to systems that don't show the problem.

I went back and looked at the sniffer data. I allowed ICMP to leave the firewall just to watch what happens. I began to ping google, all looked good. Then after maybe 20 or so frames it changed to a firewall address.

For the good frames, each frame was 98 bytes long, with 56 bytes of data, starting with =.X and ending with 01234567. The sequence number increased with each frame and were in the very low digits. Like 10, 11, 12...

When the frames changed the destination was the firewall. 78 bytes long with 36 bytes of data. The data was all hex 0. The sequence number stayed at 22136 for each frame sent after that.

Wierd huh?

[Paula J Frazer-Campbell]

Jim

Ideas

1. Try ping/trace from another server.

2. What other devices sit in the path to firwall.

3. Try ping/trace from Windoze pc.

4. Try ping -o for route option.

5. Monitor firewall as a ping starts.

6. Do you have Windows machines in your network and if so up to date Anti virus software in use?

7. If a ping within the lan is ok and remains so them fault is at firewall.

8. Have you tried a reboot of firewall.

9. At the same time as 8 reboot all devices associated with external connectivity via firewall.


Just a few idea.

Paula

[Christopher Caldwell]

can you give the make/model of the firewall?

[harry d brown jr]

Your IP address, going out of the firewall to the internet should be changed to a registered IP, unless you have your own domain, but you are right that it shouldn't be changing the IP address of the device you are attempting to ping.

What kind of firewall are we talking about here?

live free or die
harry

[sven verhaegen]

Hello

What you see here could be explained by something called IP Masking or Substituting , on CISCO it is calle NAT (Network Address Translation) , this is a procedure in which your IP address is substituted with a general IP , it has 2 functions , the first to allow you to use non-legal ip ranges within your intranet but still maintain access to the outside internet trough valid internet IP range addresses , second to avoid some problems with attacks on your IP , substituting your IP will redirect traffic to the owner of the substitute IP itself being the firewall or router doing the Masking , both routers and firewall are able to do this , however the strange thing in your description is thatthe behaviour changes after a while , normally with this kind of aliassing the IP changes automatically at each packet handeled , where did you trace the data with the sniffer in the intranet before the firewaal , before the relaying router or behind it ?? that might give more data to invest on...

[Tom Danzig]

Sound like the router is using NAT. Expected behavior for a NAT'ed router.

[Jim Scott_2]

The firewall is a Nokia running checkpoint. Ok, now for an even stranger thing. In looking at the firewall logs I see that system is attemping to access the internet every 3 minutes?? I can find no process that should be doing this, but since everything is being changed to the address of the firewall its being logged and dropped. Sounds like I've been hacked. Any clue how to track down what process is doing the deed. ps -ef run at the time shows nothing out of the ordinary, no cron or at jobs are running. Darn curious if you ask me.

[Bill Vitaniemi]

This problem happens to be my first taste of hp unix. (I'm totally a Solaris type.) Firewall problem? I think not! I am running Checkpoint on Solaris and I can't see that this is in anyway a firewall issue. First off, I don't see this extremely odd behavior on any other machine, only the hp boxes. Second of all, my policy will allow icmp across the firewall to the dmz where I am pinging. If I snoop the firewall interface, I can see the hp box trying to ping the firewall and not the ip address I told it to ping. Hmmm....

You can temporarily fix the problem, at least I can, without rebooting. Delete the default route and add it back in. When I do this, ping will work for about 150 pings and then dies. Then I start seeing pings of the firewall again. Sounds like this is a hp routing/networking issue.

[Mark Fenton]

Jim,

this is pretty strange. Tried to reproduce this behavior myself, but was unsuccessful -- all pings resolutely fired off in succession with appropriate sequence numbering and etc (let it run for ~10 minutes, generating 500+ pings...).

As far as the firewall somehow effecting this behavior, I can't see that as a high probability. It can only rewrite packets after it's received them, if they're being received with the firewall address on them already, that's something else entirely. On the other hand, I know of no circumstances where HP-UX will dynamically re-address a packet, nor where it would autonomously start forging packets (which is what that looks like, especially with the sequence number not incrementing, and the contents switching to nulls).

Interesting problem.

Do other protocols suffer from the same fate (FTP, Telnet, etc) or just ICMP?

Mark

[sven verhaegen]

I think I can answer this one easily , you have not been hacked , you simple ran into one of the neat features of hp-ux 11.x called dead-gateway detect , from 11.x on the system will look for gateways in it's routing table and after a set interval (mostly 3-5 minutes)will start pinging them , then if it gets no reply (as firewalls stop ping) they simply deactivate that route and keep trying to ping it at set intervals to see if it doesn't reactivate .... as this is a changable kernel parameter you can disable this function using ndd....

ndd -set /dev/ip ip_ire_gw_probe 0

will stop this behaviour , make sure to also add it into the /etc/rc.config.d/nddconf file because at reboot this setting will be lost

[Jim Scott_2]

I had tried using ndd to turn of gateway detection, that was one of the very first things I did. I was assuming it worked since an ndd -get showed it was set to zero. But I still get the ping to the firewall every 3 minutes. After the ndd didn't seem to fix the problem I went down this other rabbit hole.

I agree that dead gateway detection seems to be the most likely cause. It would also sort of explain how any ping is morphed into a ping to the firewall. If it thinks the gateway is down it first pings the gateway/firewall before sending out the real ping. And why pings to internal addresses still work.

So I'm left with why ndd isn't able to turn off the behavior.

Below is what I've done numerous times:

ndd -set /dev/ip ip_ire_gw_probe 0

followed by:

# ndd -get /dev/ip ip_ire_gw_probe

0

[sven verhaegen]

that is indeed strange , once the parameter is set no more pings should be sent , however it still could be one of the gateways is set to dead , you can check this with the option :

ndd -get /dev/ip ip_ire_status


it'll tell you exactly if any gateway is dead and what gateway it is , perhaps a gateway marked dead still pings even if gw-probe is deactivated , scanning trough some cases I found out this gateway feature wasn't always changable/de-activatable , depends a little bit on the version of the OS and the ARPA Transport Stack you run , you should have at least PHNE_17662 Arpa Transport Cummulative patch installed , the latest of that series is PHNE_26771 so if you have no ARPA transport patching or very old it still could be that changing the parameter could not work.. then the message is a classic within HP support answers : "please could you patch the system"

[Jim Scott_2]

I'm running PHNE_21767. Once I was convinced that gateway detection was the answer I found 2 ways around the problem. First make sure ip_ire_gw_probe is set to 0. Then enable ICMP traffic to the firewall long enough to allow the IRE_GATEWAY flag to change to not DEAD.

ndd -get /dev/ip ip_ire_status | grep -e IRE_GATEWAY -e flag

The other way was to add the paramter to /etc/rc.config.d/nndconf
and reboot.

The piece of the puzzle that really through me off was the ping packet that was sent to verify if the gateway was up or down since the sequence number never changed and it was sent before the ping I was sending out to the outside internet. Making it appear my packet was changed.

Why is it that when a feature is added that changes expected behavior since the time of dirt, it is set to on, and not off to allow the user control?

Thanks to those that responded

[George Abraham_1]

hai

I faced a similar problem.. The trouble was with the NAT setting in the Router.. See the Router logs to see if u can make out something...

Just to check.. have you go the latest Hardware extension patches loaded?

keep smiling
george