default route issue

[kemo]

Hello

HPUX server stopped serving other networks because of an issue in the default gateway

# netstat -rn
Routing tables
Destination Gateway Flags Refs Interface Pmtu
127.0.0.1 127.0.0.1 UH 2130706433 lo0 4136
108.97.110.48 0.0.0.168 U 1 16384
10.10.51.1 0.0.0.5 U 4294967295 2130706432
0.0.0.199 * U 4 4294967295
0.0.0.5 * U 0 168440830
default * U 9925576 16
#

attached netconf file.

uname -a
HP-UX HNAME B.11.11 U 9000/800 4217070896 unlimited-user license

thanks

[Stephan._1]

Hi,
you have 2 default routes in the netconf ...

ROUTE_DESTINATION[0]="default"
ROUTE_MASK[0]=""
ROUTE_GATEWAY[0]="10.10.51.254"
ROUTE_COUNT[0]="1"
ROUTE_ARGS[0]=""

....

ROUTE_DESTINATION[1]=default
ROUTE_GATEWAY[1]=10.10.51.245
ROUTE_COUNT[1]=1

Delete the second one and run '/sbin/init.d/net start'

hth
Stephan

[kemo]

yes you are right, i was trying to add 10.10.51.254 using sam on the console, but it refused saying that this route is already used, so i was trying to change to any "10.10.51.245" so that i can change it back againg to the default. but no luck.
anyway, i have removed all the entries except for 10.10.51.254 and restarted the box " It is development".
it starts serving other networks for 5 minutes and STOP again.


# netstat -rn
Routing tables
Destination Gateway Flags Refs Interface Pmtu
127.0.0.1 127.0.0.1 UH 2130706433 lo0 4136
108.97.110.48 0.0.0.118 U 1 à ºfwúà à º^t 16384
8.10.34.0 0.0.0.5 U 4294967295 2130706432
0.0.0.149 204.186.127.161 U 4 wúà à º^t 4294967295
default 204.186.94.116 U 9864560 à à à ºfXwúà à º^t 16


attached the new netconf after reboot.

[Bob_Vance]

A little more info might help.
What do you mean by

"serving other networks"

Show/tell us something about what fails.
bv

[Bill Costigan]

The routes listed in netstat -rn do not match what is in netconf. Where are these other routes coming from?

Looks like you are getting bogus routes added after the server starts.

[kemo]

Hello

i haven't done any changes in the server, and also all these routs are junk we don't have such networks and we never add more routes in any other file.

serving other networks means, the server can serve only who ever in its subnet. 10.10.4.x only, but because the default gateway got corrupted, the server cannot connect to any other subnet.

i have a strong feeling that this might happen in case hardware failure.

please advice!
thanks

[Bob_Vance]

Wow, this is weird!
The "U" entries are normally for direct local network/IPs

**Immediately** after a reboot,
please do
netstat -ivn
netstat -rvn
.
The -ivn will show IPs on all the LANs.

and then run the same commands after the 5 mins when crap starts showing up.
Maybe some rogue program is adding IPs to the LAN(s).


However, it does appear that perhaps memory is being clobbered.
Or at least the routing table is being clobbered.
Look at the HUGE ref counts !!!!
and the garbage chars.



Make sure that you do NOT have gated running.
(The netconf file shows that it should NOT start.
)



Is it possible that this server has been hacked??

Does
"www.thedoghaus.com."
mean anything to you :

## dig -x 204.186.94.116
;; ANSWER SECTION:
116.94.186.204.in-addr.arpa. 86400 IN PTR www.thedoghaus.com.




bv

[kemo]

i have changed the network card hoping to fix the issue, but no luck

here is the netstat output after reboot.

# netstat -rn
Routing tables
Destination Gateway Flags Refs Interface Pmtu
127.0.0.1 127.0.0.1 UH 2130706433 lo0 4136
108.97.110.49 0.0.0.153 U 1 16384
10.10.51.2 0.0.0.5 U 4294967295 168440576
0.0.0.153 * U 3 4294967295
0.0.0.5 * U 4278190080 2130706433
default * U 1818324529 152
default * U 16 201326607
# netstat -rn
Routing tables
Destination Gateway Flags Refs Interface Pmtu
127.0.0.1 127.0.0.1 UH 2130706433 lo0 4136
108.97.110.49 0.0.0.182 U 1 @O`@B@Q٠@>x@Kh@Qp@;٠@X� 16
384
0.0.3.0 0.0.0.5 U 4294967295 @>x@Kh@Qp@;٠@X� 16844
0576
0.0.0.48 64.1.73.88 U 3 @;٠@X� 4294967295
default 64.1.81.240 U 4278190080 2130706433
64.1.73.88 64.1.81.112 U 1818324529 181
64.1.81.240 * U 16 201326607
# netstat -rn
Routing tables
Destination Gateway Flags Refs Interface Pmtu
127.0.0.1 127.0.0.1 UH 2130706433 lo0 4136
108.97.110.49 0.0.0.189 U 1 16384
0.0.51.0 0.0.0.5 U 4294967295 168440576
0.0.0.144 0.4.55.144 U 3 4294967295
default 0.4.55.144 U 4278190080 2130706433
0.4.49.144 0.4.55.144 U 1818324529 @ 188
0.0.16.16 0.4.55.144 U 16 201326607
# netstat -rn
Routing tables
Destination Gateway Flags Refs Interface Pmtu
127.0.0.1 127.0.0.1 UH 2130706433 lo0 4136
108.97.110.49 0.0.0.195 U 1 16384
10.10.51.2 0.0.0.5 U 4294967295 168440576
0.0.0.195 * U 3 4294967295
0.0.0.5 * U 4278190080 2130706433
default * U 1818324529 194
default * U 16 201326607


changing rapidly, so weird.

[kemo]

Any Clue?

[Stephan._1]

Hi (again),
i had once a system where hosts and netconf got replaced every time the system has been booted.

It was as well a test system and the solution after nearly a day of search was the hammer new install.

Did you check if in single user mode the routes get screwed up as well?

Did you check profiles, start scripts, cron, ... for something strange?

After that i would go for the hammer solution.

Best,
Stephan

[kemo]

As Usual, when you stuck inside an issue, you try to involve other parties with you, Netwrok Team :(
they put access list on the VLAN. when they removed the list. it worked.

i put them as the last resort, and the problem usually comes from their sides.

thanks everybody

[Bob_Vance]

What does the nestat -rvn show now?

I don't see how an ACL could cause bad entries to appear in the routing table.

These entries were not dynamic entries added because of ICMP redirects; they were trash.

Doesn't make sense to me.

bv

[kemo]

still showing rubbish, but at least i have got my problem fixed.

# netstat -rvn
Routing tables
Dest/Netmask Gateway Flags Refs Interface Pmtu
127.0.0.1/255.255.255.255 127.0.0.1 UH 2130706433 lo0 4136
108.97.110.49/108.97.110.49 0.0.25.191 U 2 16384
10.10.51.1/0.0.0.0 0.0.0.5 U 4294967295 2130706432
0.0.25.222/0.0.0.0 * U 4 4294967295
0.0.0.5/0.0.0.0 * U 9864560 16

[Bob_Vance]

That makes me feel better ;>)

I don't want to beat a dead horse, but I think the critter is still breathing, or maybe a zombie or something.

I don't understand why originally it worked for the 5 minutes.

Also, I still would like to see
netstat -ivn

and also both
netstat -ivn
netstat -rvn
IMMEDIATELY after a reboot.

bv

[kemo]

i have already announced to the dev team that the server is back. i am sorry i won't be able to reboot, but below is the current output

# netstat -inv
Name Mtu Network Address Ipkts Ierrs Opkts Oerrs Coll
lan1 1500 10.10.51.0 10.10.51.1 16 0 3 0 0 2147483647 none 255.0.0.0 849 0 0 0 0
#
=============
# netstat -rvn
Routing tables
Dest/Netmask Gateway Flags Refs Interface Pmtu
127.0.0.1/255.255.255.255 127.0.0.1 UH 2130706433 lo0 4136
108.97.110.49/108.97.110.49 0.1.13.179 U 2 16384
0.0.0.1/0.0.0.1 0.0.0.5 U 4294967295 2130706432
default/0.0.9.64 * U 9864560 16
=========

thanks

[Bob_Vance]

There is something very wrong with your system.
That output is totally trash.
Your output should be like the attached file.

I've never seen anything like this, except when something is stomping around on memory, but that usually portends more serious problems, like crashes.

OTOH, the default route seems to be working, so maybe it's just an output problem.

I would look for the latest patches
and also call HP.

bv