network freezes

[George_206]

We have an HP9000 server hosting payment gateway sw. In general it accepts incoming web online tranactions.

The last days we face a strange behavior regarding the network. Every 3-4 minutes the network freezes and all connections freeze, including ftp, telnet, web access. It seems that for some reason network stops.

We looked the network infrastructure but seems ok.

Is there any suggestions what to look to HP?

George.

[Chan 007]

Hi George,

Can you provide the following

1. What is the OS version and what is the patch level?
2. Or you sure that there is no IP conflict in between your systems?
3. Check the netstat -nr, write a script to keeping the Gateway

Also look for the following

1. Any NFS mounts on the system?
2. Check the /var/adm/syslog/syslog.log
3. Run dmesg

Post these logs so that we can check help.

Chan

[George_206]

HP-UX B.11.00 U 9000/800 125901557
Model:9000/800/L2000-44

Dmesg (we replaced a root disk previous week)

Deferred swap reservation failure pid: 18845
Deferred swap reservation failure pid: 18845
Deferred swap reservation failure pid: 18845
Deferred swap reservation failure pid: 18845
Deferred swap reservation failure pid: 18845
Deferred swap reservation failure pid: 18845
Deferred swap reservation failure pid: 18845
Deferred swap reservation failure pid: 18845
Deferred swap reservation failure pid: 18845
Deferred swap reservation failure pid: 18845
Deferred swap reservation failure pid: 18845
Deferred swap reservation failure pid: 18845
Deferred swap reservation failure pid: 18845
Deferred swap reservation failure pid: 18845
Deferred swap reservation failure pid: 18845
Deferred swap reservation failure pid: 18845
Deferred swap reservation failure pid: 18845
Deferred swap reservation failure pid: 18845
Deferred swap reservation failure pid: 18845
Deferred swap reservation failure pid: 18845
Deferred swap reservation failure pid: 18845
Deferred swap reservation failure pid: 18845
SCSI: Reset detected -- lbolt: 691504928, bus: 1
0/0/1/1.2 tgt
0/0/1/1.2.0 sdisk
LVM: Failed to restore PV 1 to VG 0! Identifier mismatch.
cdfs_check_rr(): Don't want RR



netstat -nr
Routing tables
Dest/Netmask Gateway Flags Refs Use Interface Pmtu
127.0.0.1 127.0.0.1 UH 0 57885333 lo0 4136
100.100.100.1 100.100.100.1 UH 0 57469 lan1 4136
172.22.5.2 172.22.5.2 UH 0 287430902 lan0 4136
172.22.5.0 172.22.5.2 U 2 0 lan0 1500
100.100.0.0 100.100.100.1 U 2 0 lan1 1500
127.0.0.0 127.0.0.1 U 0 0 lo0 4136
default 172.22.5.11 UG 0 0 lan0 1500

[Chan 007]

George..

Also can you check the NIC by running lanadmin
commands
There may a full-half/speed duplex issue? Did you change any Network Port settings on the Switch?

Any NFS mounted FS/application?

Also has it started after the root disk problem was sorted out?

What is the tail -f /var/adm/syslog/syslog.log say?

Chan

[George_206]

As far as I know there were not any changes in the network.

However, I think that it started after the rooot disk replacement. We did not reboot the server after the replacament.

At the same time the HP engineer also installed a version of diagnostics which however hangs.

Is there any possibility of stm conflict?



syslog

Mar 7 15:22:49 cbog_pw sshd[16175]: Accepted password for zafeiro from 172.22.253.14 port 18623 ssh2
Mar 7 15:22:49 cbog_pw sshd[16177]: subsystem request for sftp
Mar 7 15:23:00 cbog_pw inetd[620]: registrar/tcp: Access denied for cbog_pw (172.22.5.2) at Wed Mar 7 15:23:00 2007
Mar 7 15:25:00 cbog_pw inetd[620]: registrar/tcp: Access denied for cbog_pw (172.22.5.2) at Wed Mar 7 15:25:00 2007
Mar 7 15:27:00 cbog_pw inetd[620]: registrar/tcp: Access denied for cbog_pw (172.22.5.2) at Wed Mar 7 15:27:00 2007
Mar 7 15:22:49 cbog_pw sshd[16177]: subsystem request for sftp
Mar 7 15:29:00 cbog_pw inetd[620]: registrar/tcp: Access denied for cbog_pw (172.22.5.2) at Wed Mar 7 15:29:00 2007
Mar 7 15:31:00 cbog_pw inetd[620]: registrar/tcp: Access denied for cbog_pw (172.22.5.2) at Wed Mar 7 15:31:00 2007
Mar 7 15:33:00 cbog_pw inetd[620]: registrar/tcp: Access denied for cbog_pw (172.22.5.2) at Wed Mar 7 15:33:00 2007
Mar 7 15:35:00 cbog_pw inetd[620]: registrar/tcp: Access denied for cbog_pw (172.22.5.2) at Wed Mar 7 15:35:00 2007
Mar 7 15:36:11 cbog_pw sshd[16274]: Accepted password for grig from 172.22.253.14 port 18624 ssh2
Mar 7 15:36:17 cbog_pw : su : + 8 grig-root
Mar 7 15:37:00 cbog_pw inetd[620]: registrar/tcp: Access denied for cbog_pw (172.22.5.2) at Wed Mar 7 15:37:00 2007
Mar 7 15:39:00 cbog_pw inetd[620]: registrar/tcp: Access denied for cbog_pw (172.22.5.2) at Wed Mar 7 15:39:00 2007
Mar 7 15:41:00 cbog_pw inetd[620]: registrar/tcp: Access denied for cbog_pw (172.22.5.2) at Wed Mar 7 15:41:00 2007
Mar 7 15:43:00 cbog_pw inetd[620]: registrar/tcp: Access denied for cbog_pw (172.22.5.2) at Wed Mar 7 15:43:00 2007
Mar 7 15:45:00 cbog_pw inetd[620]: registrar/tcp: Access denied for cbog_pw (172.22.5.2) at Wed Mar 7 15:45:00 2007
Mar 7 15:47:00 cbog_pw inetd[620]: registrar/tcp: Access denied for cbog_pw (172.22.5.2) at Wed Mar 7 15:47:00 2007
Mar 7 15:47:51 cbog_pw sshd[16177]: Received disconnect from 172.22.253.14: 11: Disconnect requested by Windows SSH Client.
Mar 7 15:49:00 cbog_pw inetd[620]: registrar/tcp: Access denied for cbog_pw (172.22.5.2) at Wed Mar 7 15:49:00 2007
Mar 7 15:51:00 cbog_pw inetd[620]: registrar/tcp: Access denied for cbog_pw (172.22.5.2) at Wed Mar 7 15:51:00 2007
Mar 7 15:53:01 cbog_pw inetd[620]: registrar/tcp: Access denied for cbog_pw (172.22.5.2) at Wed Mar 7 15:53:01 2007
Mar 7 15:55:01 cbog_pw inetd[620]: registrar/tcp: Access denied for cbog_pw (172.22.5.2) at Wed Mar 7 15:55:01 2007
Mar 7 15:57:01 cbog_pw inetd[620]: registrar/tcp: Access denied for cbog_pw (172.22.5.2) at Wed Mar 7 15:57:01 2007
Mar 7 15:59:01 cbog_pw inetd[620]: registrar/tcp: Access denied for cbog_pw (172.22.5.2) at Wed Mar 7 15:59:01 2007
Mar 7 16:01:01 cbog_pw inetd[620]: registrar/tcp: Access denied for cbog_pw (172.22.5.2) at Wed Mar 7 16:01:01 2007
Mar 7 16:03:01 cbog_pw inetd[620]: registrar/tcp: Access denied for cbog_pw (172.22.5.2) at Wed Mar 7 16:03:01 2007
Mar 7 16:05:01 cbog_pw inetd[620]: registrar/tcp: Access denied for cbog_pw (172.22.5.2) at Wed Mar 7 16:05:01 2007
Mar 7 16:06:55 cbog_pw sshd[16524]: Accepted password for grig from 172.22.253.14 port 18625 ssh2
Mar 7 16:07:01 cbog_pw inetd[620]: registrar/tcp: Access denied for cbog_pw (172.22.5.2) at Wed Mar 7 16:07:01 2007
Mar 7 16:07:10 cbog_pw : su : + 4 grig-root
Mar 7 16:09:01 cbog_pw inetd[620]: registrar/tcp: Access denied for cbog_pw (172.22.5.2) at Wed Mar 7 16:09:01 2007
Mar 7 16:11:01 cbog_pw inetd[620]: registrar/tcp: Access denied for cbog_pw (172.22.5.2) at Wed Mar 7 16:11:01 2007
Mar 7 16:11:50 cbog_pw sshd[16571]: warning: can't get client address: Connection reset by peer
Mar 7 16:11:50 cbog_pw sshd[16571]: Could not write ident string to 0.0.0.0
Mar 7 16:11:54 cbog_pw sshd[16572]: warning: can't get client address: Connection reset by peer
Mar 7 16:11:54 cbog_pw sshd[16572]: Could not write ident string to 0.0.0.0
Mar 7 16:12:41 cbog_pw sshd[16569]: Accepted password for grig from 172.22.253.14 port 18628 ssh2
Mar 7 16:13:01 cbog_pw inetd[620]: registrar/tcp: Access denied for cbog_pw (172.22.5.2) at Wed Mar 7 16:13:01 2007
Mar 7 16:15:01 cbog_pw inetd[620]: registrar/tcp: Access denied for cbog_pw (172.22.5.2) at Wed Mar 7 16:15:01 2007
Mar 7 16:17:01 cbog_pw inetd[620]: registrar/tcp: Access denied for cbog_pw (172.22.5.2) at Wed Mar 7 16:17:01 2007
Mar 7 16:19:01 cbog_pw inetd[620]: registrar/tcp: Access denied for cbog_pw (172.22.5.2) at Wed Mar 7 16:19:01 2007
Mar 7 16:20:12 cbog_pw : su : + 4 grig-root
Mar 7 16:21:01 cbog_pw inetd[620]: registrar/tcp: Access denied for cbog_pw (172.22.5.2) at Wed Mar 7 16:21:01 2007
Mar 7 16:23:01 cbog_pw inetd[620]: registrar/tcp: Access denied for cbog_pw (172.22.5.2) at Wed Mar 7 16:23:01 2007
Mar 7 16:24:36 cbog_pw sshd[16664]: Failed password for grig from 172.22.253.14 port 18629 ssh2
Mar 7 16:24:39 cbog_pw sshd[16664]: Accepted password for grig from 172.22.253.14 port 18629 ssh2
Mar 7 16:24:47 cbog_pw : su : + 4 grig-root
Mar 7 16:25:01 cbog_pw inetd[620]: registrar/tcp: Access denied for cbog_pw (172.22.5.2) at Wed Mar 7 16:25:01 2007

[George_206]

LAN INTERFACE STATUS DISPLAY
Wed, Mar 7,2007 16:28:18

PPA Number = 0
Description = lan0 Hewlett-Packard 10/100 TX Full-Duplex TT = 1500
Type (value) = ethernet-csmacd(6)
MTU Size = 1500
Speed = 100000000
Station Address = 0x306e0664a2
Administration Status (value) = up(1)
Operation Status (value) = up(1)
Last Change = 3282693249
Inbound Octets = 100152231
Inbound Unicast Packets = 87096362
Inbound Non-Unicast Packets = 456958
Inbound Discards = 73026
Inbound Errors = 0
Inbound Unknown Protocols = 24
Outbound Octets = 1878211256
Outbound Unicast Packets = 100451202
Outbound Non-Unicast Packets = 73026
Outbound Discards = 0
Outbound Errors = 0
Outbound Queue Length = 0
Specific = 655367

Press to continue


Ethernet-like Statistics Group

Index = 1
Alignment Errors = 0
FCS Errors = 0
Single Collision Frames = 0
Multiple Collision Frames = 0
Deferred Transmissions = 0
Late Collisions = 0
Excessive Collisions = 0
Internal MAC Transmit Errors = 0
Carrier Sense Errors = 0
Frames Too Long = 0
Internal MAC Receive Errors = 0

[rick jones]

Payments makes things sound rather important - you do know that 11.0 is past its EOL date right?

The swap stuff sounds ominous - perhaps a lack of swap space after the root disc switch? Also, the PV/VG stuff looks ominous. Of course, not being an LVM guy I cannot say for sure.

When ftp, telnet and web freeze does "ping" freeze too? How about linkloop?

Does the default gateway respond to ICMP echo requests when the network is not frozen? If not, what is the setting of the ndd variable ip_ire_gw_probe?

If you are on the console when this happens, what does the CPU utilization look like on each of the CPUs?

[singvey]

1 check the tail -f /var/adm/syslog.log

2 check who are doing something: whodo

3 netstat -an | grep -i established
check the most number of the connection type

4 with the tool sniffer to capture the data .

5 with nettl check the network traffic


good luck