Operating System - HP-UX
1830935 Members
2470 Online
110017 Solutions
New Discussion

database network connections failing

 
Steve Lewis
Honored Contributor

database network connections failing

We have a 3 tier application with the database on a VPAR connected via a 10GbE fibre wire to a Procurve switch, then via 1Gbit link to the app. server. We have 4 instances of this configuration with problems happening on just one of them.
We are getting frequent errno 235 errors in our database log which causes the application programs to fail to connect to the database through a TCP socket.
I managed to replicate the error (alongside 400 concurrent users) and got hold of the netstat -s output either side of a 15 second delay when I reproduced the error myself.
I found a few interesting differences, these are the error stats:
TCP:
before 23113 dup acks
after 23244 dup acks
before 17 out of order packets
after 36 out of order packets
before 537660 window update packets
after 539018 window update packets
before 5398710 segments updated rtt
after 5415132 segments updated rtt
before 346 retransmit timeouts
after 351 retransmit timeouts
UDP:
before 42 keepalive timeouts
after 44 keepalive timeouts

The thing is, before the weekend our database connection was going via a 1000-base-SX lan through another server which was acting as a lan bridge/router, we were only getting about 5 of these messages ever 24 hours, now we have both servers connected directly to the same switch we are getting one error every 10 seconds.

I am not an expert on the TCP/IP stack, do those error stats indicate anything out of the ordinary?

We also tried disabling our 10GbE and going 1Gbit coppper to 1Gbit copper and got the same rate of errors.
The pattern of work has not changed since last week and neither has the database configuration.
Is there anything else I can investigate?
1 REPLY 1
Steve Lewis
Honored Contributor

Re: database network connections failing

Any ideas stateside?
I can only think - dodgy network card, or
missing patches.
I am checking the patch level on the server now. The 10GbE cards are on 11iv2 with a slightly higher patch level, so I will try to get permission to put the same ones on this box to see if anything is improved.