TNS buffer overload

[Jakes Louw]

Hi guys
Currently having timeouts on a heavy load of sql connections into a Ora db on a 11.11 SD-64. Error is "TNS-12560:TNS:protocol adapter error, TNS-00530: Protocol adapter error, HPUX Error: 233: No buffer space available".
Now obviously the buffer for incoming connections is too small. We can't start another listener with a different port ID at this stage, so what to do? Is this a kernel issue, an SGA issue, or a ethernet driver issue?

[twang]

Problem is generally seen during high workload on machine.
Can you take a look at this HP-UX system file /usr/include/sys/errno.h

Oracle will usually retry the operation when it detects this error from a
system call since it indicates there is a transient resource shortage.
It is the Operating system that refuses the resource request from the listener.
The virtual memory allocation routine of the OS will determine if a swap can be
made to disk of a real memory segment thereby allowing the listener access to
some more real memory.
So now the only fix for your case is to free up memory on the machine.

[Jakes Louw]

Nice try. I don't have a memory problem on that server. It's a 64-way Superdome with 64 GB RAM, and only using 40% memory. i have seen some forum responses quoting DBC_MAX_PCT, but I'm not risking a lengthy downtime for a kernel param change unless I have a definite answer.

[Indira Aramandla]

Hi,

I agree with Twang. The reason / cause for the error â TNS-00530: Protocol adapter error, HPUX Error: 233: No buffer space availableâ is generally seen during high workload on machine. Connections to the database hangs and errors will be reported in the listener.log as â TNS-12560: TNS:protocol adapter errorâ , â TNS-00530: Protocol adapter errorâ .

The underlying problem of these errors is the operating system. The HP UX (HP 9000 Series) system file /usr/include/sys/errno.h will show the error.

ENOBUFS means there is no sufficient memory available and the system(kernel) can not allocate any more.

Application will usually retry the operation when it detects this error from a system call since it indicates there is a transient resource shortage. It is the Operating system that refuses the resource request from the listener.


I hope this expla

[Jakes Louw]

Thanks, guys, I'm pretty happy with the explanations: what I need now is a fix from the OS side: what extra resource must now be allocated? See my comment above regarding DBC_MAX_PCT which is currently set at 10% (of 64GB) which gives 6.4GB buffer cache.

I can't justify a reboot on a 1 TB database server unless I'm pretty sure I can fix this. Please ask your HPUX OS guys if this was the fix, or query Oracle Meta for the real solution.

[Brian Crabtree]

What version of Oracle are you currently using?

How many connections are being used?

OLTP or DSS?

Are you using memory windows?

Thanks,

Brian

[Brian Crabtree]

And please post your listener.ora, and the tnsnames.ora connection that you are using to connect to the database (one file if possible).

Brian

[Jakes Louw]

We are on 8.0.6-64 at the moment.
Normal connection expectancy is around 2500 max concurrent. OLTP during the day, DSS at night, also snapshotting to another DSS D/B.
Been running live like this for over a year with this precise config, and have never seen symptoms as were seen as per initial posting.
BTW: problem mysteriously vanished yesterday at around 10h30 local time (GMT - 2).
Subsequent discussions with other sysadmins on other vendor kit (we run a huge operation) indicated that varying symptoms were experienced by them also. What we are possibly looking at here is an intrusion attempt into our corporate databases, but this fellow must have been on a local subnet, as we are pretty extensively firewalled (although, is that effective any more?). I think what we were seeing was hundreds of Ora login attempts per minute, and the listener couldn't handle the load. Surely that will show up in the listener log?

[Jakes Louw]

Oh, yes: we are running memory windows. The Ora error pointed straight at the OS, though.
I also suspected the SGA until we found the error message. TNS and listener files haven't changed in months, but I can post them if you're really interested?

[Brian Crabtree]

It sounds like you have a plausible explanation. You might want to consider setting up a MTS system for OLTP connections, and a direct connection for DSS connections, which should help reduce the amount of connections required.

Thanks,

Brian