open files

[Shivkumar]

Dear Sirs;

We are dealing with a situation of open files issue on Web Logic 7.x on hpux 11.00.

Vendor had suggested to increase the hardware limit of the socket (maxfiles_lim) in one their config file.

But it has been observed that one of the managed instances crashes every after approx 2 months.

I am thinking if i increase this hardware limit in the config file of the weblogic; i will also need to change it in os level ie. /stand/system...right ??

Secondly, i believe there could be some application who is not closing the socket efficiently. My question is how to identity this application which is opening the socket but not closing it ?

I am also in touch with the vendor in parallel.

I know this forum has gurus with various applications running on hpux; and hence posting this question here.

Any help would be appreciated.

Thanks,
Shiv

[Fabio Ettore · Accepted solution]

Hi Shiv,

I didn't understood very well your first question, anyway about maxfiles_lim:

To be useful, the value assigned to maxfiles must be less than the value of maxfiles_lim. maxfiles_lim is useful only if it does not exceed the limits imposed by nfile and ninode.


So I suggest you to check other kernel parameters above before to increase maxfiles_lim.


About your second question here what you can:

netstat -an

reports all sockets opened, usually in LISTEN status. When you know the port used by application (check /etc/services) then you should see in netstat output sockets in LISTEN or TIME_WAIT or CLOSE_WAIT corresponding that port application.


HTH.

Best regards,
Fabio

[Shivkumar]

The current values set are as shown below:

$ uname -r
B.11.11
$ grep -i maxfile /stand/system
maxfiles 10240
maxfiles_lim 12288
$

Is it okay ?

I observed with lsof|wc -l command that the weblogic managed instances were using only 1057 sockets.

Even if we assume that other managed instances were using approximately 1000 sockets then for 4 instances it might come to 4000 sockets ?

Are these maxfiles values are being used by operating system and other applications running on it ?

Thanks,
Shiv

[Hakan Aribas]

We are running Weblogic 8.1 on Integrity rx4640 servers as application server. I don't remember stuck any file limitation, but I needed to increase max_thread_proc and nproc kernel parameters..

[Fabio Ettore]

Shiv,

I should say that values are ok even if I usually see them with the same values. The most important thing is that maxfiles is not bigger than maxfiles_lim.

Also in order to reply your following question

"Are these maxfiles values are being used by operating system and other applications running on it ?"


I can say what these values mean:


maxfiles sets the soft limit for the number of files a process is allowed to have open simultaneously.

maxfiles_lim sets the hard limit for the number of files a process is allowed to have open simultaneously.


Then they are used by processes independently if they are of operating system or applications.
The important thing is to know that each process can open max that values at the same time.

Have you a problem or do you need just info?

Best regards,
Fabio

[Shivkumar]

Hi Fabio Ettore;

I am facing a problem and this is just not for informational puprose.

Thanks for your help.

Thanks,
Shiv

[A. Clay Stephenson]

Vendor had suggested to increase the hardware limit of the socket (maxfiles_lim) in one their config file.
------------------------------------------
This is hardly a definitive diagnosis of the problem. You could increase maxfiles and maxfiles_lim with no effect. What you need to do is actually gather some data on the failing process. If it crashes then make sure that you capture the core file and do a stack trace. Something failing after two months sounds suspiciously like a memory leak but that too is only a guess. Use glance to observe files in use by this process as well as memory usage. If you don't have Glance then a combination of ps and lsof can be used insterad. The key is to gather and store data on regular interval (~10 minutes). This will allow you to see the approach of the problem. Gathering and analyzing data beats using "The Force" everytime.

[Arunvijai_4]

Shiv,

Try using # nestat -na to find out open sockets. You can find more information about maxfiles_lim at http://docs.hp.com/en/939/KCParms/KCparam.MaxfilesLim.html

max limit is 60000 and Default is 1024.

-Arun

[Shivkumar]

A. Clay Stephenson;

Thank you so much. I just verified now and found weblogic jvm memory size for various managed instances are same as 2 days ago.

My apprehension is some application might be opening the weblogic jvm and are not closing it. But i am not sure under what condition these open sockets crashes the applications.

Thanks,
Shiv

[Rajesh SB]

Hi Shiv,

Why don't you use the utility called "lsof" i.e. List of Opened Files for monitor opened files/socket activity for your analysis?

Download from attachment for HPUX 11.0.

Example:

# lsof -i tcp
Shows the output like
rpc.mount 1121 root 4u inet 72,0x72 0t0 TCP *:49201 (LISTEN)
dtlogin 1220 root 6u inet 0x4283da68 0t0 TCP *:49213 (LISTEN)
diaglogd 1284 root 1u inet 0x42901468 0t0 TCP *:49231 (LISTEN)

Cheers,
Rajesh

[Rajesh SB]

Oops, I forgot to attach the lsof.gz file.
Gunzip first and use it.

Regards,
Rajesh

[Arunvijai_4]

Hi Shiv,

Here is the latest version (4.76) of lsof. You can install and try using it on 11.11

http://hpux.connect.org.uk/hppd/hpux/Sysadmin/lsof-4.76/

-Arun

[Zigor Buruaga]

Hi,

If sockets are not being correctly closed, and they remain at a CLOSE_WAIT status,
I guess people from WebLogic will be able to determine/analyze what are the possible causes/solutions for that. I'm not an expert, but don't see how increasing the number of sockets ( that can also be incorrectly closed ) can help here.
You can observe if your sockets are in CLOSE_WAIT status by running "netstat -a" or lsof ( ok, you already know that ). Try to see if the number is growing.

Regards,
Zigor

[Zigor Buruaga]

Just found an interesting link:

http://support.bea.com/application_content/product_portlets/support_patterns/wls/Too_Many_Open_Files_Pattern.html

Regards,
Zigor

[Bill Hassell]

When you say that weblogic crashes, is there any indication at all that the problem is related to a limit on the number of open files? As Clay said, it sounds like the vendor has no clue and is just guessing. You could change nfile=500000 and maxfiles=maxfiles_lim=250000 and see what happens, but I would ask the vendor to provide a more difinitivae error message or ask the vendor to analyze the core file to determine the real reason for the crash.