Is this a memory leak???

[Don Spare]

Within the past 2 weeks both my development and productions L2000 servers have been having serious problems related to Oracle operation. The development server will lock up on any attempts to run any Oracle process or interface. It locks to the point that sometimes I have to pull the power plugs to get the server to reboot.

The production server keeps using more and more of swap in both memory and disk until it locks up. A reboot seems to clear this up for a while but it always returns (at least during the nighttime batch jobs) and causes processes to abort abnormally. But there are no errors in the logs and no trace files.

Any idea what I should be looking for to fix this? We haven't made any changes to either system (other than disk storage from the SAN) in over 6 months. Both are running HP-UX 11.0 and Oracle 8.1.7.4.

[Alan Meyer_4]

Have the developers made any changes to the oracle applications?

[RAC_1]

Any changes to ora apps??
do you have glance on your system?? Monitor their memory usage over a period of time.

You put following in crontab and run it every 1/5 minutes and check if memory usage goes up or not.

UNIX95= ps -ef -o "vsz,args,pid,ppid" | sort -nr1 | tail -20 >> /tmp/some_log_file

Check log file. The first column is memory usage and unit is KB.

[Don Spare]

There have been no system or apps changes in 3-6 months.

RAC: The ps command didn't like the -o option.

[generic_1]

Look at /var/adm/syslog for any issues.
Look at your /var/tombstones/ts99 and see if you have any hardware issues.
Look at your disks in cstm or stm
Look at your san switch logs if you have san and make sure your san looks healthy.
In addition to a runaway apps or memory leaks you could have a hung waiting on I/0 sitution.
Pop open glansplus and make sure you have no disk hotspots. Have you made any recent san or storage changes?

[Mel Burslan]

If there are no intentional changes made to the system before these symptoms started to appear, unintentional changes nneds to be observed.

A component in the system (hardware) may be slowly dying. dmesg, or viewing syslog and looking for errors may be good help to find out something like this.

Another source of locking up is waiting for storage. As you are talking about oracle, I assume your data resides on SAN. Check to see if someone else is beating on the same disks your database resides, which will slow you down immensely if it is the case.

[Don Spare]

Look at /var/adm/syslog for any issues. NO ERRORS SHOWING
Look at your /var/tombstones/ts99 and see if you have any hardware issues. NO ERRORS SHOWING
Look at your disks in cstm or stm FC ERRORS SHOWING BUT THIS MAY BE DUE TO OUTDATED SOFTWARE/PATCHES.
Look at your san switch logs if you have san and make sure your san looks healthy. THIS SEEMS OK. THE SAN SUPPORTS OTHER SYSTEMS WITHOUT ISSUE.
In addition to a runaway apps or memory leaks you could have a hung waiting on I/0 sitution. THIS SEEMS LIKELY BUT SO FAR HAVE BEEN UNABLE TO FIND A CAUSE OR CATCH IT WHILE IT IS HAPPENING
Pop open glansplus and make sure you have no disk hotspots. THE BUSIEST DISKS ARE / AND /ORA_SW1 (WHERE ORACLE IS INSTALLED)
Have you made any recent san or storage changes? WE REARRANGED SOME SAN SPACE ASSIGNED TO THE DEV SERVER. 210GB WAS REASSIGNED TO THE NEW SQL SERVER AND 56GB (A NEW VOLUME GROUP) WAS ASSIGNED TO THE DEV SERVER

[Mel Burslan]

I am not sure what SAN solution you are using, but unless you are the SAN admin or know deeply about it, I would strongly suggest logging a tech support case to the SAN solution provider and asking them to run an anlysis of your physical disk utilization. After the rearrangements, you may have made one or more of the disks too hot, i.e., extremely loaded for i/o and it may be having hard time catching up. This is very hard to determine by using just OS tools.