It is likely your server is having a fit of sudden memory shortage. Maybe your App has a mis-configuration and a process (or a gang of them) all of a sudden comes up rapidly depleting memory.
How long has this been up and were there any changes? DO you have performance monitoring enabled? If not and you do not have any of those fancy monitoring add ons - I sugegst you install SAR (install sysstat package) and set it up to gather daily and per 5 minute stats gathering. You know how to set up and use sar - right?
If no, it is easy:
install sysstat package (yum or etc..)
set up it up in cron for 5 minute gathering:
cd /etc/cron.d
vi sysstat
One set up, it should be gathering and storing performance historicals for up to a month.
To view the current memory historicals for the day:
sar -r
To view SAR memory stats last 5th of the curent month:
sar -f /var/log/sa/sa05 -r
To View memory stats real time in 5 second intervals for up to 10 intervals:
sar -r 5 10
So set SAR up and see if you have progressive Memory/Virtual Memory decay. SAR should log the current day's stats even after you've rebooted your server. It should have captured the stats.
Hakuna Matata.
