Awash in syslog...

[Sorrel G. Jakins]

I have 650 access switches, 45 distribution switches, 45 HP (live free or die UN*X) servers, 60 IA32 servers, and a hodgepodge of monitors and agents, as well as CA?Unicenter. All produce logs of one type or another.

I tried funneling the syslogs to a single server, which promptly overflowed and died.

What do the rest of you do? (Generous points will be assigned)....

[Pete Randall]

Sorrel,

I thought that was supposed to be where CA-unicrap excelled. If it did, indeed, handle all the monitoring then couldn't you just routinely clean out all the other logs. I'm sure it would take a while to set up Unicenter but . . .

Pete

[Sachin Patel]

Hi Sorrel,
I run a script every couple hour which goes one by one system and collect the information that I am interested and sends me email.

Attached is that script. It is not pro. script but it works for me.

It is sigle script it run on individual system. You have to put wrapper so it runs on many systems

something like
for i in sys1 sys2 sys3
do
remsh $i system_watch
done

Sachin

[A. Clay Stephenson]

I use IT/O (excuse me VP/O) but I assume that these agents are going to mess up at some point so that ALL syslog stuff is logged locally and the agent's job is to forward important stuff to the IT/O server. I suggest that you use syslog for the purpose it was intended on each box and Unicenter's job is to make sense of this. You should also have daemons/agents whose job is it to clean up the syslogs on a periodic basis.

[MANOJ SRIVASTAVA]

Hi Sorrel

We use ITO for event notifications and errors , I ahve small scirpts which work fro stuff like disk space util etcetc . Howeever for the historical data part we ahve perfview configured in which the data is stored locally on the servers for pulling the data we use something like a monitoring server which pulls the data ( incedntly we use the same server for ITO adn Omni back) . Also in case we include the back up of the logs with the fielsyste abck of the server so that we donnot tread into unknown waters . typically we scale down to some application stat and ofcourse Oracle .


Manoj Srivastava

[John Payne_2 · Accepted solution]

The problem, (I work for Sorrel) is that we were logging things locally, but some machines ended up being way more verbose than others, and as we brought more and more servers in, Unicenter did not seem to scale it's ability to handle all these messages well. Also, there is the problem of losing logs and wanting them weeks later. (Doesn't happen much, but it does happen.) We thought that if all messages were moved to a central repository, then at least there was a point to find things much later after logs were trimmed. Unicenter's saf (Store and Forward) facility also stores messages if the cci bandwidth was not sufficient at the time.

How does IT/O scale? Sorrel posted the magnitude of our environment, does anyone have something about our size that they can manage well? (We are a University. We use cheap unskilled student labor for our Operations staff. The easier things are to use, the better...)

John