Properly sizing syslog servers

[Jeff Schussele]

I need to come up with configurations to properly size syslog servers for 2 separate locations. These servers will log syslogs for all HP, Sun & AIX systems at these locations.

Current server counts are:
Site A -> 250
Site B -> 450
Estimated server growth is approx 5-10%/year

My initial thoughts were:
Site A -> N-class, 4CPU/6GB dual 1000TX NICs & an XP array - dual pathed

Site B -> N-class 6CPU/8Gb 3-1000TX Nics & an XP array - dual pathed

Dedicated V-LAN for the traffic - 100TX to the servers & switched to gigabit for the server.

Are these configs overkill/undersized?
Has anybody set up syslog servers to handle this many servers?
Is this even do-able with single servers?
Thoughts on storage size required for 30 days (minimum)of logs? Log rolled daily but at least 30 days on disk prior to tape archive.
Thoughts on necessity of HA for syslog servers?
Is tape the best long-term archive media?

I want this to be able to handle traffic loads under crisis situations (internal or external attacks/penetrations, etc.) and/or increased storage consumption (SAs want to turn up logging levels, server growth spurt, etc.)

Any real world experience & all thoughts would be greatly appreciated.

Thanks,
Jeff

[harry d brown jr]

Jeff,

The only necessity on HA I can think of is do you need accurate logs of everything, if so, then HA is the way.

For archiving, have you thought about cd's?

live free or die
harry

[Steven Sim Kok Leong]

Hi,

I have channelled syslogs to a central server but not from as many servers as you have indicated.

One point I would think is important to note is that syslog by default uses UDP which is unreliable.

There is however a version of modified syslog which is reliable and that is TCP syslogging. Cisco PIX supports TCP syslogging so no data is lost. The downside is that if the log server gets filled up, the firewall will stop functioning properly ;-)

The other issue with syslogging is that it is cleartext. If you are concerned with a hacker sniffing out system information, then you will need to use either an out-of-band channel or tunnel it eg. over ssh.

I suggest that you test out the load first, log to both local and remote syslogs and compare the records (to check for any loss of data).

The concern that the syslog server could possibly be DoS'ed is valid. Thus, it is important in my opinion to restrict the UDP or TCP syslog traffic from legitimate IP addresses to the remote syslog server via a host-based firewall (eg. IPFilter/9000, Netfilter) or a network-based firewall (eg. Checkpoint, PIX, you name it).

Hope this helps. Regards.

Steven Sim Kok Leong

[Celso Medina Kern]

Hello Jeff,

I have no real life experience with syslog servers, but I see no point about sizing without having a syslog budget for each of your hundreds of servers(at least patterns). You should collect the amount of syslog records on these servers to make your sizing, since amount of data in syslog may greatly vary. I have seen syslogs from 20kb/day to 20MB/day. There is a lot of possibilities about

I think there is nothing new here, but that??s what I think should be your starting point.

Good luck

Celso

[Sandip Ghosh]

With around 80 concurrent users and logging enabled the size of syslog file goes to 5 MB per month. It greatly depends on the No of users and the mode (logging/debug) of the inetd.

In my opinion you need to have the powerful server because it will not do any kind of processing. It will only collect the data. So one L class with 2 cpu will be more than enough. But you must have a 100 Mbps network Card and lot of disk space to store the data according to your requirement. It will be better if you can add a cd writer to store the data alongwith the tape drive.

Hope it helps you to take your decision.

Sandip

[Mark Greene_1]

Jeff,

I would think that a single-cpu L2000 or L3000 would be sufficient. The load you are talking about is not cpu-intensive; rather, your bottlenecks are going to be the network (which you appear to have adequately addressed) and disk I/O.

If you only need 30 days' of logs on-line, you could almost go with a JBOD array: either the SC10 or the FC10. They can have 10 drives each from 18gig drives up to the 70gig drives, and you can probably get away with using Mirror/UX to build mirrored pairs out of them. This will save you a ton of bucks that you can better spend on filling the system with the maximum amount of memory, multiple LAN cards, redundant power supplies, PDU's, power leads into the system, and a UPS if your room is not already UPS'd (but my guess, from the number of servers you have, is that you are probably all set there).

A couple of DLT's, if you don't have an enterprise back-up solution in place, wouldn't hurt either.

HTH
mark