bandwidth usage very High

[kcpant]

Hi friends,

from time to time, I 've discussed in this forum some stories of migration from M$ to RH Linux.Now , in this stream, I want your help on a problem desperately.

One of my sites, which was on M$ exchange/M$ ISA srv before, is running RH Linux 8.0 with sendmail ( sendmail+mailscanner+clamav+ fetchmail etc.) and squid. the problem is with ISP bandwidth usage. the bill of DSL provider was only one-third of what we are getting after migration. I've tried to tune everything, like tighten the security with iptables, closed all unneccessary ports etc, but no luck!

so in this regard, please suggest what can be done. I've also tried to analyze with Ethreal, but it's a very cumbersome job to analyze packets of 4-5 hours or a day.please suggest any other tools available to analyze bandwidth usage port-wise/application-wise and get a summary of this all. also about any tool for analyzing squid usage and find per user bandwidth usage on SQUID.

waiting for reply.

[Steven E. Protter]

At this point you are not sure what is using the bandwidth.

You need to find out.

Let me lay out some possibilities.

1) Someone is relaying spam on your server. Check the file /var/log/maillog for unexplained mail. If you see some, its time to tighten up the sendmail configuration. Lots of threads here on that issue.
2) Users are sending large attachments. By default, sendmail places no limits on attachments. I limit attachments by having users do the webmail route which lets me keep attachment size down.
3) Other websites could be abusing your httpd server to serve pages or attempting to relay popup adds. Look at /etc/httpd/logs/access_log for unusual entries.

Have the ISP give you a report on the traffic.

Get a new ISP.

Many ISP's don't bill by the byte.

I can recommend one offline. Let me know and I'll give you a contact link to discuss this via email.

SEP

[xyko_1]

Hi kcpant,

I suggest you to begin your investigation using MRTG to see if your traffic is what your ISP is accounting you.

Second, you may run SARG to analize Squid logs.

I guess that those information will help you at the first moment. Probably you will have to do a more deep investigation after, but this is a good starting point.

The links :
MRTG => http://people.ee.ethz.ch/~oetiker/webtools/mrtg/
SARG => http://sarg.sourceforge.net/sarg.php

Regards,
Xyko

[kcpant]

Hi steven & xyko,

thanks for immidiate reply.Steven, I've checked the maillog, no spamming as per my knowledge is relayed through my server. moreover, as sendmail security concern, I've permitted relaying from my LAN clients only ( relay_hosts_only feature). please let me know briefly if I can use any other feature/define rule to further tighten it. moreover, please tell me what to see in maillog about 'unexplained' mails. I do not have httpd service activated on this server. the only services running are
1. http (for squid),smtp (open from both sides) 2.POP, SSH (open only for LAN)

Hi xyko, thanks for sending links, I'll try to run MRTG and SARG for monitoring.
I'll appreciate if anybody can send me information about any tool monitoring SQUID usage machine-wise/user-wise on the LAN.

thanks & regards

[kcpant]

Hi xyko,

I've installed SARG and it's working successfully ( for sarg, I'll open httpd for a while, whenever I want to read reports of it, or I'll directly open the indexs file from it's location without compromising security by opening httpd), but MRTG is not working. though all prerequisites installed, it gives error about gd. I've tried to re-install gd, but the pckage gives error when I try to makefile through perl. can you please send me any link to rpm version of MRTG?

Thanks & regards,

[kcpant]

Hi friends,

I've managed to install MRTG through rpm, but a new problem waiting! when I run cfgmaker pointing to localhost, it cannot determine snmp parameters of the same. I've tried to diagnose snmp by running snmpwalk, but there is no response!it seems there is some problem in snmpd and it's configuration.
please guide me in this regard how to get MRTG running.

thanks.

PS: I'm running RH Linux 8.0, and iptables configured. I've also tried to telnet port 161 , but got no response.

[xyko_1]

Hi kcpant,

I was out for a few days that's why I didn't reply you earlier.

About SARG, you may scp squid logs to another machine to not open http server on that server. This will reduce your expose.

About MRTG. I'll do some research because I never had problems running it asis.

I'll be back as soon as possible.

regards,
xyko

[xyko_1]

Hi kcpant,

From where did you get your MRTG rpm ?

There is a RH8 version at rpmfind that may suite your needs.

Go to http://fr2.rpmfind.net//linux/RPM/redhat/8.0/i386/mrtg-2.9.17-8.i386.html

and get the package.

Before install this package remove (rpm -e) the one that you have installed before.

good luck.
xyko

[kcpant]

hi Xyko,

I believe you have already read my reply on another thread.please guide me if there are other tools to determine security leaks, and threats which are responsible for choking my DSL bandwidth.

thanks & regards

[xyko_1]

Hi kcpant,

We are not sure at all that your problem about high bandwidth usage is because you are been hacked.

Let's monitor your network trying to discover if there is any site with a very intense communication with your server.
I think NTOP can be useful to you to accomplish that.

NTOP => http://freshmeat.net/projects/ntop/?branch_id=7279&release_id=183226

regards,
xyko

[KristofH]

If you want 'per interface' statistics, have a look at mrts, wich works together with mrtg: http://apt-get.dk/mrts/

Or, if you need a per-service analyzer, ntop can help you, or some iptables-analyzer. (you could find some tools for this on freshmeat.net or sf.net)

Cheers!

[Dave Falloon]

Another tool that is very useful is nessus, it is a penetration tool that gives you a list of open holes in your system and ways to close them.

I used to work for an ISP, we had a quick way to see if there was a mail problem on a server. Typical servers had 300 users, if I ran this command:

expr $(mailq|wc -l) / 3

which just gives me the rough number of messages in the queue by pumping the output from the mailq command to word count to get the number of lines and then divides that by three because mailq outputs three lines per message.

If the mail queue on a server that has 300 people is over 2000 messages, you probably have a problem. Run mailq by itself to see who the recipients are, if you have 2000 messages being sent to name@aol.com, chances are pretty good you are relaying spam.

Also its helpful to graph your sendmail stats:

http://nrg.hep.wisc.edu/sendmailstat.html

You are running squid on this machine as well?

I would double check those logs to make sure squid is locked down.

--Dave

[kcpant]

hi friends,

thank you very much for participating on this thread.I 'm sorry for posting after a long time.

Xyko, I've used ntop, but it's very cumbersome to find the information actually needed, like, top most sites used in a specific time , from a specific host, or top most processes eating up the tcp traffic from WAn link etc. can you tell me any utility which I can use for this purpose?

Hardy, thanks for sending the link of mrts, but I've n't found much superiority of it on normal mrtg reports, except it also gives per day usage in GB count.

friends, I'm desparately looking for a utility which can be used for tracing which process or host on LAN is eating up my WAN traffic , on a specific period of time? I believe their must be one based on mrtg and rrdtool for this purpose.please help me in this regard.

thank u all again

[kcpant]

closing thread