Strange problem with FTP server on HP Proliant DL380 G3

[Hannes Piirsalu]

We have anonymous FTP server running on mentioned server and serving data to 800 simultaneous connected users. OS is Debian testing with 2.4.29-grsec kernel. FTPd is vsftpd. All was working well for one month, when one morning server load was 100 and more. No major changes to configuration was made. Only chroot for another ftpd. There is usually 10 connected users. Problem might be IO throughput. Data comes through SA6404 SCSI controller (onboard 5i is disabled) and 6 72gb 15k disks in RAID0 mode, reiserfs filesystem in MSA30. Max data rate is about 450 Mb/s. Both FTP -s are on GB ethernet and Tg3 NIC -s.
Is there any way to change some cciss values in
cciss.h? We have tried other kernel versions, but load of the server remains the same (with 2.6.8-1 and 2.6.9 was horrible - load 500 and max data rate about 40 Mb/s instead of 450, CCISS v.2.6.2 driver was problem I think). Most strange is that load one day just went up and stays there. Vmstat 1 output is attached. Its made when server is under load average: 88.93.
We allsaw tried to run it on RHEL 3.0, with HP scsi drivers and etc, but performance was even worse than on Debian. Debian uses HP CISS Driver (v 2.4.52). Any ideas, or where I should look into?

[Steven E. Protter]

I would look into the vsftp version, check for a patch. There may be a problem with this server, which scales up very well and is used by RedHat for their massive ftp server.

I would also look into who is using the server and make sure they are limited to three or four sessions. You may be under attack designed to deny ftp service to your legitimate users.

SEP

[NiCK_76]

Hi Piirsalu,

My suggestion that reproduce every setting and service to last konw good environment to have a try.That will help you to find out real problem. If not , upgrude kernel,vsftpd
,firmware and driver to have a try.

NiCK

[Hannes Piirsalu]

Steven, Nick, thank You for quick reply :)
It's hard to do this rollback You mentioned, because no canges to RAID config and etc was made. And patch for vsftpd is not available, at least not on their webpage.
Only thing we did, was this chroot.
Yesterday I started to think, that maybe one of these 72gb disks is somehow broken. Few weeks ago max speed was ~530Mb/s. Now its 420Mb/s. Even if disk usage is low. So disk problem might be possible. But how to detect this? cpqarray doesnt give that information. Is there any tools for debian?

[Hannes Piirsalu]

In the meantime I made some checks on the system. Load average is still 200. Allsaw tried mpstat -P ALL 2 5. Output is attached, and strange is that iowait% is 0,00. It means, that DISK output is not the problem?
Firmware of disks and SA is upgraded. Kernel is latest 2.4 available and compact as possible, all unnecessary is taken out. Checked firewall logs, and there is nothing unusual either. We are using FireHOL as iptables firewall builde. And protection strong 50/sec is activated.

[NiCK_76]

Hi,Piirsalu ,

I need more infomation to find out the problem. Could you attach sysreport to me?
Perfrom command "sysreport" in the RHEL3,it will generate a compress file of os information.

NiCK

[Hannes Piirsalu]

Hey, Nick! Like I mentioned in my first post, we used RHEL 3 before, but changed to Debian, because Debian worked better until this slowdown. Now, after I deleted about 1000 small files - disk usage is about 75%, performance is much better - max speed is about 480Mb/s instead of 420Mb/s. So disk usage must be all the time below 75%.

Allsaw I searched google and forums here - what is the max speed, what expect from those 6x 72gb U320 15k disks connected to SA6404. Anybody have done benchmarking? I think 66MB/s
(530Mb/s) is maximum.

[NiCK_76]

Hi Hannes Piirsalu ,

For your situation that should exclude possiable problem and tuning performance
1/upgrade driver,firmware,kernel,ftpd software
2/Tuning network performance. I need to konw how about your network environment? I need more information ,such as NIC ,NIC driver version,cable, switch
3/Tuning DISK performance.
a/ tuning raid write/read percentage . For your situation(read operate more than write) I suggest that set write rate as 25% ;read rate as 75% . Use ACU from HP to do it.
b/ tuning raid strip size . Set small strip size ,if you have more small file in ftp directory. Otherwise , set large strip size. Use ACU from HP to do it.
c/ tuning file system. For most read operate system, use reiser4 file system . For more file system benchmark information that read following URL
http://www.namesys.com/benchmarks.html

Good luck
NiCK

[Hannes Piirsalu]

I solved my problem. increased value in /proc/sys/vm/max-readahead file. Originally it is 31. Echoed this to 127 and disk read performance went from 420Mb/s to 500Mb/s. Tried some other values too, but 127 is best.
Allsaw tuned some other proc parameters and now average load is 10, instead of 100.