BIG Performance Problems

[Nicolas Gagnon]

Hi,

I have 2 L2000 4CPU 2GIG RAM (HP-UX 11i) connected to 9TB SAN (VA7400) via Brocade switch. These 2 servers are SAMBA server only (2.2.3). I have about 25 users per server. filesystems have about 1TB. We have special need. We proces about 2-4 TB of stuff per day!

Sice 2 weeks, I've had major performance problems. Proccessors are 100% busy! Each SMBD processes takes large percentages. I've started glance+ and found that each SMBD processes spend all of the time in system calls (mostly read and select). I haven't change anything for last year...

Is there any survey about SAMBA and system calls.

I've attached a screen shot of glance+ system call output.

thanks.

nicolas Gangon

[John Poff]

Hi Nicolas,

Has anything changed in your environment in the last two weeks? Patched your systems? Updated your application that uses the Samba shares? Any updates to the Windows that your users are on?

Here is a thread about a performance problem with Samba 2.2.5 and a workaround, but I don't know if it will help you.

http://forums.itrc.hp.com/cm/QuestionAnswer/1,,0x4415237a4bc6d611abdb0090277a778c,00.html

JP

[Nicolas Gagnon]

Since the problem have occured we've putted some patches and upgraded SAMBA. but we didn't do anything before.

[James Murtagh]

Hi Nicholas,

I don't think anyone can give you a magical answer for this, you get have to dig deep into the problem, using glance, sar etc. There are some good performance cookbooks on the ITRC I believe. Basically you have to find what subsystem(s) is causing the problem. Perhaps use the Performance Advisor window from gpm?

From what you have said, the 100% CPU utilisation could possibly be red herring. If nothing but Samba is running on these servers, these figures could be caused by a number of other performance problems. If nothing has changed on the servers, and you are processing a lot of files per day, it could possibly be down to fragmentation of the filesystems. This would certainly get worse over time.

I assume these are vxfs version 4 filesystems as you are on 11i? Do you have Online JFS? If so, I would be interested in seeing a fragmentation report of one of the filesystems :

fsadm -F vxfs -E -D

Also, check all the usual system logs for any hardware problems.

Regards,

James.

[rakotoarisoa]

Hi Nicolas,

Here are somes ways you can use in order to identify where your pb can come from:
* What change(s) did you made since the last time you can remember?

* Try to locate where the bottlenecks are:
i/o
memory
swap space
switch
network components
application(s)

Goog luck!

[harry d brown jr]

One, consider MORE memory!

What are your kernel parameters set to?

and what does your swap look like?

How many fibre channel's per L2000 to the switch do you have?

How are the lv's built? Size of lun's? striping?

live free or die
harry

[Paula J Frazer-Campbell]

Hi

From just your glance screen I would take a serious look at your disks during the busy periods.

sar -d 1 100

Also glance and examine lvols and then disks.

Paula

[Rory R Hammond]

My samba log file is
located at
/var/opt/samba/log.smb

I would look there first.

I would then look to see what data files are being accessed at the time of your problems
(find /director -exec fuser -u {} \;

To see if anything is unusual.

I would also check for grins what my network load is. At one site I luckly solved samba performance by simply putting my server on a 10/100 switch.

good luck