Unexpected samba 'refresh'

[Wouter Jagers]

Hi all,

Since a while, our samba server is getting into trouble twice a week. Here's what happens: On monday & thursday at 14h00 the load of the machine goes sky-high. This is what usually happens after reconfiguring smb.conf: the machine refreshes every single samba process. However, this happens out of the blue ?!?

We've been looking around, and there are no scheduled jobs concerning the machine at these times. However, we can sync our clocks to the event, so that's worrying..

I do find these in the syslog:

Jul 5 14:00:43 myserver inetd[2766]: netbios_ssn/tcp: accept: No buffer space available

They occur as long as the problem is there.

So, that would mean inetd is having trouble accepting tcp connections. The "no buffer space" error looks like a memory issue at first sight, but apparently it doesn't have to be: Should we be facing some performance issue (system/network) it might happen that this error gets logged when a connection attempt was aborted by the client before it got acknowledged by the server.

Note that the error only spawns on the NetBios session manager.

I do have about 3% physical swap in use (which is not perfect), but I don't see an acute issue in that.

The samba version is 2.0.9.. I know that's not too recent, however that's no reason to suddenly start hicking up.

Any (any!) ideas would be welcome ! With a load of 600 not many of my users can access their files during 10 minutes :/

Thanks in advance
Greetz
Wout

[Simon Hargrave]

Perhaps on of the servers clients (an NT box for example) has an "at" job scheduled to run against one of your Samba shares, and this is causing problems?

Can you enable detailed logging on the Samba shares just before you expect it to happen, and see who is doing what? Sorry not really used Samba so can't suggest a specific option, but presumably there are options in smb.conf

[Geoff Wild]

Hmnn...sounds network related...

Check for errors on the lancard(s)...

Investigate ndd - maybe a tuning parameter...

Maybe, change the network card, may be with network cards buffers.

Try to increase the TCP receive buffer size.

The only thing in samba I can think of is to set:

keepalive = 0

in smb.conf, then restart samba...

Rgds...Geoff

[Wouter Jagers]

Simon: yes, we thought of that, too. We're trying to figure it out, but there are lots of machines here so that might take a while. In the end we might want to sniff its connection.

Geoff: Indeed, the further we look the more it looks like network things. We'll investigate more on that track. (apparently DHCP is not doing much either at that time)

There has been an upgrade of our backbone lately, and of course machines were placed in new network segments while I was on sick leave :-/

Will let you know
thx already

greetz
Wout

[Charles Harris]

Just a quick thought, I've seen this within our environment when one of our NT boys decided to schedule virus scans of networked drives during the day! You could always check your tcp settings via a netstat -p tcp and grep for dropped connections due to a full request que to see if the server it's self is stalling / dropping the connections........

-=ChaZ=-

[Wouter Jagers]

Thanks ! Ok, here comes:

17807 connect requests dropped due to full queue

That's the output now, though.. The problem occured the last time 2,5 hrs. ago. I can't really tell whether this number is significant or not. Would be nice if you could let me know !

thx again
Wout

[Steven E. Protter]

Hmm.

I'd take a look at the HP-UX kernel parameter of maxusers, which usually controlls nproc and nfiles by formula, if not them as well.

Don't forger maxuprc

Seems we are bumping into an OS limit. A quick read of the smb.conf file might be of use in directing this.

Some peformance data collection would be useful as well.

I recommend a conservative, one kernel parameter at a time approch, as far as forumula dependencies allow.

SEP

[Wouter Jagers]

Yep.. it -really- looks like limits are hit.

However, since this only occurs at 2 specific times every week (with unequal times in between: mon & thu @ 14h, that's 3 days and then 4), I'm thinking something abnormal is happening at these times.

Since everything works fine at any other time, I'm a bit reluctant to start oversizing things to try and cope with this boost.. For now I want to assume that this boost should not be there ;-)

All my shells running netstat, iostat, vmstat, ... hang together with the machine, so except for increasing context switches at the beginning I can't get too much info.

Since we're really starting to suspect one 'external' machine to be the culprit, we're going to sniff all the traffic to the machine on thursday when it happens.

Will keep you updated
Thanks a lot
Wout

[Wouter Jagers]

Ouch, this one was never finished, and it's almost been two years. *blush*

Well, just so you know: the problem "magically" ceased to exist some time after the above posts.

I remember seeing one weird thing in the sniffing results, which was a huge cascade of netbios packets coming from certain PCs.

These PCs were found to have dynamic DNS enabled while our environment didn't support this. All PCs were checked for these settings shortly thereafter.

Unsure whether they were related.

No worries about that anymore now, since my datacenter has been migrated away to a cheaper place :(

Wish me luck at my new job ;-)

Cheers

[Wouter Jagers]

The problem ceased to exist.. see last post.