Interesting Samba/CIFS problem

[Steven E. Protter]

This may get a little long, but I'm trying to provide some detail.

4 samba/cifs nodes

node 1 jerusalem
node 2 shalom
node 3 storage
node 4 hpweb running hpux cifs/9000

Storage is an old kayak running Fedora Core 2
node 1 and 2 are Red Hat AS 3 boxes.

node 1 3 4 can all do cifs/samba connects to each other no problem.

The command syntax
mount -t smbfs -o username=root,password=nottelling //nod1/shared /share

Syntax is slightly different for hpux but it works to node 1 and node 2 no problem.

None of the nodes can do a samba mount to node 3.

ssh, telnet works great. No firewall on the node 3 box because its for internal disk storage, not Internet exposure.

Here is the error you get from trying to do a mount on node 3:

Error connecting to 192.168.0.50 (No route to host)
24353: Connection to storage failed
SMB connection failed

I have searched the net in google far and wide and gotten no solution.

Thing is I have complete connectivity as far as i can tell.

Here is an interesting entry from the samba log of the client attempting the mount. I've also searched on that and gotten nothing useful.

getpeername failed. Error was Transport endpoint is not connected
[2004/09/05 20:35:20, 0] lib/util_sock.c:get_peer_addr(952)
getpeername failed. Error was Transport endpoint is not connected
[2004/09/06 00:00:17, 0] smbd/server.c:main(747)
smbd version 3.0.2-6.3E started.
Copyright Andrew Tridgell and the Samba Team 1992-2004
[2004/09/06 01:55:35, 0] lib/util_sock.c:get_peer_addr(952)
getpeername failed. Error was Transport endpoint is not connected
[2004/09/06 04:02:01, 1] smbd/server.c:open_sockets_smbd(342)
Reloading services after SIGHUP


The server resolves in DNS.

It has dual unteamed gigabit NICs through a cheap gigabit switch. All traffic to and from the nodes is travelling through that switch.

I'm attaching a log of network activity from tcpdump to this case.

I am mystified and have no idea how to proceed. I've patched up the target system and can find nothing wrong with the networking setup. netmask and all other parameters are fine.

Any ideas?

From the log it would appear that there is some kind of response from node 3.

Extra: node 3 has no problems doing samba mounts on nodes 1 2 and 4.

SEP

[Steven E. Protter]

I did try the smb.conf file from nodes 1 and 2 with modification. Same problem. No dice. Seems like a networking problem on nodes 1 2 and 4. But what do I look for?

SEP

[Sridhar Bhaskarla · Accepted solution]

Hi SEP,

//
storage.investmenttool.com > shalom.investmenttool.com: icmp: host storage.investmenttool.com unreachable - admin prohibited //

I believe storage is not able to resolve it's fully qualified domain name (storage.investmenttools.com) on itself.

I would try looking in that area.

-Sri

[Steven E. Protter]

Thanks for looking at the log Sri.

I'll investigate and award points accordingly

If I have trouble with the log, I'll let you know.

SEP

[Steven E. Protter]

fascinating:
PING storage.investmenttool.com (192.168.0.50) 56(84) bytes of data.
64 bytes from 192.168.0.50: icmp_seq=0 ttl=64 time=0.264 ms
64 bytes from 192.168.0.50: icmp_seq=1 ttl=64 time=0.293 ms

--- storage.investmenttool.com ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1013ms
rtt min/avg/max/mdev = 0.264/0.278/0.293/0.022 ms, pipe 2
[root@haifa root]# mount -t smbfs -o username=invest,password=swa1965 //storage/shared /share
Error connecting to 192.168.0.50 (No route to host)
2704: Connection to storage failed
SMB connection failed

If ping works, why won't samba. Note: seemingly no network activity in a tcpdump on node3(storage.investmenttool.com) when this mount fails.

I'm wondering if this is a samba flaw but the versioning is totally different between nodes 1 2 and 4.

SEP

[Massimo Bianchi]

Hi,
is there any strange configuration, like routing table ad-hoc or a socks config ?

HTH,
Massimo

[Massimo Bianchi]

Hi,
just some additional thought:

- are you able to run nmap against "storage" ? What is its output ?

- are you able to mount, from storage , a share, using loopback ?

- are you able to properly show the shares from the node itself ?

- did you check the mac address against the ip address (just to be sure...) ?

from your statement looks like the problem is in the cifs server part, not in the client. Are you able to use nfs mounts ?

- is there the file /etc/protocols ?


- looking from google http://lists.samba.org/archive/samba/2004-May/085856.html

looks like the server maight be listening on another address. Is this possible ?

HTH,
Massimo

[Geoff Wild]

Just a stab in the dark, are they all in the same subnet and if so, do they all have the same netmask? (note - this is for the primary lan interface).

Rgds...Geoff

[Steven E. Protter]

Here are the routing tables netstat -r

Seems subnetting is okay. The difference between node 3 and the others is that its totally intenrnal. Its mean to be a shared nfs/cifs mount for all machines and not to be exposed to the Internet.

Right before I hit submit the subnet thought occurred to me so I ran the config files on all four workstations and found the 192.168.0 network all properly subnetted to 255.255.255.0

output:
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
192.168.0.50 192.168.0.41 255.255.255.255 UGH 0 0 0 eth0
10.1.10.0 0.0.0.0 255.255.255.0 U 0 0 0 eth2
192.168.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
66.92.143.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth2
127.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 lo
0.0.0.0 10.1.10.254 0.0.0.0 UG 0 0 0 eth2

I did the route add for the first entry myself. the 169 and 10.1 network are inactive, assigned to a card i'll eventually use for clustering.

SEP

[Sridhar Bhaskarla]

SEP,

Did you ping to 'storage.investmenttool.com' from storage box itself?. The packets from storage are indicating that the box 'storage' itself is unable to recognize it as 'storage.investmenttools.com'. I think you may have to reconfigure samba on storage to allow connections for storage.investmenttools.com.

-Sri

[Massimo Bianchi]

Hi SEP,
you said:

the 169 and 10.1 network are inactive, assigned to a card i'll eventually use for clustering

I think that the samba file sharing got bind to that card, and is not answering to the other subnets.

Can you disable that card, and restart samba ?


HTH,
Massimo

[Steven E. Protter]

No Sri, I did not do the ping test from the box itself.

I think you have actually identified the problem.

The ping on the box to itself returns the following:

64 bytes from storage.investmenttool.com (127.0.0.1): icmp_seq=0 time=0.055ms

A ping on a properly working box returns this:


64 bytes from storage.investmenttool.com (192.168.0.40): icmp_seq=0 time=0.055ms

It actually returns the public IP address but I'm not going to post that.

So the obvious issue is that networking configuration is just plain broken on storage.

I'm open to general ideas on what might be wrong but since it's Linux may have to go to that forum for how to fix it. As number 1 Linux point getter, I should be able to figure it out myself.

HP-UX pings locally to fully qualified domain name also return proper answers. Something is messed up and my box defaults to loopback.

I doubt its dns

Fascinating. Thanks for the help, points will be assigned liberally.

SEP

[Steven E. Protter]

Massimo,

Not ignoring you. node1 does not have any spare or unused network cards and gets the same results.

Since I have changed the networking I think a new tcpdump is called for.

I'm unclear what the hp-ux syntax should be to attempt the mount. If hp-ux can do it and linux samba can't it points to a bug in the client.

SEP

[Sridhar Bhaskarla]

Hi Sep,

It can be easily fixed if it is a HP-UX box. I believe you will just need to reconfigure the 'local' hosts file to resolve the IP address to it's short hostname as well as it's alias - the fully qualified hostname. I believe you know how to fix it.

-Sri

[Steven E. Protter]

I have actually fixed it.

Consider the thread on hold until i get hands on and re-run the tcpdump. Still can't connect and there may be other networking issues.

Quite a fascinating little issue.

SEP

[Steven E. Protter]

Mystery solved, hopefully.

01:41:50.206262 shalom.investmenttool.com.33230 > 192.168.0.30.netbios-ssn: . ack 435 win 5840 (DF)
01:41:50.207412 shalom.investmenttool.com.33233 > storage.investmenttool.com.microsoft-ds: S 498849426:498849426(0) win 5840 (DF)
01:41:50.207616 storage.investmenttool.com > shalom.investmenttool.com: icmp: host storage.investmenttool.com unreachable - admin prohibited [tos 0xc0]
01:41:50.223480 shalom.investmenttool.com.33234 > storage.investmenttool.com.netbios-ssn: S 492637380:492637380(0) win 5840 (DF)
01:41:50.223725 storage.investmenttool.com > shalom.investmenttool.com: icmp: host storage.investmenttool.com unreachable - admin prohibited [tos 0xc0]


storage.investmenttool.com does not have a machine account on the primary domain controller.

How the Windows PDC got involved I'll never know, but 192.168.0.30 is that box and needs to be configured. I bet the connection works after that. Sri earned a couple bunnies for sure.

I'm not going to get to fix this until Saturday night.

SEP

[Steven E. Protter]

As usual, Sri was right, hence the bunnies.

He didn't have enough information to know WHAT the network problem was.

Even with full knowledge and an exhaustive comparison of the network configuration with working boxes I was unable to determine the cause of the problem.

Since it was a Fedora Core box, I merely reinstalled the OS. That worked.

I think the souce of the problem was I started out with unsupported Gigabit NIC card's which would not work and eventually switched to Intel Cards which worked well, but something, somewhere was wrong. Even with auto detect working fine, something somewhere was out of whack.

It only took two tries and about 2 hours to fix the OS.

The project is complete and the box will undergo reliability testing and serve as shared disk for a home made cluster that includes an HP-9000 server.

Thanks for the help.

SEP