Users unexpectedly back to "login:"

[OldSchool]

Here's the situation. I've two server, A and B, and three "service centers" (1,2 and 3). The service centers connect to either of the the servers and all run the same application.

Users in centers 1 and 2 and connect to either server A or B over a WAN and run the application w/o problems.

Users in service center 3 connect to either server and run the application, but they sometimes get kicked back to the "login:" prompt while working. No error messages, core dumps, etc... They've even been in the middle of entering data when this has occurred. Getting something like 4 incidents an hour.

All users have the same .profile (its a link) and it simply runs the app and exits.

Note that service center 3 was added recently, however all of the users at that site came from the other two service centers.

All of the connections use HOSTACCESS over Windows Sockets.

It almost appears as if the user's sessions were issued a "kill -9", but they weren't (as far as I can tell).

I've talked to our network folk who assure me that there "aren't any issues". I'm not convinced that's the case, but I'm still looking.

Any other items I should check before heading back to the LAN folks?

Thanks....Scott

[Peter Godron]

Scott,
you state "Users in service center 3 connect to either server". Is done automatically or does the user decide which server they use.
Just answing, as , if it automatically (i.e.load sharing) it may have problems with DNS returning multiple addresses.

Can you see any of the 'killed' sessions on your server A and B ?

[OldSchool]

User selects server themselves, based on which warehouses they need to work with on a given day. No load balancing going on.

I'm not sure what you meant by "see any killed sessions"?

[Peter Godron]

Scott,
thanks for the quick update.
I meant things like who/last/ps -ef | grep userid etc.

So users select their server, log in, the application starts and some application sessions are then lost along with the associated Unix session and the user is returned to the login prompt.

Can you please check the syslog.log file.

[OldSchool]

OK, here we go:

1) "last" shows the users loggin in, and logged out when returned to prompt. No "still logged in". ps -ef | grep userid doesn't show anything running for the user

2) your summation is correct. note that it isn't always the same user(s) that this happens to.

3) syslog is filled with the following:
severa syslog: gethostbyaddr: != xx.yy.zz.aaa

Now the LAN folks are telling me that "we recycled some IP addresses".....

[Peter Godron]

Scott,
something along the lines of:
http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=980740

[Peter Godron]

Scott,
also, whilst sniffing around I found another thread:
http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=1041884

Perhaps adding the load of center 3 users has pushed the machines over the limit ? Although I would have expected the login to be denied in that case ?

[A. Clay Stephenson]

There are just not enough data to diagnose this yet. The lack of a core dump may mean nothing at all as it depends upon which signal handler is currently active for a given process and signal at any particular time. This does have the appearance of a network problem. I would start the application and disconnect the PC's network cable. Is the behavior on the host the same? You might also try connecting tusc to several user sessions and hope that you get lucky and are able to capture a failure. I would also check the network settings between this host and its corresponding ethernet switch port(s). Surprisingly, a mismatched duplex setting, for example, will almost work well at low i/o rates and would probably not even be noticable in a telnet session, for example --- until the traffic became heavy. You can't even depend upon the lack of a core dump or error messages to mean anything. It all depends upon how the application was written. For example, I've seen application code that blindly assumed the open() system call would succeed. The actual error message that was produced thousands of lines later in the code had nothing to do with the fundamental problem.

[A. Clay Stephenson]

Other things to examine: Are the kernel tunables identical between these 3 boxes? Patches identical? Shared libraries equivalent?

[OldSchool]

Peter,
nstels is 1023, and like you, I'd think it would reject new logins, not kick out existing sessions.

Clay,
I think you may have mis-read something..2 severs, 3 service centers, all centers use both servers, only one center has the problem. That tends to make me believe its not the app itself or either server, as then I would think all three centers would then have issues. "Feels" more network related to me (at least at present)

Never used tusc before, any hints? Also, I need to double check my HOSTACCESS config to make sure its the same as what's on the floor, before I try disconnecting the cable.

Also getting in touch w/ the HOSTACCESS folk, to see what tracing I can do there as well.

I probably won't get back to this until tomorrow.

thx

[A. Clay Stephenson]

Using tusc is straightforward although it helps if you have some UNIX C/C++ programming experience. You are essentially monitoring each and every system call a process makes. You get to see the arguments and the return value. In any event, download and swinstal;l tusc from:
http://hpux.its.tudelft.nl/hppd/hpux/Sysadmin/tusc-7.8/

Next, determine the PID of the process that you want to minitor.

Then, "tusc -p PID". Man tusc for details and more options.

[OldSchool]

Used tusc on 8 processes (4 users) located in the problem location..no errors..from anybody..

will continue tomorrow and update as reqd.

[NilesStein]

Hello OldSchool

Im sorry for not being able to answer your question but Im very curious since no one has been answering my questions do you know how do actually link hardrives together and make the Master and Slave run as one?

[OldSchool]

Niles,

I have no clue as to what you are referring. Sounds like windows to me, but....

People aren't replying because you don't seem to be placing your questions in the forums they are relevant to. for example, you asked for help writing VB Script in the Data Protector forums, when what you had was a shell script. DP is backup software, VB is an Windows-based product, and shell scripts are unix.

Not to be insulting, but you *appear* to be a novice at these things. If that is true, you are trying to learn about Unix, hit the bookstores and look at the self-training manuals for Linux (as an example). Note that there are many differ U*x's, and, as with shells, they aren't exactly the same.

also, try to avoid *hijacking* other's threads, as you have done here. I realize that you didn't intend any offense, and none was taken, but think about this: how many people will see (and respond) to your question regarding disks, when the original question concerned problems w/ broken lan connections? Please review the "Forums guidelines" thread for more information.
http://forums1.itrc.hp.com/service/forums/helptips.do?#overview

[OldSchool]

Well, no reported issues on Tuesday, one issue Wednesday (not reported until noon thursday, no known issues yesterday.

Was contacted by the lan team today, and they are going to take ownership of the issue as "... there are users at that location having problems getting to other servers as well"

So, its unresolved, but out of my hands / not my problem (for the moment at least). I'll open a new thread on this should the need arise.

Thanks to all who responded.

Scott

[OldSchool]

FYI.

This issue turned out to be a problem w/ switch at remote site dropping connections momentarily / randomly.

Switch has r&r'd, issue has not reocurred.

Again, thanks to all that offered their assistance