topic Re: Linux server hang in Operating System - Linux

Linux server hang

Alex Lavrov. — Tue, 11 Oct 2005 09:02:33 GMT

Hello,

I'm facing a strange problem with Linux server. It's RedHat 7 (yep, old, nothing I can do about it), and sometimes, several times a week, I find it stuck at the morning.
No new connections can be opened, cann't login with terminal either. The database on it still running and and users can work with it.

Beside checking memory and file systems status at the moment of the problem, what else can be checked?

Thanx.
Alex.

Re: Linux server hang

Rick Garland — Tue, 11 Oct 2005 09:05:57 GMT

Any NFS mounts it is exporting? Or trying to mount?

Any nightly processes that are hanging? Check the cron jons.

If you go to another virtual terminal on the system (the ALT-CTRL-F[1-7] key sequence) is the system accepting logins?

Re: Linux server hang

Gopi Sekar — Tue, 11 Oct 2005 09:22:35 GMT

most likely some jobs are running in the night which hangs the server (especially network related process). Are you doing any network backup, if so stop them.

Next check for any updated NIC driver for your NIC card, if so use it.

Gopi

Re: Linux server hang

Alex Lavrov. — Tue, 11 Oct 2005 10:28:33 GMT

I see that every night at 1:57 logging stops and it starts with a syslog restart at the noon (server restarted by local admins).

I don't see any job near that time in crontabs, so probably it's done from outside (that's why it's 3 minutes before 2am, at remote server it's probably exactly 2am) or it's done inside application (oracle jobs etc ...)

I put a script to monitor the systems state every 5 minutes and I hope at the next hang it will help me.

Thanx all for the help.

Re: Linux server hang

Steven E. Protter — Tue, 11 Oct 2005 15:56:26 GMT

You may have a problem with network gateway's and stuff external to this box.

If networking is interupted. Red Hat 7 could hang.

The fact it happens the same the same time means that it could be some external factor.

When you check cron jobs, see what kind of jobs interact with the machine.

Also, check the maillog html and other logs for events that could be triggering the problem.

If the kernel has never been updated, the original RH 7.x kernel did that to me, I solved it with up2date.

SEP

Re: Linux server hang

Laurent Laperrousaz — Mon, 17 Oct 2005 04:56:46 GMT

I had the same problem an an RH 7.3 server. It was due to "reaching the maximum opened fd on the system" .
I could still login at the console with root but no other user could.
The only solution was then to shutdown/reboot the server.
We could not identify the origin of the excessive fd but it seemed related to an NFS or SAMBA share.

We do not use this server any more, but I will restart it soon to check this issue

regards

Laurent

Re: Linux server hang

Alex Lavrov. — Mon, 17 Oct 2005 05:33:37 GMT

I thought about the same reason, but as I know, I can open FD as long as there is memory for this, and I didn't see memory problems and I can't login with the console too.

I think only the kernel upgrade will help.

Alex.

Re: Linux server hang

KristofH — Sun, 23 Oct 2005 12:21:53 GMT

We had a similar issue with an ML350G3 running Red Hat 8. I never found out what caused it, but after 'disabling' the HP agents, the server wouldn't block anymore.

Now the server has been reinstalled with Debian, so the problem's gone..

Cheers..

Re: Linux server hang

Alex Lavrov. — Sun, 23 Oct 2005 13:33:44 GMT

Thanx, but the server runs on IBM hardware :)

Alex.

Re: Linux server hang

Steven E. Protter — Sun, 23 Oct 2005 16:39:47 GMT

I'd look into a systemic problem.

I'd think power supply but not the same time every day. I'd look for some kind up upload or something that is perhaps filling a filesystem.

I'd also wonder about the integrity of the rpm database and the physical hardware. Perhaps open tail -f /var/log/messages on the console and physically inspect it when it freezes.

Did the new kernel help?

SEP