Linux server hang

[Alex Lavrov.]

Hello,

I'm facing a strange problem with Linux server. It's RedHat 7 (yep, old, nothing I can do about it), and sometimes, several times a week, I find it stuck at the morning.
No new connections can be opened, cann't login with terminal either. The database on it still running and and users can work with it.

Beside checking memory and file systems status at the moment of the problem, what else can be checked?

Thanx.
Alex.

[Rick Garland]

Any NFS mounts it is exporting? Or trying to mount?

Any nightly processes that are hanging? Check the cron jons.

If you go to another virtual terminal on the system (the ALT-CTRL-F[1-7] key sequence) is the system accepting logins?

[Gopi Sekar]

most likely some jobs are running in the night which hangs the server (especially network related process). Are you doing any network backup, if so stop them.

Next check for any updated NIC driver for your NIC card, if so use it.

Gopi

[Alex Lavrov.]

I see that every night at 1:57 logging stops and it starts with a syslog restart at the noon (server restarted by local admins).

I don't see any job near that time in crontabs, so probably it's done from outside (that's why it's 3 minutes before 2am, at remote server it's probably exactly 2am) or it's done inside application (oracle jobs etc ...)

I put a script to monitor the systems state every 5 minutes and I hope at the next hang it will help me.

Thanx all for the help.

[Steven E. Protter]

You may have a problem with network gateway's and stuff external to this box.

If networking is interupted. Red Hat 7 could hang.

The fact it happens the same the same time means that it could be some external factor.

When you check cron jobs, see what kind of jobs interact with the machine.

Also, check the maillog html and other logs for events that could be triggering the problem.

If the kernel has never been updated, the original RH 7.x kernel did that to me, I solved it with up2date.

SEP

[Laurent Laperrousaz]

I had the same problem an an RH 7.3 server. It was due to "reaching the maximum opened fd on the system" .
I could still login at the console with root but no other user could.
The only solution was then to shutdown/reboot the server.
We could not identify the origin of the excessive fd but it seemed related to an NFS or SAMBA share.

We do not use this server any more, but I will restart it soon to check this issue

regards

Laurent

[Alex Lavrov.]

I thought about the same reason, but as I know, I can open FD as long as there is memory for this, and I didn't see memory problems and I can't login with the console too.

I think only the kernel upgrade will help.


Alex.

[KristofH]

We had a similar issue with an ML350G3 running Red Hat 8. I never found out what caused it, but after 'disabling' the HP agents, the server wouldn't block anymore.

Now the server has been reinstalled with Debian, so the problem's gone..

Cheers..

[Alex Lavrov.]

Thanx, but the server runs on IBM hardware :)

Alex.

[Steven E. Protter]

I'd look into a systemic problem.

I'd think power supply but not the same time every day. I'd look for some kind up upload or something that is perhaps filling a filesystem.

I'd also wonder about the integrity of the rpm database and the physical hardware. Perhaps open tail -f /var/log/messages on the console and physically inspect it when it freezes.

Did the new kernel help?

SEP