topic Re: Lost / partition in Operating System - Linux

Lost / partition

Neme — Tue, 17 May 2005 07:11:00 GMT

I have lots of customers (in different cities) that have linux installed (fedora core 3 , mandrake 10, 10.1, 10.2) and working. The machines stopped to work with the same error. When I try to reinitialize the system, the following message appears:

No inittab found
Enter your runlevel

And when I try to boot with runlevel 3, 0, 5, it didn't work. I boot the machine with the rescue option and fortunately my data partition (that I've created in another partition) was there. Only the root partition was lost. The /etc directory only have some files.

I mount the / partition and the /data partition. Only the / partition was compromised. The /data partion was ok. Then I installed again withot formating the data partition, restore the /etc from my backup and it goes fine.

But I don't know what's going on. Why suddenly the machines lost the inittab and / partiton? I look at the messages and nothing related to the problem appears. Once I try to repeat the problem with one machine at my office and I find the same error trying to use cups lots of times in a recursive shell, but it was only 1 time.

The hardware is different, i.e., one machine has scsi, another ide or sata, dell machines, ibm machines, out of brand machines, kernel 2.4 or 2.6.

They run only dataflex , and some use only a server in a lan environment (The users only access the server via putty on windows machines or ssh on linux machines), they don't have internet access.

The users don't have access to the server and the disk space is ok.(only 20-30% of the disk is used) so probably virus is not the problem...
In one customer, there is also a netware server using the same power supply and the server never hungs (so we discard power problems).

Re: Lost / partition

Claudio Cilloni — Tue, 17 May 2005 07:57:04 GMT

different kernel... different hardware... but (I guess) same applications and same final use of the machine.
Maybe a bug in the software they use. That's the common thing between them.

Be sure there are no applications/services/daemons/logins that runs as root user if it isn't absolutely needed.

Does dataflex (I don't know what it is) run as root or ssh user logged as root?

ciao
Claudio

Re: Lost / partition

Gopi Sekar — Tue, 17 May 2005 08:08:52 GMT

initially i thought that machines are compromised by hackers and they removed / partition also. but later u said no machines are connected on internet.

it could be problem with the application they are using, some silly bug in the application which instead of removing temporary file, simply removes / (?)

the application(dataflex) does it run as root? if so try to run it as normal user if it can run properly. Also all the users logs in to that machine as root or as normal user?, may be this could be user mistake too.

if all the machines (located in different cities and used by different people) crashes at the same time(i assume so) then its more likely the bug in the application which gets exploited only under specific condition (date/time, routine maintenance job, backup etc)

Regards,
Gopi

Re: Lost / partition

Neme — Tue, 17 May 2005 09:14:21 GMT

The dataflex utility runs under a normal user that have write access only on his home directory. The errors ocurred at diferent times and days. The most interesting thing is that we have the same environment in lots of places, with exactly the same configuration and with no errors!!
We thought that maybe the filesystem type, so we try reiser, ext2, ext3 and we did not resolve the problem. Why the data partition does not get compromised?
We created a partition for the dataflex utility and the problem continued. The / partition get lost but the /data and /dataflex partition was there!

Re: Lost / partition

Neme — Tue, 17 May 2005 09:15:30 GMT

I forgot to thank you very much for the help!!!

Re: Lost / partition

Gopi Sekar — Tue, 17 May 2005 09:25:14 GMT

are you using any backup utility or some periodic application which does system maintenance?

does any of the component in dataflex runs with root access?

how do the users power down their dataflex servers(if at all they do), i am asking this silly question because one of my customer who is from DOS background simply switch off the power to power down linux box :)

how the server is maintained? is it maintained by one of the local non-linux user or by some linux expert/system admin? are there any front end tools used for system maintenance.

Re: Lost / partition

Neme — Tue, 17 May 2005 09:54:05 GMT

We create a user to power off the system. This user automaticaly runs a halt command. We managed the system remotely. We have the same environment here and in other customers that is up more then 80 days...

Thanks!

Re: Lost / partition

Neme — Tue, 17 May 2005 09:56:36 GMT

For backup we use the tar utility that runs once in a day with cron (tar zcvf xxxx.tgz ...).

Re: Lost / partition

Gopi Sekar — Tue, 17 May 2005 10:08:47 GMT

I dont think tar can be rude enough to erase a file system :)

the file name is it always static or is it automatically decided by a script. in which case check the script closely for any malfunction.

the backup job, is it running as root user or again as dataflex user?

Gopi

Re: Lost / partition

Neme — Tue, 17 May 2005 10:15:42 GMT

the backup runs as root user, but the problem occurs usually at work hour, not at the same time that backup is running..

Re: Lost / partition

Gopi Sekar — Tue, 17 May 2005 10:24:56 GMT

i am not very sure, but it will be still worth a time to re visit the backup script to check for any potential error.

also if it happens during work hours, then some scripts or some utility used by one of the user is causing the problem.

since you have mentioned that it runs on several locations without any problem, find out the difference of daily work routine done between working location and crashed location. may be that will give some insight on to the problem.

Re: Lost / partition

Neme — Tue, 17 May 2005 12:27:31 GMT

The users don't use any special scripts. They only use putty to access the server, run a script to enter the dataflex program. The script is the same on all the instalations, the ones that are ok and the others that some times have the problem...

The users don't access the server for a shell, they access only to use the system. When they stop to use the system, the script log them out.

Re: Lost / partition

Wim Van den Wyngaert — Wed, 18 May 2005 01:08:47 GMT

The inittab file is normally read by the init process that is started with /sbin/init.

How did that get started if / isn't available ?

Wim

Re: Lost / partition

Neme — Wed, 18 May 2005 07:56:09 GMT

The / is not completely lost. Most of the files, mainly at the /etc directory that get lost. Some times the /var isn't lost and the logs do not show nothing about the error. Some times the /var is not there. It is a very silly problem.
Maybe the kernel is trying to access the swap area and write data in the wrong partition?
I can't guess why the data partition is always ok...

Re: Lost / partition

Stuart Browne — Wed, 18 May 2005 08:27:55 GMT

So you're saying that the partition is fine, and there's still a valid filesystem on it (fsck in single user mode), but the content of the filesystem gets corrupted?

Re: Lost / partition

Wim Van den Wyngaert — Wed, 18 May 2005 08:29:09 GMT

And if so, did you check lost+found ?

Re: Lost / partition

Neme — Wed, 18 May 2005 09:35:14 GMT

lost+found is always empty and the data isn't corrupted, the files disapear. If we boot with a rescue cd and mount the partition / for instance, the /etc/hosts is there, but the /etc/passwd isn't. Some files are there but lots of them are not anymore! The inittab is not there, so the message no inittab found enter runlevel appears.
This problems are causing lots of trouble, because we need to reinstall the linux os sometimes in places that are so far here.
In one case, the problem happened in a machine that has been used for more then 2 years with novell os and never have one single problem!

Re: Lost / partition

Claudio Cilloni — Wed, 18 May 2005 09:45:20 GMT

did you made any software upgrade or new installation into this machine recenty?

Re: Lost / partition

Gopi Sekar — Wed, 18 May 2005 09:54:02 GMT

is it possible to get a list of daily routine works done by the users from two different place. one where the system never crashed and the other where system crashes too often.

this will give us some insight on what is going wrong.

Re: Lost / partition

Bruce Copeland — Wed, 18 May 2005 11:03:32 GMT

Do any of the systems that have failed do regular forced filesystem checks?

I once saw something pretty similar to this on a Fedora 2 system. The problem was ultimately traced to recent changes in distributions that no longer do regular forced filesystem checks. Under certain conditions that can allow a lot of filesystem errors to accumulate.

Bruce