topic Re: Worst system corruption ever experienced.. in Operating System - HP-UX

Worst system corruption ever experienced..

Marc Dijkstra — Wed, 18 Sep 2002 09:46:13 GMT

Hi all,

Before I send this disk off to HP Labs for comment, though I'd get comment from y'all.

Scenario:
MC/ServiceGuard clustered HP L1000 servers running 11.0 with a VA7100. Running Oracle 8.0.6 as a package.

On the fabulous Friday the 13th we were testing a move of the package from the ProductionA to productionB servers and the switch failed, the logfile complained that the VG's could not be deactivated.
The ProdA server was rebooted and came up with +/- 15 failures in the boot up sequence, mainly loadable modules, network and file systems -- we could not log on. The server was rebooted again, and this time it gave us a GSP level 13 software panic as soon as the root check was done.....

The server's boot disk was rebuilt -- time factor -- and I vgimported the mirrored boot disks that had failed. What I found was simply incredible... All files to do with LVM, networking and servicing had been zero's out, that is, although they were there, and their sticky bits were there, they were 0 in size. EG:
-rSxr--r-- root sys 0 lvcreate

Most of the binaries for LVM were suchly infected ( I realise that they are ELF shared executables), but I also found that some ASCII's were zero'd...
/etc/services
/etc/fstab

As well as the utmp, btmp, last, lastb etc....

All these files were zero'd at exactly 1600hrs on the 10th, but the failure was only picked up on the 13th when the files associated with the package switch were utilised (LVM, network etc.)

Could this be inode corruption? I checked syslog and all important logs and NOTHING was logged at that time. This is too involved to have been a hack, and we all know there is no such thing as a virus for HP-UX????? ;-)
I have not in the years of doing UX from 9.X seen a corruption of this magnitude, I have never had to rebuild an HP because of a corruption like this....

Comments please...!!!!!

MND

Re: Worst system corruption ever experienced..

Bill McNAMARA_1 — Wed, 18 Sep 2002 09:48:12 GMT

I've only seen this type of damage with CPU failure..

anything interesting in the PIM / HPMC (from bch)

Later,
Bill

Re: Worst system corruption ever experienced..

Pete Randall — Wed, 18 Sep 2002 09:55:22 GMT

Marc,

While I doubt this has anything to do with your problem, it does relate to the notion that there's "no such thing as a virus for (Unix)". While there may not be virii, per se, there are hacks:

http://www.mi2g.com/cgi/mi2g/press/100902.pdf

Courtesy of SANS NewsBites.

Pete

Re: Worst system corruption ever experienced..

Marc Dijkstra — Wed, 18 Sep 2002 10:04:41 GMT

Bill,

Thanks, but the server was not complaining, and it is back up and running. Will check PIM though.... good pointer.

Pete,

Again, thanks, really interesting -- did not think there was a hacker bored enough to write a UN*X virus.... although I heard about the slapper virus yesterday....

MND

Re: Worst system corruption ever experienced..

Dietmar Konermann — Wed, 18 Sep 2002 10:49:51 GMT

I don't believe that inode corruption could be a cause. Usually such corruptions trigger vxfs errors logged to dmesg/syslog.log before they get marked bad. The next fsck should stumble though...

Its's also no surprise that several LVM executables are currupted... in fact there only two different ones (one in /sbin and one in /usr/sbin) whith lots of links to it. Corrupting one means corrupting all.

To be honest, I would start intense investigation what happened on Sep 10th, 1000hrs. What persons have root access? Who logged in as root? What was done at that point in time?

I'm not excluding the possibility that a SW bug or a HW defect could cause these special symptoms, but I personally don't believe it.

Regards...
Dietmar.

Re: Worst system corruption ever experienced..

Rita C Workman — Wed, 18 Sep 2002 11:12:09 GMT

I have never seen or heard of (in my limited time on UNIX) anything like this happen, that was caused by hardware to 'specific' kinds of files.

Frankly, I lean towards thinking this was done by 'someone' not 'something'. Someone who wondered what would happen if she/he took certain files and did a simple >"file" to them all.

Just a thought,
Rita

Re: Worst system corruption ever experienced..

Marc Dijkstra — Wed, 18 Sep 2002 11:14:15 GMT

Deitmar,

Thanks, I fsck'd the filesystems before I mounted them on the new disk, and there were no vxfs errors at all.

As to the logins, well, I can see from the OLDsyslog.log that there was nothing that happened there at that time, I know who was working on the server at the time and there is no chance of them doing anything dangerous as far as I am concerned, Good unix engineers. Unfortunatly utmp, btmp and all commands to do with them (last, lastb)are also zero'd, the .sh_history shows nothing untoward.

My first though was hack attack, but now I am uncertain......

I am seriously thinking of sending the whole disk to HP labs for their thoughts, it is just too wierd for my liking.

MND

Re: Worst system corruption ever experienced..

Dietmar Konermann — Wed, 18 Sep 2002 11:24:25 GMT

Do you have a complete list of all affected files?

Re: Worst system corruption ever experienced..

Jeff Schussele — Wed, 18 Sep 2002 11:24:56 GMT

Hi Marc,

It's suspicious that the utmp & btmp are zeroed. What about the /var/adm/wtmp file?
Any suspicious gaps in syslog.log?
Did you check dmesg prior to rebooting? Anything in there?
Check the /etc/passwd & group files for UID 0 accounts. Check for hidden "." files in key dirs.

I've had files zeroed by system crashes, but not nulled out while no system problems were encountered. They were files that were being accessed at the time the system nosedived.

Rgds,
Jeff

Re: Worst system corruption ever experienced..

Bill McNAMARA_1 — Wed, 18 Sep 2002 11:34:04 GMT

do you have any cron jobs?

Later,
Bill

Re: Worst system corruption ever experienced..

fg_1 — Wed, 18 Sep 2002 11:51:21 GMT

Marc,

Not that this is a cause but more of an investigative angle:

If your environment is a trusted system then check the accounting trail from that timeline (9/10/02 10:00AM). BTW, if your not on trusted systems, this might be the perfect opportunity to do so, it keeps nice little trails of who is doing what on your boxes.

Also have your networking engineers run an audit of the logfiles on their switch since they can trace back who was logged on to that system IP address at that time and possibly be able to see the traffic that was passed to it.

Re: Worst system corruption ever experienced..

Martin Johnson — Wed, 18 Sep 2002 14:16:51 GMT

You may want to check the .sh_history files for clues.

HTH
Marty

Re: Worst system corruption ever experienced..

Marc Dijkstra — Thu, 19 Sep 2002 07:31:48 GMT

Firstly, thanks all for the comments...
I have checked the tombstones Bill, and all seems well...

Dietmar, I am experiencing power problems at the mo, but will send a list, it is very interesting, because at 16h00 half of the files were changed and at 16h03 exactly, the other half.

The only thing that took place at those times was a (trusted) ftp....?????

Jeff, my mistake on the btmp,utmp... they are there, it is just the commands that parse these files that were zero'd -- ie last, lastb.

Bill, yeah, I checked cron -- nothing dangerous, just an oracle script to copy logfiles to the DR server.

Frank, due to the nature of the application, I cannot (unfortunately) run C2.... pity!

Marty, checked .sh_history -- clean as a whistle.... oh and by the time I got the message you already had your CROWN! Welcome to royalty sire! May you wear it with pride!!!! (10 points just for the hell of it, once I have resolution, otherwise peeps think I am happy! ;-)

MND

Re: Worst system corruption ever experienced..

Pete Randall — Thu, 19 Sep 2002 09:41:46 GMT

Hi Marc,

I only need 3112 points to make Pharaoh. Please .. !!

;^)

Pete

No points, please - just kidding.

Re: Worst system corruption ever experienced..

Stefan Farrelly — Thu, 19 Sep 2002 09:53:13 GMT

Very interesting. One thing you could try is on the mirror youve imported run an ncheck on the files which are zeroed and see if ncheck reports 0 clusters for each file, or lots of clusters. If lots of clusters note them and see if the range of clusters for all zeroed files is similar - ie. indicating an area of the disk was may have developed a bad spot and thus corrupted all the files residing on that part of the disk. Certainly possible, IMHO.

ie.

ncheck -F vxfs -S - /dev// | grep

This will list the clusters on the disk that file resides on. Repeat for lots of zeroed files and compare clusters.

Re: Worst system corruption ever experienced..

Jac Kersing — Thu, 19 Sep 2002 12:36:17 GMT

Could you check if the files affected are the targets of symbolic links? It wouldn't be the first time someone accidently copied and pasted a screen full of 'ls -l' output and as we all know running something like:
'lrwxrwxrwt 1 root sys 13 Oct 6 1999 wtmp -> /var/adm/wtmp'
creates a nice empty file...

Regards,

Jac

Re: Worst system corruption ever experienced..

Marc Dijkstra — Thu, 19 Sep 2002 13:40:53 GMT

Hey all,

Dietmar/Stefan, I have attached a shortlist of affected files (there are more) and the associated nchecks for them... Not sure what I read from this.

You will notice that on the ncheck of the /usr/bin directory the last does not exist, but this is also true for the /usr filesystem.

/etc/services is the same -- zero'd -- and so is /etc/fstab.

Thoughts?

Re: Worst system corruption ever experienced..

Stefan Farrelly — Thu, 19 Sep 2002 14:43:55 GMT

Marc,

checked your output, I was really after ncheck with the large -S option, not lowercase s. Large S gives sector numbers. Can you redo with large S as with the ncheck command in my previous post ?

Re: Worst system corruption ever experienced..

Shannon Petry — Thu, 19 Sep 2002 15:32:20 GMT

IMHO, you were broken into or someone with root access had a big wooops.

If hardware fails, then open files can be zero'd out. However, I have no clue as to what would be opening those files simultaniously. Someone syncing network services... understandable, but not likely that someone was writing fstab, services, at the same time as last, lastb, arp, etc... were all running and then the disk farted. Could happen, but very slim chance.

I would look more at who would be scripting something which would look at system critcal files, and gathering information.
Copying services and other system files to an archive, and at the same time running a bit of user accounting with last/lastb make more sense in this case than hardware, especially since it is back up and running.
If that is not possible though, then look for internal/external hacks and attacks. While you say you cant run C2, can you turn off services like NFS, NIS, FTP, Telnet, rlp??? Also, I'd really watch that 2nd server for similar activity.

Last thought on this as to another possible cause.
One time, I accidentally copied a Solaris intel binary to a Sparc system. It tried to run the binary, and did whack out the system. anyone with root access playing with strange/foreign binaries?

Regards,
Shannon

Re: Worst system corruption ever experienced..

Marc Dijkstra — Fri, 20 Sep 2002 06:15:10 GMT

Good morning Stefan.

Oops, helps to do the man ncheck_vxfs...... and to read the post correctly innit???? Missed the -

Attached the ncheck for /sbin* and /usr/*bin*, hope this makes more sense.

Thanks all for your continued interest and help with this problem.

I am still not convinced about the hack. The environment is just too clean.

MND