greatest blunders

[Yogeeraj_1]

hi,

Here is mine!

Running an ORACLE Database on NOARCHIVELOG and deleting a datafile accidentally!

Fortunately, we had not gone live on that new server. Anyway, that was hours wasted.

Best Regards
Yogeeraj

[U.SivaKumar_2]

Interesting...

[donne007]

A collegue of Mine deleted some files in /etc (not knowing What he had done)and removing some permissions for Root.. the system Was completely screwed up.. even could'nt recover/repair from a backup kernel.. so i had to install all the way from the scratch inspite of spending a day trblshooting all the stuff in vain..

cheers
Asif

[Russell Gould]

Having executed 'crontab -l > /tmp/crontab.root' on one of our servers, I quickly pasted it out onto each of our other servers quickly hitting enter.

When I got to our main production box, instead of crontab -l, out came 'shutdown -r 0' quickly followed by enter !!

I'd previously issued a shutdown -r 0 on a workstation.

Thank god, I was not in / so it did not allow me to do so ! 300 users nearly kicked off. I went up the pub soon afterward for a stiff drink and some air !!

[U.SivaKumar_2]

.

[Chakravarthi]

Written a small script to reboot a set of 20 machines, and in the input file, i gave my hostname in the first, !!!!! this should be a greatest blunder.

[KapilRaj]

Great !!!,

01. Have started learning UNIX Basics 6 years back & wanted to selectively delete each & every file in a directory... I did

rm * -i (It said "-i not found" )

instead of

rm -i *

Hope what happened ???. ls -l told me what had happened ?????.

02. While housekeeping i went to /etc &

find ./ -mtime +2 -exec rm {} \;

hooooooooooooooooo

kaps

[Keely Jackson]

My greatest blunder:

The guy who set up the live database had done it as himself rather than aa a separate dba user. He left the company and his user id was re-allocated to somebody in HR. The guy in HR subsequently left as well.

One day I decided to tidy up the system and remvoe the this user. I did this via sam, selected the option to delete all the users files thinking that nobody who was in HR could possibly own any important files.

Unfortunately I was somewhat mistaken. Of course the guy in HR now owned all the database files. The first thing I knew was when the users started to complain that the database was no longer available. I got the db back from restore but everybody had lost half a days work.

Needless to say, I now do not delete old users files but re-allocate them to a special 'leavers' user and check them all before deleting anything.

A good HP blunder.

HP were moving the live server - a K420 - between sites and the remvoal men managed to drop it down a flight of stairs. It landed on one of them who then had to be taken to hospital. Fortunately he was only bruised while the machine had a huge dent in it. Anyway, it got moved to the other site and booted up straight away with no problems. That is what I call resiliant hardware. As a precaution disks etc were changed but it is still running quite happily today.

Cheers
Keely

[Pete Randall]

Speaking of dropping things, Keely - I don't know whether this belongs here in the blunders thread or in the achievements thread.

Several years ago, I had an HP C.E. on site to replace a failed 2GB drive in a 715 workstation. Since he has to be escorted around the building, I was hanging around watching over his shoulder and trying to be helpful. While I was "helping", I figured I would get the replacement drive out so it would be ready as soon as he got the old one out. I pulled the silver anti-static bag out of the box only to discover it wasn't taped shut!!! The drive dropped to the floor, bounced twice and then lay still while the C.E. and I exchanged horrified looks. We tried installing the drive, but when we turned on the workstation, it made a sickening, grinding noise. The C.E. went back out to his car and came back with the only other drive he had with him - and I ended up with a free upgrade to a 4GB drive.

Pete

"Never look a gift horse in the mouth"

[Steven E. Protter]

In July of 2001, I did the great patch of death, (LITS, Line in the Sand Patch) PHKL_18543 on a forced reinstall.

It cost me about 200 hours and a dozen support calls, but I managed to keep the system going with no user downtime and did NOT have to reinstall the operating system.

Still today, several overlaid patch sets fail patch analysis.

My biggest, most painful blunder.

What was I thinking?

[Terrence Johnson]

Removing a dustball from the an internal tape drive of a customer's only production machine with a standard vacuum cleaner on a Friday night. I wanted to do a backup before performing an OS update but the tape drive had this huge fur ball in it (didn't get a lot of use apparently). The customer offered to let me use the office electolux so I accepted. It was on a Friday night and I wanted to save time so why bother shutting down, right? As soon as the metal vacuum nozzle got within four inches of the tape drive, the built up static electricity discharged into the drive and ZAP, down went the system. The last backup had been performed by the user a month previous. I think it only took a week for our office staff to type in a months worth of transactions...

[Paula J Frazer-Campbell]

Hi
I spent half a day trying to trace a fault on a machine- could not find anyhthing wrong - stange!!!

Went back to start, to discover I was working on the wrong server.

Paula

[John Dvorchak]

We had 8 servers on one of those monitor keyboard switches. I verified that the print server (NT) was locked up, reached over and promptly powered off the HR server which had no problems until I got there. Not all bad though, I called the help desk, asked if there was a problem with the HR server. When the help desk confirmed it had suddenly went offline, I said "No problem I am back there by it and I'll take care of it".
Whew....dodged that bullet.

[U.SivaKumar_2]

pushing to top

[Printaporn_1]

I create shell script named "reboot"
Then call the script with root w/o it path.
production server was reboot !!!!

[V. Nyga]

I was setting up the newest OS to serveral workstations with ignite using a created image.
And ever client had got his old hostname.
But at one client our CAD program didn't want to run properly. It found no tablet.
I checked the scripts serveral times - at least I called the support.
We checked the same scripts again. Nothing wrong.
Then I saw the mistake:
All clients had their hostnames written with small letters - during my ignite installation I gave the name to thie client with captal letters. So for the script it was a unknown client.

Volkmar
Never touch a running system

[Chuck J]

Apart from all else, we had two mailing lists both with the words "global" in them. One was for all the IT staff globally, the other was for every single person in the company (globally). Yes, you have already guessed what I did, but that's not all, what I sent out was IT instructions for a topic so had users emaling me back left right & centre. Lucky though I caught my mistake straight after hitting the send button and we managed to find the message in the lotus notes mailbox and remove it. In the end, the message only got sent to my site (biggest global site), but noone else in the world at the other sites got it, thanks to our removal from the mail.box

Chuck J

[Ian Lochray]

Fortunately this was on a test server that nobody else was using. I decided to remove stuff from a lost+found directory. I did a cd lost+found and ls showed directories called things like #12345. As root, I did a cd #12345 then an rm -rf *. The cd took the #12345 as a comment and put me in root's home directory of /. Bye-bye machine.

[Chuck J]

Oh no Ian, what a blunder that was. Hope you had a laugh about it later.

Chuck J

[Steven E. Protter]

A second greatest blunder.

I once managed to make my root disk unbootble, while playing around, trying to get an extra disk into vg00.

The disk itself was probably dead and in fact died several months later, while not in use.

I spent all night on the phone with HP Support, putting humpty dumpty back together again.

My Ignite tape had a bad kernel on it generated by my greatest blunder. The system file was corrupt. Turns out this machine won't do an ignite tape boot unless you unplug the fiber card, figured that out six months later.

The good news? With support through three hours of hpux -is and hpux -lm we fixed it.

No loss of data, no loss of service to users.

:-)

The lesson learned that if a disk won't joint a volume group, (vgextend I think), check the darned thing out before pulling out sledgehammer commands.

Steve

[Ian Lochray]

Chuck,
I certainly did have a laugh about it later as it was only a test machine that nobody else was using. I doubt that I wold have been quite so jovial if it had been a production server though.

[U.SivaKumar_2]

Procura , you are smart :-)
No points without matter. :-)

[Ramkumar Devanathan]

I don't know. I deleted /etc/* once. (just a few config files, eh?? nope, had to reinstall the OS);-)

And on linux, I tried kill -9 -1 as root. The entire machine froze. had to reboot it.

- ramd.

[Radhakrishnan Venkatara]

hmm,
I was new to NAS, want to configure NAS for 20 workstations which are in production environment.Configured the NAS and pushed some files from NAS to workstations and left.Suddenly got a call saying root filesystem full in all workstations and all their applications running got crashed.
I was totally clueless. booted the system in single user mode. removed all the log files still system root filesystem shows 100% usage.

then just du on /etc found that /etc/passwd file was 150 MB , it was shock.
the file pushed from NAS, had made something wrong and all of the workstations gone down.

there was another blunder.
I was configuring another root hard disk for the prodcution system.I got the down time from the customer.customer is good enough to gave me the down time at the night , but he told server should be up at any case by morning 5 o clock.I assured him it will available by 1 o clock it self.
But really i did nasty thing.there was no mirroring software available on the system. I added the pv into the root vg and did lvlnboot the system gave me error that one anothe root filesystem exsist.
then i understood, that the procedure is to create a separate vg and do.
but mean while,I have to add one hard disk on the system so we down the system and booted it .
the system didn't come up.it was giving unable configure the swap space and going to crash.
clueless how system gone down.I couldn't enter in to single user mode also.
then i got struck up the problem was with lvlnboot command i gave, which changed the swap and boot area on different disk on the same root vg.
thank system went thru recovery cd and i could do lvlnboot and brought the system at 4:45 pm
more horrible thing is customer was sitting beside me, he doesn't know that is production was totally down and one engineer is trying to recover it.

greatest blunder

tempuser::450:20:temporary user:/home/temp:/usr/bin/sh > /etc/passwd

one live setup and went out.
nobody no that passwd file was changed.I was leave for 4 days ,they recovered the system from backup.
when came back i was going thru my history and found this.

radhakrishnan

[Ollie R]

Hi,

When I was a rookie, I was on customer site. I was having problems installing some software because the root disk was full (back in the good old days before LVM).

My decision was to move some sections of the filesystem to a second internal disk and create required symbolic links.

I moved a few bits around using "cpio -p" including, sadly, the contents of "/etc". Everything worked fine until reboot.

Ever seen the message "init not found"???

Wonce I cunnot even spel Administrator - now I are it!

Ollie.