greatest blunders

[Sandman!]

Here's my two cents...

Confusing a production system with a test system and confidently rebooting it during the middle of the day :)

cheers!

[Gavin Clarke]

I did that too.

I had a telnet session running from the console of a test server to the production server.

I ran shutdown -h now, then started to wonder why messages were coming up on the production server console!

[mirco_1]

a cofee break!
M.

[Borislav Perkov]

Hi,

In '90 on ULTRIX trying to delete directories and files in /tmp

#pwd
/tmp/.oracle
#cd
#rm -r *

The brightside is that it was not production server.
Regards.

[Doug Burton]

Back in 92 I thought I was on a test box but was on a prod box. Bounced the beast which made the other diskless workstations hanging off of it to go down too..... I bought lots of pizza the next day to pay my way out of that one. Funny how engineers can be just as hungry as IT folks. When bellies are full, all can be forgiven.

[Steve Lewis]

1. Confused test system window with live and typed in a rush order for two RAF Tornado jet pipes to be sent to Saxa Vord in the Shetland Isles. (For those that don't know, Saxa Vord is an old radar station and I am not sure it even has a runway, let alone any Tornados.

2. Not one of mine but worth a mention so that nobody else does it. In this company a common abbreviation for Image File Server was IFS (warning bells ringing already) The log files for all these were held in a directory for each server under /. This person was told to write a cron job to periodically clear down these directories, hence the terrible command:
find /$IFS -type f -mtime +30 -exec rm...

Doh! NEVER try to use IFS as a variable.

[Kent Ostby]

1) Open telnet window to test machine
Telnet as root to production machine
Wander off for a while to have a Pepsi
Come back and decide to reboot the test machine.

2) Type command to back up personal files onto tape but dont hit return.
Go to machine and load blank tape.
Stop at someone's desk to talk.

Hit return on my command to back up files

3) Had a "friend" do this one time:
Log into machine.
Decide to delete files including ..
Enter command:
rm -r * .*
reinstall system

[Mel Burslan]

proof that longer workdays are hazardous to the health of not one but many workers and managers:

Scenario : Large manufacturer of retail goods, which does about seven digit dollars worth of transactions via EDI, taking orders from customers.

8 PM of a long working day, EDI developer calls the sysadmin and asks the data on dev machine to be deleted and copied over from the live production data (this is 2 hours before the daily backup no less)

Sysadmin open 2 telnet sessions to two machines. Machine names differ by one letter in the middle of 8 character sequence, asin a "d" for dev vs. a "p" for production.

sysadmin clicks on the windows of presumed dev machine and confidently issues the commands:

cd /EDI/data/today
rm -r *

and realizes the terminal window he was in was the production one 45 seconds into running the rm command.

Magical disappearing act of 22 hours worth of real life transaction data. Needless to say it was a sleepless night for us, sysadmins, trying to salvage whatever we could and a long week for managers hashing out how to explain this blunder to the customers.

There are two types of sysadmins: One who lost data and one who will.

[Rick Garland]

DBAs were in process of moving database to another directory. Another SysAdm created LV with similar name to original.

After data was copied to new LV I was asked to removed the old LV.

As stated, the LVs had similar names, LV1.new and LV1.old.

I removed the wrong one. Crashed the database.

The restore brought back everything but 2 min of data.

[Michael Steele_2]

I just wanted to bring back one of my favorite threads. I came across it in a search of my name, reread my blunder and noticed that the thread hadn't been updated for a long time.

Three cheers for U. SivaKumar for starting this.

PS My latest blunder is more of a personal one, getting divorced and having to pay alimony and child support after the divorce. (not the happy camper that I use to be)

[Asif Sharif]

Once working with newly production machines, attached storage VA7410 reset button pressed mistakenly while setting front bezel.
OOOPSSSS

Storage reseted.

[spex]

First day on the job: 'crontab -r' instead of 'crontab -e' on our production box. There was no backup of root's crontab, so I had to reverse engineer it from cronlog entries.

[Rick Garland]

Similar to spex.

Notice the 'r' key is right next to the 'e' key

Want to edit the crontab file - didn't move my finger over far enough.

[TKeller]

Here's my two biggest errors, one HPUX and one RH related.

For the HPUX one, I was busy restoring a backup tape that I "thought" was for our production server. About halfway through, the console said I ran out of space in /usr and hence the restore failed. Looking around I start noticing that I've got all these new files. Come to find out, I was restoring Solaris binaries on an HPUX 11.11 machine. Whoops.

For the RH one, blew out my /etc/password file. Apparently, some developer had a sense of humor because the next root command I did came back with "How did you log in?" and that's when I figured out what bonehead mistake I had done.

I'm sure I'll do some more spectacular stuff soon, hehe...

[Frank de Vries]

This blunder was simple, but effective.

I was in charge as a senior sys admin
In 1999.
I had Created and configured datbases
on hpux with the name corep for production
server and cored and coret for development and test server. All nicely setup with
Mc/Serviguard.

In the nightshift there was a file system
full on the production server. One of the operators launched a cleanup
command find . -name core* -exec rm {} \;

The production database whose datafiles and controlfiles all started with corep where gone !!!

When Someone complaint that they could not access the database, I was called out in the
middle of the night, I could not understand
why the database had gone ??

I restored it without wondering why.

The next day after some research I figured
out the whole story.

I never ever create anyting with the name core, and secondly I make it an extra point to document properly on actions to take
and not to take when cleaning up.

[Fabian Briseño]

Hello.
I once Using SAM selected a bunch of disk and deleted them only to find out that they were in use by our SAN on a production server.

I work the nigth shift so I had to stay like 20 hours working on recovering the production system. needless to say that my bosses were not very happy with my cute little blunder.

[Shrikant Lavhate]

Hi all,

It happens when I was new to HPUX and didn't know much about system files. I just edited the /etc/lvmtab file manually during performing LVM operations of vgchange on Test server and blew up the server..lol..
I still wonder what i have done so weired that it screwed up the server. Because after editing i restarted server once and all happened very well but next reboot it started memory core dump. I then tried vgscan to restore file but it didnt worked out.

Finally, we land up with reinstalling whole OS from scratch.

[Hasan Atasoy]

this is from my sql prommmer friend ,he wrote a sql file containing

delete payments ;
where payment_no=123456;
commit;


and call this sql file

sqlplus user/pqd @delete.sql


Hasan.

[Lexxx]

My first blunder... My boss asked me to shutdown the machine...

yeah right I turnoff the power :-)

[Aneesh Mohan]

Hi Siv,

The greatest blunder happened frm myside is created a lvol named /dev/vg00/lvol11 and did newfs on /dev/vg00/rlvol1 :)


The second greated blunder from my side is corrupted the root filesystem by the below 2 steps :)


#lvchange -C n /dev/vg00/lvol03

#lvextend -L 100 /dev/vg00/lvol03


Cheers,
Aneesh