greatest blunders

[U.SivaKumar_2]

Benoit , apologies to you, I meant 9 points to you . This Scroll problem .......

Post again please

[Dave Johnson_1]

Hey Bill,
When I reinstalled the OS from the make_recovery tape it wiped out the scirpt I wrote and the item on the cron. There is no evidance of what happened or who was responsible. I did however go straight to my boss to confess and take the blame. That above all is probably the strongest reason next to being able to recover at least some of the data why I was not terminated for it.
Did I mention in the first post this happened Feb of 2002????

[Tom Jackson]

Boy this is tough. But here's the worst:

I was supporting a group of 600 developers. We were developing plant floor systems for one of the big three automakers. Our platform was DEC VMS. I had a VT terminal with several sessions running. I also had a footer on my VT terminal that told me my current directory location. Well, I thought I was in one session, in a particular directory, but noooooo. I was in the wrong session and deleted a days worht of work for many developers. Oops. The bright side is that I found some errors in the backup procvedures and was given the opportunity to fix them.

To

[Chris Vail]

The worst???? Which of the last 19 years in
Unix??? Only the really truly horrible ones in the last few years stand out.

remsh $DESTHOST;reboot

This was probably the most embarrassing semicolon in 19 years of systems administration. We'd already one system that was completely, totally hosed, and then had two.

[Simon Hargrave]

1. On a live Sun Enterprise server, you turn the key one way for maintenance, and one way for off. I wanted to turn it to maintenance but wasn't sure which way to turn it. Guess which way I chose...

2. On an XP512 I accidentally business-copied a new LUN over the top of a live LUN, because I put the wrong LUN ID in!!! Luckily the live datas backup had finished a full 3 minutes earlier...phew!

3. I can't take credit for this one my ex-boss did it, but I had to include it. On Solaris he added a filesystem in the vfstab file, but put the wrong device in the raw-device field. Concequently all backups backed up the wrong device, so when the data got trashed and required restoring, it...um...didn't exist on tape! Luckily for him he'd left the company 2 months before and I was left to explain what a halfwit he was ;)

[Darrell Allen]

Once upon a time, I reset the date within a DCE cell running DFS - on all but one server. Trust me, you don't want to do that.

I had a difficult time explaining why online banking stopped working for the large bank where I worked.

Darrell

[Dave Chamberlin]

I have stepped in TAR on a couple occasions. I Moved a tar file from production box to development box but I had tarred it with an absolute path. When I untarred it - it overwrote the existing directory - destroying all the developers updates! I have also been burned by the fact that xvf and cvf are very close on the keyboard - so my command to extract a tar came out once as tar -cvf - which of course erased the tar file.
Only other bad blunder was doing an lvreduce on a mounted file system - thought I was recovering space without affecting the other files on the volume. Luckily - they were backup up...

[Martin Johnson]

We have the HPUX console ports connected to a terminal server so you can "borrow console" to gain console access of a system.

One day I did a "borrow console" to a hung system and proceeded to do a "^B" and a "RS" at the "CM>" prompt. Unfortunately, I was working from the console of a production machine. Brought down the wrong machine.

Marty

[Martin Johnson]

One of my coworkers decided to set up a pseudo root (UID=0) account for himself. He used useradd to create the account and made / his home directory. He was unaware that useradd does a "chown -R" to the home directory. So he became the owner of all files on the system. This was a pop3 mail server system, and the mail services did not like the change.

My coworker left for the day, leaving me with angry VPs looking over my shoulder demanding to know when email services will be back.

Marty

[fg_1]

Greatest GAFF: Taking the word of someone who I thought knew what they were doing and had taken the proper precautions to ensure a recovery method for a rebuild of filesystems,

to make a long story short, no backup, no make_recovery, and then rebuilt filesystems. Data lost and had to rebuild. Recovered most of the data except for previous 24hrs.

MORAL of the story: Always have backups and make_recovery tapes done.

[Richard Darling]

When I upgraded from 10.20 to 11.0 I finished the system installed, and then used cpio to copy my user applications. One of the vendors had originally had their app installed in /usr (before my time), and I copied the app up one directory and wiped out /usr. By the way, I didn???t back up the installation before the cpio copy. It was a Friday night and I wanted to get out...figured I could backup after getting the apps copied over...learnt an important lesson regarding backups that night...
RD

[Belinda Dermody]

writing a script to chmod -R to r/w for the world on a dir. Not doing a check to see if I was in the proper directory and all of a sudden my bin directory files were all 666. Lucky enough I had multiple windows and it hadn't gotten to the sbin directory yet. Had a few inquiries why certain commands wouldnt work before I got it all back correctly. From then on, I do $? and check the return status before I issue any remove or chmod commands.

[Ian Kidd_1]

I was going to vi a script that performs a cold-backup of an oracle database. Since we prefer not to be root all the time, we use sudo.

So I typed, "sudo", but then was interrupted by someone. I then typed the name of the script when that person left. Nothing appeared on the screen immediately, so I got a coffee.

When I came back, I saw " sudo {script}" and realized - 1 minute the DBAs started screaming that their database was down - that I started a cold backup in the middle of a production day.

[Chris Wong]

export HISTFILE=$HOME/.profile

- chris

[john korterman]

Most of my blunders have already been well covered by other members, although I must admit that I may not have managed them as well as others. However, it made me think of a reception I attended many years ago. I had just started to work for the company that held the reception, whose purpose was to honour a life-long employee now to be retired. The usual speeches about how valuable this employee had been to the company were made, etc. etc. Around where I stood people were talking respectfully about the main character of the event; he was especially well-known for never to have made an error in his long career - it should be mentioned that the person in question had actually managed a number of promotions. I was particularly impressed by my elder colleagues talking about the person, who apparently was not capable of making mistakes - really a person to be held with reverence. Later the same day when I told my boss about it, he's only comment was:" That's right, he never did a day's work in his life!"
I will leave the morale of the story up to you.

[George_Dodds]

Giving up forklift driving and going into IT ;)

[H.Merijn Brand (procura]

Thanks for starting this thread. The correct conclusion might be drawn that we are all human :)

Now we can have a laugh, and learn at the same time, hoping we are not to be the donkeys that hit their head to the same stone twice.

Enjoy, have FUN!

[F. X. de Montgolfier]

Not mines, but some things I saw the result on clients' systems...

remove localhot from /etc/hosts ;-)

ln -fs [full path of the current dir]/foo foo

find / tmp -exec rm {} \;

rm /etc/password

# ll bar
lrwx------ 1 root sys 3 Dec 6 11:50 bar -> foo
# ln -fs bar foo

Technical support can be fun sometimes ;-P

Francois-Xavier

[Duncan Edmonstone]

My worst two:

Installing a server in a major call centre of a US bank...

I built the OS as required by our apps team in the US, and following our build standards put the system into trusted mode.

They installed the app, and realised they'd forgotten to ask me to put the system into NIS (system could be used by any of the call centre reps in over 40 call centers - a total of 15,000 NIS based accounts!) It's the middle of the night in the UK, so the apps team get a US admin to set up the system as a NIS client. (yes it shouldn't work when the box is trusted, but it does!)

Next day, the apps team is complaining about some stuff not working - can I take the system out of trusted mode so we can discount that? Sure course I can - I run tsconvert and wait.... and wait.... and wait.... hmmm - this usually takes about 30 seconds - what gives?

Try to open another window to check whats happening - can't log in as root, the password that worked two minutes ago no longer works!

Next root file system full messages start to scroll up the screen!

It turns out that tsconvert is busy taking ALL the NIS accounts and putting them in the /etc/passwd file (yes all 15,000 of them) and guess what? There's a root account in NIS!

All I can say is thank god for good backups!

The other one was a typical junior admin mistake which comes from not understanding shell file name generation fully:

A user can't log in, I go take a look at his home directory and note the permissions on his .profile are incorrect. I also note that the other '.' files are incorrect, so I do this:

cd /home/user
chmod 400 .*

I call the user and tell him to try again - he says he still can't log in! Huh?

So I go back and carry on looking for the problem, but before I know it the phone is ringing off the hook! No-one can log in now!

And then it dawns on me

I type the following:

cd /home/user
echo .*

and that returns (of course)

. .. .cshrc .exrc .login .profile .sh_history

Oops I didn't just change the permissions on the users '.' files - I also changed the permissions on the users directory, and (crucially!) the users parent directory /home!

These days I always use echo to check my file name pattern matching logic when doing this kind of thing...

We live and learn


Duncan

[Systeemingenieurs Infoc]

is it a coincidence that we have almost 2 times as much blunders as achievements ;-) ?

[Michael Elleby III_1]

Wow, this one is easy-

First job ever as an admin, asked to copy some directories from one lv to another, can't remember the commands I issued, but wound up doing an mv on the entire /etc dir..

As I looked out the window and saw my job flying away, a Sr. Admin had compassion on me and walked me through fixing it step by step..

Mike-

[Vincent Fleming]

I have been way too fortunate not to have really blundered all that bad (I've mostly done development), but one I've seen was a real good one...

The "security auditor", who apparently knew absolutely nothing about UNIX, was reviewing our development system, and decided that /tmp having world read/write permissions was not a good thing for security - so, in the middle of the day, he chmod 744 /tmp ... suddenly, 200+ developers (including myself) on the machine (it was a *very* large machine back in 1990) were unable to save their editor sessions!

So, of course, I use the "wall" command to point our their error so they can fix it quickly and I can save my 2+ hours of edits:

$ wall
who's the moron who changed the permissions on /tmp????
.
$

The funny thing was that I was the one they escorted out of the building that day...

The hazards of being a contractor and publically humiliating an employee...

[Jerry Jordak]

This one wasn't my fault, but is still funny.

One time, we had to add disk space to one of our servers. My manager at time also was in charge of the EMC disk environment, so he allocated an extra disk to the server. I configured the disk into the OS, did a pvcreate on it, and proceeded to add it to the volume group, extend the filesystem, etc...

At about that same time, another one of our servers started going absolutely nuts. It turns out that he accidentally gave me a disk that was already allocated that other system. That drive had held the binaries for that server's application. Oops...

[Tom Danzig]

As root:

find / -u 201 -chown dansmith

Did this afeter changing a user ID to another number. Should have user "-user" and not -u (I had usermod on my mind). System gladly ignored the -u and started changing all files to user dansmith (/etc/passwd, /etc/hosts, etc). Needless to say, system was hosed.

Was able to recover fine from make_recovery tape. Fortunately this was also a test box and not production.

Oh well ... live and learn! Mistakes are only bad if you don't learn from them.

[Mark Fenton]

Back in '92 on a NIS network, meant to wipe out a particular user's directory, but was one level up from same when issued rm -r *. Took three hours to back up all home directories on network....

Last year, I discovered that new is not necessarily better. Updating Db software I blithly stopped the Db, copied new software in, and restarted. Users couldn't get any processing done that day -- seems that there was a conversion program that was *supposed* to run that didn't. But that wasn't the blunder -- the blunder is that the most recent backup had been two days previous, so all the previous day's processing was gone... (and that had been an overtime day, too!)