greatest blunders

[Gene Kornacki_3]

Mine had to be while looking at the Dev DB Server console and seeing that it was hung I flipped the switch on the Production DB Server and stood there wondering why the console didn't go blank.

[Massimo Bianchi]

Hi,
i had to change the netmask to 25 server, mixed hpux and windows.

rebooted, checked all -> O.K.
It was sunday

On mondays customer complains that cluster where doing strage things... some saw the propres nfs share, some other not.

I spent the evening rebuilding cluster configs, because it stores the netmask value in the binary file, damn !

Massimo

[Fragon]

My root's home directory is set to /.
When I'm a beginner I get none root access to the system. And there is chance that the system administrator went away form his PC for a monent and a telnet session still on with root login. I want a user with root permission so I make:
#useradd -d / tmproot
And this make system crash(that's really unexpected).
It's under 10.20.

Another:
#cd /tmp
#ls
...
mfg1
...
I want to remove this /tmp/mfg1, so I type:
#rm -r /mfg1
OK, /mfg1 is the ERP DB directory...

[Paula J Frazer-Campbell]

Scene

Two K class server side by side logged into backup server telnet to live to check a setting forgot to exit and wished to reboot backup :-

shutdown -r now

Many cries from outside of the server room as the live system with 650+ users trundled around on a 6 way 8 gig K class reboot - this is measuded in days not mins.



Paula

[Basheer_2]

As a New HP Admin,
1) Didn't create Backups, ignites or mirros for 1 year.

2) Learnt after system crashed and now i do regular backups/ignites

3)In a sql live database, used update clause without where condition.

Basheer

[Tim Sanko]

"Could you give me a hand, this should only take a minute" San guy to HP admin. "We won't change a thing with your HPs."

Young SAN wizard removes a cable,
and increases bend slightly while tidying up. Openview starts sending pages servers start crashing. Proiduction servers can't find 150 volume groups...l

San guy says "I didn't do anything."

HP's lose disk, tapedrives through SAN.

We rediscover all of above. OS rebuild re-import volume groups (this is not a quick response type fix) ioscan, insf. recover from backups as DB is hosed big time...

We are back to running. it is 42 hours from the young SAN wizard "I am just straightening up your mess..."

Can I get paid overtime for this.? No.

[doug mielke]

I bumped the emergency power off / fire alarm button in a client site's computer room.

and learned that computer people can run as fast as any atheletes.

[doug mielke]

..and I should add, that the operators were running out, while the admins were running in.

[Q4you]

Few years ago, was working on an 10.20 to 11.0 upgrade. We had 8 x 4G disks for primary(4) and mirror(4) in hot swappable jBOD. OS upgarde left system unbootable/panic. The mirror disks were taken out before upgarde for back-out were kept on top the K-class system ( without labels)

Guess, what ?! Upon OS upgrade failure ( paniced situation), inserted disks in the wrong order...had taken ignite just before the upgrade but there were 2 more unlabeled tapes suddenly appreared on top of the server.

Leanred first rule of SysAdmin : never insert or disconnect anything w/o labels !!
Learned second rule of Sysadmin : never make any change w/o good/labeled ignite !!

[W.C. Epperson]

I have sprinklers in my computer room.

[Zafar A. Mohammed_1]

My patience break-out whenever I type Escape in MS-Windows environment, because I am very much used to vi editor. Even most of the time I lost my typing messages here in the forum. Now, I am dependent on gVim.


Zafar

[Cecil Funderburk_1]

my greatest blunder was getting 2 HP-UX boxes without doing the research in OS for the boxes and not getting the codekey for the software throwing good money out the window. not getting a book and reading what I was really getting into with HP-UX. I did learn something that I can load it, configure it, and get it to connect to the internet with out a web browser so I guess all is not lost yet, E-bay for these 2 headachs LOL .

Cecil

[Tim Adamson_1]

I had a HP Engineer on site to install a heap of hardware in a machine. I shutdown and halted the server, then proceeded to power off the entire cabinet. Pity I was 4 feet out and powered off production cabinet.

My excuse - damned lowsy power, we really should have a UPS :p

[hp_user_1]

One time started a reverse BCV sync on database volumes on a production EMC frame. That caused an 8 hour outage...

[John Clogg_1]

We had a 996 (HP 3000 equivalent of a T-520). I was planning to add some hardware, and went into the computer room to look into the back of the cabinet to confirm the number and locations of available slots. When I was closing the door, I was having some trouble, because some cables were in the way. While trying to push the cables out of the way, I hit the main power breaker with my toe, and powered down our production box in the middle of the day!

I guess I'm not the only one to have this happen, because when we later upgraded to a 997 (T-600), the upgrade kit included a little plastic guard to place over the switch.

[Trond Haugen]

This isn't mine but I think it's a "good" one.
Back in the days when open system was a hot thing I heard about a system administrator that did "chmod -R 777 /usr" because he wanted an open system.
(Hint: SUID?)

Regards,
Trond

[Isralyn Manalac_1]

There was a big scheduled change. I felt ill a couple of hours before the change. Went ahead with it anyways. I decided to prepare a script to automate the filesystem extensions (we still didn't have OnlineJFS license back then). Ran the script without checking. Guess what??? Instead of using extendfs, I used newfs! And it was a 100GB filesystem containing production databases!

Regards,

Isralyn

[Stephen Keane]

Two instances spring to mind

1) We were told that the painters/decorators were coming in over the weekend and that we should remove posters etc from the walls so that they could paint them. Someone (not me) came up the bright idea that we should also remove the whiteboard that was screwed to the wall so they could paint behind it.

I watched (more out of boredom than anything else) as two colleagues proceeded to unscrew the bottom right screw, then the bottom left screw and finally the top right screw which were holding the whiteboard to the wall. At this point, the whiteboard was held on to the wall by one screw, which it proceded to pivot around as gravity took over.

The whiteboard did a fair impression of a guillotine blade as it sliced the emergency power kill switch clean off the wall, leaving a couple of wires sticking out from a hole in the plaster. Luckily the lights weren't connected to the kill switch, just the remainder of the computer room equipment.

2) A new colleague wrote a script to tidy up log files on a production machine. They of course didn't test it before installing it. They crucial part of the script was along the lines of the following

LOG_DIR="/blah/blah/..."
cd $LOGDIR
rm -fr *.*

Needless to say, it was run as root user and root's home directory was naively set as '/'.

[Jan van den Ende]

Well,

I also had my "moment of learning".

It happened in 1990, but it still makes me feel humble, and ashamed.

In those days, I was used to Vaxen of the 8*** series and of the 11/*** series.

I was hired to implement a new cluster and train the staff that was to work with it.

The cluster was up & running smoothly, hosting some 50+ developers.

I started the operator training with the hardware tour.

One of the demonstrations was switching the system controls from "Locked" to "Unlocked"

Now, _THOSE_ systems were 6*** series.

... and on the 6**** series the position of the switch I thought knew as "Unlocked" was labeled "Init" .... in front of the trainees.
Makes one really blush :-(

But to this day I maintain, that if, eg, they swap the pedals in a new make of car, no matter how well documented and instructed, the potential for this type of error will lead to that car not being allowed on the road.

Proost.

Have one on me.

jpe

[renarios]

Well,

This morning: Trusting my colleagues to care of backup and restore. Result: 900 lines of (new) backup script down the drain. As we say in Holland: "It keeps you off the streets..."

Cheers
Renarios

[Devesh Pant_1]

I have had an ocassion when I removed a LUN that had been allocated to a production server
That caused a havoc
Had to restore 2 TB of oracle files from tapes

thanks
Devesh

[Anupam Anshu_1]

I was new into system admin, built a new Libc, moved libc.2 and logged out. Hence couldn't login again.

At last single user mode helped me to put the libc back in place.

Other things were:

# ls /abc/xyz
wanted to remove it,
# rm -rf /abc
It deleted all the directories inside /abc, and only /abc/xyz was un-wanted :-(

Cheers,

Anupam Anshu

[Robert Bennett_3]

disk failure on an L.

Simple enough - call in to get a replacement and get on to other things. Replacement disk shows up and I just walk over to L and look at disks and pull out the only disk that isn't lit up. Lo and behold I have another failed disk - I actually didn't realize I had shot myself in the foot for an hour or so - multi-tasking at it's best.

The kicker is - the L's have the hardware address right on the slot for the disks.

So now I always double-check before yanking unlit disks.

[TwoProc]

OK,

High on the stupid chart. Many moons ago, I was a C systems programmer - started a new consulting gig, and I was coding a new module into a very large program which didn't have a standard template .h file for starting all modules. So, I started off with a few quick includes ...
#include
#include
#include
...
...
Then I needed to start my #defines... first of course - define my basic TRUE and FALSE, but was rapidly typing away as I needed to get this done quickly and show the new team what I could do quickly... so as to make a good impression on the senior programmers and management...

#define TRUE 1L
(and then I had a brain fart, and I didn't automagically type 0L as usual)
#define FALSE 2L

Well ... 2 comes after 1 when you're in a hurry right ... right ??? 1... 2... 3...

Anyways, it took me a day to find out why I broke the new build... but the NEW GUY was definitely suspect!!! And - THEY WERE RIGHT!!!

It was a while before I could laugh that one off.

:-)

[Murat SULUHAN]

Hi

crontab -r :=)

approximately 30 lines

I want to edit my crontab but r and e letters are too close in the keyboard :P anyway I listed my crontab with -l option before the -r; therefore crontab -l is good backup solution :=)