greatest blunders

[Dario_1]

Got a page once at 4:00AM about a filesystem almost full. Got up and started working on it without checking if someone else was in the system. I was recalling my commands using ESC-k but my supervisor was removing files in a different directory using rm -r *. The rest is history.

Regards,

Dario

[Michael Steele_2]

When I was first starting out I worked for a Telecom as an 'Application Administrator' and I sat in a small room with a half a dozen other admins and together we took calls from users as their calls escalated up from tier I support. We were tier II in a three tier organization.

A month earlier someone from tier I confused a production server with a test server and rebooted it in the middle of the day. These servers were remotely connected over a large distance so it can be confusing. Care is needed before rebooting.

The tier I culprit took a great deal of abuse for this mistake and soon became a victim of several jokes. An outage had been caused in a high availability environment which meant management, interviews, reports; It went on and on and was pretty brutal.

And I was just as brutal as anyone.

Their entire organization soon became victimize by everyone from our organization. The abuse traveled right up the management tree and all participated.

It was hilarious, for us.

Until I did the same thing a month later.

There is nothing more humbling then 2000 people all knowing who you are for the wrong reason and I have never longed for anonymity more.

Now I alway do a 'uname' or 'hostname' before a reboot, even when I'm right in front of it.

[Caesar_3]

Hello!

When i just begin as a sys admin, i learn how
to use UNIX and i learn about the remote
connection that you work on your machine but
with one command you can work on the other
so i try it and like it so one day i was
connected to our file server and i didn't
saw that i'm on him and wanted to restart
my machine i made a reboot and didn't understand why my machine is up?
Til peoples start to came and ask questions
and then i understand that why my machine is up.

Caesar

[Jerome Henry]

Hi,

Here is mine.

Not so long ago, I learned about RAID system. At that time, I was a little bit confused between RAID 0, 1, 5 and so on, all that seemed to be kind of win 3.11 then 95 then 98, you know, quite the same but slightly different.

I was thinking about raid 5, but the only machine I knew that had a raid card was the university server, I had access to (root ! Never do this to a newbie !). From my reading I thought that if 1 HD would be reased, the other(s) would keep the data.

To try this, I had a format c: (yes, windows) on the main hard disk...

Unfortunatly it was a RAID 0.

Guess what remained on re-booting...

(Oh so strange, I hear runor coming from the classes...)

J

[Claudio Cilloni]

I just started my DBA & system administration career... now I know what kind of... surprises are waiting for me :-)

sometimes this forum becomes really fun. Great!

Ciao
Claudio

[Chris Vail]

remsh DESTHOST; "shutdown -ry 0"





Chris

[Donny Jekels]

cat >/etc/passwd

[Geoff Wild]

Problem Exists Between Keyboard And Chair:

Just did this yesterday:

tar cvf - /sapmnt/XXX | tar xvf -

Meant to do:

tar cvf - /sapmnt/XXX | (cd /sapmnttest/XXX ;tar xvf -)

Needless to say, I corrupted most of the files in /sapmnt/XXX

Rgds....Geoff

[Suhas_2]

1> Imagine what would have happened when, on a Solaris box, while taking backup of ld.so.1, instead of doing "cp", "mv" was done !!! As most of you would be aware, ld.so.1 is the library file that is accesses by every system call. The next 1 hour was sheer chaos .. and worst hour ever experienced!!!!
Lesson Learnt: "Look before you leap !!!"

2> Was responsible for changing the date on the back-up master server by nearly a year . That night was a horrifying night of my life.
Lesson Learnt : "A typo-error can cost you any-thing between $0 to infinity."

Keep forumming !!!!
Suhas

[Steven E. Protter]

I wish to revise my entry.

No points required.

I actually had the audacity to think when my biggest client(nearly $10K spendable after expenses) had a complete failure of his backup system.

I thought I could spec the new system, order the parts and get everything working in three days before my vacation.

I wasn't wrong. It was working before my vacation. There simply wasn't enough time to test and see if it was reliable.

So, first day on vacation, I get a call. The entire system burped and kept booting into the Windows NT blue screen of death.

I am now at the client investing a Fathers day(for which I will be compensated for) in doing the job right and adequately testing the scenario.

An important part of sysadmin on any OS is knowing when to say no.

SEP

[Gary Cantwell]

A trainee student of one of my customers deleted /sbin on a HP-UX 11 system and copied it back in from a HP-UX 10.20 system and tried to deny it...


He's no longer a trainee student :-)



Gary

[Oliver Stoklossa]

Here's my greatest blunder from my beginnings:
I once had to update some patches on a HPUX 11.0 for Unigraphics and there was a patch missing in my server depot. I installed without the missing patch (by deactivating the dependency check in SAM) and it seemed to work fine...
Then I had to rebuild the kernel and it always failed because of two unresolved symbols. I did a check in the patch list and found that there were some patches not committed. So I downloaded every patch needed for those (also a small and unimportant patch called PHKL_18543) made a swcopy on the local machine and thought it would be a great idea to reinstall every needed patch + subpatches (by explicitely checking the reinstall button)... I guess you know what followed: I had a couple of hours and a lot of sweat to spend to get the system running again. Now I know ... Hands off the 18543 one, if there's no real need, and I mean REAL ;)

[Tomek Gryszkiewicz_1]

Trying to remove all hidden files:
rm -rf .*

Tomek

[SSP_1]

1. Instead of "reboot last" . I issued "reboot | last" & that's also with root priviledge. The result is it booted the server without showing me the output of last command.

2. lvreduce the var logical volume in init3 level which resulted in a gr8 mess.

[Rajkumar Basuveswaran]

Wanted to test the newly connected to our NIS master server, I entered this command,

tar cvf /etc/passwd /dev/rmt/0m.
Guess what, I was working in a DOS command window just few mins before that.

My problem ticket box filled up. Phone calls too!

Raj

[Tim Sanko]

I hired a young fellow to be my caddy.
I was hoping to pass on my knowledge and towering competence(Is it deep enough)

His mistakes are legendary. Tripping and pulling out powerplugs, fibre cables, and
then he asked for a really significant raise, I actually have snickered and commented negatively. I forgot shutting down wrong servers, reloading the 11.0 LITS patch.

He fell into a brocade switch and we got to replace many gbics and fibre channel cables.

I once merely set off a fire suppressant system by fogging a lens many years ago when trying to see if the dim led was on.

[YLTan]

We have two cdrom drive mounted side by side. Both are use by different server. In a mid of rush, I put in a CD in the wrong drive and try to mount that CD in another using "mount /dev/dsk/c1t2d0 /CDROM" and it keep saying "Device Busy".

PANIC!!! thinking my server have broken, I called HP for help, only to discover the CD is in the wrong drive....how embarrassing.!!

[Balaji N]

hmm....
nice thread.

once was asked to delete a few home directories from our main server for a project.

started deleting them. Halfway throug found that the list contained those users / home directory to be preserved and the rest to be deleted. Horrified, started restoring the same from backup and could finish it with out anyone noticing it. Till date, nobody except me and a fellow sys admin knows that.

-balaji

Experience is a comb which nature gives when you are bald. - A chinese proverb.

[YLTan]

I got another one;

Trying to be helpful....I use a find . -name *.log -exec rm -f {} \; to rid all unused log files in apps temp and log dir. but I got a bit creative and put this in root cronjobs and guess what!! the next day, some very angry DBA come banging...cos it removed all Oracle database re-do log files.

Spend the rest of the weekend restoring database....:(

[Evert Jan van Ramselaar]

Biggest blunder I kind of had to blame myself for, being part of a team, was not making a boottape for a production system without a mirrored rootdrive. Took us a day to completely restore the system when the drive crashed. :(

[Alexander M. Ermes]

Hi there.
Trying to fix a new database on a production system. Creating tables on .... production database. Cost me 16 hrs to restore and a nice speech from my boss.
Rgds
alexander M. Ermes

[SSP_1]

Sorry Guys , it should be instead of "last reboot" I tried "last | reboot" . Actually the intention was to grep the reboot in the output of last. But was a real mess after trying thsi command.

[Steve Coates]

My boss needed some patches on his workstation. I came in early to get it out of the way. After the patches finished I rebooted the system. For some reason I did not use the xterm that I installed the patches in. Well as luck would have it the one I used was logged into a customers system that handled worldwide production.

And a long time ago (20 yrs) I did an rm -r in the root directory.

[RolandH]

Instead of using
*extendfs*
command

I have used
*newfs*
on my file system


(screaming)Oh shit - can anybody tell me where my last ignite tape is ???!!!.


Roland

[Cole L.]

$ rm -rf *

oh shhhh....., Please get the backup take.