Mistakes

[hpuxrox]

How may mistake can a systemadministrator make before he should be released from the job?

[James R. Ferguson · Accepted solution]

Hi:

One thought (within reason) is that if you aren't making mistakes then you're not doing anything!

That said, like everything, "it depends".

Is the person generally thoughtful and careful executing tasks? Was the mistake an honest one that the admin learned from making? Did the person honestly admit the mistake or try to hide it only to be found out? Has the person made the *same* ( or very similar) mistake again and again?

If the mistake was "honest", openly admitted, and not hidden; contrititon was shown; and knowledge was gained, then probably everyone has profitted.

How many times? It depends...

I'm reminded again of the adage not to throw the baby out with the bath water.

Regards!

...JRF...

[S.K. Chan]

In my opinion to be consider "released from the job", it has to be a serious offense, not mistakes. For example security violation, company proprietary violation, etc, etc. Mistakes can be corrected and learnt from.
my $0.02

[PIYUSH D. PATEL]

Hi,

It depends on the output of the mistake. If it has costed the organisation dearly then definitely some action may be taken. Sometimes even a small command can create havoc. People learn from mistakes only !!!

Piyush

[hpuxrox]

Contractors can be released without a serious offense.

[A. Clay Stephenson]

I hate touchy-feely stuff but the answer is, of course, it depends.

If he lies about what happened, one time is sufficient. I have shown SA's the door in that case. If the mistake was admitted soon after the event then I'm much more tolerant.

Nobody expect perfection but I do expect honesty and caution. I always tell staff members that if you are doing something new ALWAYS do it on the Sandbox first.

I guess my best answer is that his mistake clobbers my production uptime statistics then I'm mad. If he then lies about it, I'm real mad.

I have very little tolerance for stuff like: "I didn't realize I was on the Production Server", "I didn't know that I was logged in as root", or "I didn't know that that unused LVOL was raw data".

If it's small stuff I don't sweat it at all, but if a stupid, avoidable mistake shuts down production at an entire plant then it's probably time for this guy to update his resume.

I tend to be upfront with this sort of thing during the interview process.


Warm fuzzy thoughts, Clay

[RikTytgat]

Hi,

What is the definition of an expert?

An expert is someone that has made all the mistakes.

Release him from the job when he made the same mistake twice!

Bye,
Rik

[Jeff Schussele]

Hi Yates,

I agree w/the others i.e. the act or the following actions need to be deemed malicious.
Everybody makes mistakes & I also agree that if some learning doesn't take place then that could be deemed malicious.
But I also think a lot of these "situations" stem from mgmnt i.e. they aren't interviewing properly and/or they're just filling seats. They need to make the right decisions & properly balance quality with quantity & by quantity I mean the $ they're willing to pay.
The old adage is still true - You get what you pay for.

My 2 cents,
Jeff

[Martin Johnson]

Well, I know of one small accounting firm where the secretary was placed in charge of doing the system backups as there was no SA (she became the ad hoc SA without knowing it).

Her training was "this is how a backup is done". Things were going fine until one day they lost a key file. No problem, just restore from backup. Well, the only type of restore they knew how to do was a full restore. So they attempted a full restore. Unfortunately, there had been errors in the backup, there was no data in the backups. The restore deleted the OS with nothing to replace it with.

I was called it to try to salvage the data. I couldn't. Their choices were to find a recovery specialist company or install a new system, knowing all their data would have to be re-entered manually. They chose the new install. The firm had to hire 15 temps to manually do the payrolls that the firm was responsible for, and 5 tmps to manually re-enter all the data. It took them a week.

The secretary was fired before I had the OS installed.

So I think the answer is how much pain did you cause and how pissed off your management team gets.

Marty

[Justo Exposito]

Hi Yates,

We are person and a person is no free error by a definition.

Then, the mistakes are less important if you have a good installation, a good DRP, a good training etc...

If not, a little mistake can transform to a great, great problem...Then the responsability of this is the person that don't take the necesary care.

Even there are a mistake and a malicious mistake, and the second case is no admissible.

Then the answer is depend.

Regards,

Justo.

[Bill McNAMARA_1]

I agree with S K..
You need to get them in training, and to follow procedures. It's not easy being a new sysadmin.. even an expert will take time to learn a new environment... and still make mistakes.
Lot's of errors on the other hand, and lots of repeated rm -Rf /* 's would annoy me though!, but in my type of work, technical compentance is seen as an equal skill as personal relations (are they at least trying!)

Later,
Bill

[Hartmut Lang]

And you shouldn't fire (oops, want to say "release") someone from his job, if you cannot be shure you will be able to get someone else, who makes less mistakes.

Hartmut

[Duncan Edmonstone]

A *good* IT Manager views any mistakes made by his staff as his own mistakes:

-If the mistake was a repteated or malicious one, then the manager considers how he managed to hire the wrong kind of people, and looks to change his hiring criteria.

-If the mistake was just that - an 'honest' mistake, then the manager looks how the process his people work within can be changed to prevent it happening again. Good change & configuration management procedures can make these kind of mistakes very rare.

Just my thoughts...

Duncan

[Steven Sim Kok Leong]

Hi Yates,

I would say that the important variables are the degree of the mistakes, the rate of unique mistakes over work done, the rate of repetitive mistakes over work done and also whether sufficient planning and testing was done in advance or in short DUE DILIGENCE.

I would also say that one important factor that must not be left out is culture.

I won't want to pinpoint specific cultures here since they can be pretty sensitive. However, some cultures are more tolerant of mistakes than others. While in other cultures, there is simply no room for mistakes. So it depends on both the country and company culture as well.

Hope this helps. Regards.

Steven Sim Kok Leong

[Trond Haugen]

Would you apply for a SA job where only X (a few) mistakes where allowed? And if you did would that make you nervous and more likely to make mistakes?
In Martins story I believe the person who out the secretary in charge (and "trained" her) should be released if anyone should.

Regards,
Trond

[H.Merijn Brand (procura]

As also shown by the others: it's not the count that counts :) it's the severity

If one can prove that whatever happened is malice or disturbing the company's production on purpose, I see no reason to give him/her a second chance.

If the error is severe (rm -rf /) it depends on how it happened, how soon he/she admits, and the steps he/she takes to repair the damage. Everyone makes mistakes now and then, but it takes a /human/ to admit, and an expert to not panic, but find a way to get it right again asap.

If severe errors happen multiple times (in a short span), the person is probably not in the right place anyway

Other things are theft. When is theft considered theft? If one takes two empty blank sheets from the printer to make some personal notes? If one prints 12 color-me pictures for your children? If you take two pencils? If you take a laptop? If you use company software? And use that for off-office service to enrich yourself? Come in the office early to drink company coffee instead of doing that at home to save money?

Just my (personal) opinion

BTW I did advice to release a guy long time ago, because I could *prove* he did wrong (multiple times) and he kept denying (multiple times). He was released

[Paula J Frazer-Campbell]

Hi

Like a lot of answers on the forum "It depends.

It depends on what level the sysadmin is at.

It depends on how it affects the business.

It depends on how it affects other team members.

It depends on whether they have made the same mistake before.

It depends on the nature of the mistake - was it a simple typo or an untested script/routine run on the live server.

It depends on whether they were correctly trained on what they had been tasked to do.

It depends upon how many consecutive hours they had been working for ??? tired staff will make mistakes. No ands, ifs or buts ??? They Will.

So the definitive answer is: -


IT DEPENDS.

Each and every situation must be looked at and all of the above and more must be taken into account before a decision is made.

Just my 0.02 Euros worth.


Paula

[Nick Wickens]

My view is that if you operate an environment where fear of being released if someone screws up exists, then you run the risk that mistakes will be covered up and a covered up mistake can easily turn into unrecoverable mistake.

As has been mentioned above if the SA quickly admits the mistake and has a plan for recovery AND for ensuring it does'nt happen again then there is little to be gained from an explosion of anger from management. Just make a note that the mistake occured for future reference if it happens again and if you want to make a point do it in the salary reviews of that person.

[Steven Sim Kok Leong]

Hi,

Another point is the publicity involved. In certain cases, where the mistake was brought to the newspapers and becomes talking point, the company just have to sack the person because of the scrutiny.

The mistake might not be that grave, but the complaints by the users on the mistake would be.

The company may be tolerant of the mistake, but the customers aren't. If that's the case, the company sometimes has no choice. *shrug*

Another 2 cents. Regards.

Steven Sim Kok Leong

[Pete Randall]

Brother Yates,

To me it all depends on the honesty factor. If the "offender" has the right attitude about h[ie][sr] responsibilities, owns up to h[ie][sr] transgressions, and learns from them, I'm a lot more tolerant. On the other hand, the second hint of deceptiveness is going to send me across the hall to talk to Human Resources about getting rid of h[ie][mr].

Good to hear from you, it's been a while.

Pete

[Victor_5]

Frankly, I like this topic, I went through all the replies, I like them as well. Actually, this is a question we have to face every day as a Unix Admin.

Somebody told me, never make mistake, even a minor mistake, especially for those contractors and new comers, otherwise, you will have a big trouble.

Somebody told me, you need make some mistakes sometimes, not those critical or serious mistakes, otherwise, who can realize the value of your working?

I would like to say, it is really depends, depends on the situation, depends on the impact of the mistake, also depends on the manager...

I heard this sentence from others, I like it, "Experience comes from doing things, wisdom comes from making mistakes."

Another point I would like to mention is to be honest, from my experience, this is the best way when something happened, share the whole story with your manager, and related person, we all human being, nobody can avoid making mistake, the only difference is how we handle that after we made.

[harry d brown jr]

ONE - pretending to be one when they aren't.


live free or die
harry

[Martin Johnson]

Trond,

Unfortunately, the secretary's boss was the President/CEO of the company. It was a small company.

With his management skills, I'm surprised he is still in business.

Marty

[Jon Mattatall]

Hmmmm....
Interesting to compare the recent "You know you have got it wrong when..." thread with the responses here.

There are certain hanging offences anywhere. Theft, vandalism, harassment, and so on are usually spelled out in your contract as grounds for dismissal. In these cases, if you can't play by the rules, don't play the game. No sympathy there.

Procedural, admin-type mistakes? If you lie about it, and you're not good enough to cover your tracks, you're gone. Integrity counts with me (not that I'm consulted).

Errors that you admit, correct, and do your best not to repeat? ALMOST no limit. "He's never made a mistake, 'cause he's never done nothin'".

Constant mistakes? Who hired this person? What were/are the expectations? When I started here after 7 years with AT&T, I had a basic admin course for Cisco, NetWare, and NT under my belt. I could just about type "ls" instead of "dir" 50% of the time. Of course I made, and still make mistakes. But I was hired at the bottom, not the top of the ladder. As long as I learn from them and keep the alibis to a minimum, my employer is happy.

If I was hired at the top, my butt would be GONE. Justifiably, in my opinion.

It's a management / HR decision. If the admin can be trained and shows progress, fine. If they blow away a production box every month, they may be in the wrong line of work.

In practice, the spectacular or malicious errors tend to be career-limiting. If I push something out and drop a dozen boxes, I expect severe consequences. BOFH tactics are normally punished as well.

Jon

[Mark Greene_1]

Certainly, there will always be mistakes, suchs as typo's in a script, or forgetting about the timing of a particular cron process or when a certain application batch process is run or things to that affect.

There are different kinds of really detimental mistakes. There are those that show a sever lack of knowledge, like rebooting unix after making a dns change rather then just killing the named pid, or having to call for HP support for every issue because the admin just does not know.

There are those that reveal an inability to learn from previous mistakes. This goes beyond just making the same mistake over and over, but also not being able to see the system as a whole, not understanding that changing a config file for one issue may affect other processes, for example.

Then there are those that show no consideration for the organization, like unplanned and unannounced reboots of servers, poor back-up strategies that result in lost data, changes that are made on a whim that have a negative impact on network performance, and the like.

And finally, there are those mistakes that are borne of malice and outright bad intentions. Deliberate rebooting of systems, deliberate planning of down-time in order to inconvenience certain people, and just general abuses of authority.

Each of these types should have a different management response and escalation process, whether formal or informal, IMO. The last one, of course, very potentially being grounds for immediate termination.

Just some thoughts, HTH
mark