System Admin After Hours On-Call Response Times

[Ty Roberts]

My company is currently in the process of trying to redefine the expected response times of our sytstem admins when they are on-call for after hours support. As it stand now, we have admins in the office from 7am - 5am and an admin is on call 24x7 every other week. If called after hours or on the weekend, they are expected to return the call within 1 hour. There has been talks to change this to a 30 mins or even 15 min call back time. I am curious what other companies expect of their admins, so could you help me by giving feedback to the following questions? All answers will be rewarded with points.



1.) How often are you on call?
2.) If called, how long do you have to respond?
3.) Are you compensated extra when you are on call?
4.) Anything else you feel would be good information to share..

I appreciate all the feedback and help you can provide!

Thank you..

[SGUX · Accepted solution]

Hi Ty,
i'm On-Call every other week. But that is because we do not have HP-UX-experience.
We try to limit the on-call duty to one week out of three.
At this moment we are expected to respond to a call within 1 hour. ( In practice normally respond is immediately or within 15 minutes ) We do get an allowance for this duty, 118 hours a week (2*24 + 5 * 14) * (a percentage of the hour-salary)

There is another On-Call-agreement which has shorter response-times, like 15 minutes as you mentioned. This allowance is a little higher. We, as sysadmin's, do not want this last On-Call-duty because this will have much higher impact on your private life.
For example: you can't go out for running if you have to respond within 15 minutes.

[dirk dierickx]

1) once every month
2) we need to respond immediatly, we get a cell phone from the company. we need to be able to start troubleshooting within 15 minutes.
3) yes, it's only fair
4) to make nr.2 possible, we get an ADSL broadband internet connection from the company, when you are not at home you need to have a UTMS card (from the company) with you so you can call-in from anywhere with your laptop.

[Allan Bowman]

Hi Ty,

I recently left a company where I was part of the on-call rotation. Here are my answers:

1) Over the course of a number of years, the rotation varied between 3 and 5 people taking a week at a time. I personally preffered fewer people because I would forget some of the procedures if I went for 5 weeks between duties.

2) We were required to respond to the initial page within 15 minutes. This was really just an acknowledgement. We were expected to be working on the problem within an hour. If immediate computer access was a problem, we could call a backup person to start working the problem (as long as we didn;t make a habit of it.)

3) We received extra "pager pay" for the week of being on-call. This was a small daily amount to compensate for the inconvenience. Any substantial amounts of time working on a problem would result in an equivalent number of hours taken off during normal work hours. There was a separate daily pay schedule for anyone called for assistance when not on-call.

4) The company compensated each of us for our DSL lines at home. We also had a shared laptop and cell phone that came with the pager. (Yes, the company still uses pagers due to some automated paging systems.)

Hope this helps!

Allan in Atlanta

[Glenn S. Davidson]

1. I have been the only one on-call and part of a 3-5 person team that rotated weekly. They are both good and bad.

2. When by myself the company was pretty good about respecting my personal time because I was the only one. They would allow me to judge whether I needed to fix the problem now or not.
When on a team the process was a little different. We were expected to respond ASAP but if we had to be on-site we had to be there in an hour.

3. In both situations I was compensated. While on the team some of the folks got a little abusive with the compensation and it was taken away (which sucked). Typically it would be time off. Sometimes I would get paid. My experience is that I got compensated a day off for a week on-call (or 8 hrs pay).

4. I noticed that some folks aren't as interested in their job as they are in the personal life, so they tended to not adhere to the processes. It's usually the same person all the time so if they were on-call I would be less likely to answer the phone if they called. I also noticed that even when I wasn't on-call I would get calls because users knew that I would be more attentative to their needs. Maybe that is my fault but I just like to do the best job I can while I'm there. I don't think I'm a work-a-holic I just don't like my time wasted so I wouldn't want to waste other peoples time either and if I'm really doing my job during the week I shouldn't get called too often.

[Rita C Workman]

1) 24 x 7 x 365 (except when I can escape and take a cruise..)

2) No time set, rule of thumb is 'fix it asap". How fast depended on what they provided for me to respond with. A handy blackberry gets my alerts & our support contract for hardware (vendor calls me) gives me enough info to know if I should head for keyboard or "it's not an big issue..".
So far, nobody has had any problems with how fast things get fixed. And our calls aren't as bad as they used to be.

3) Compensated...monetarily - NO. I'm strictly salaried. I simply take off time (rarely though). So far they are way farther ahead than me, cause I rarely take off.

4) The better you keep your systems, the more stable. On HPUX we put up ISEE, so we get alerts and calls from HP to know if it's just a warning with hardware. We have similar for disk with EMC.
Software issues, the operators know which developer to call if their job bombs. We have a HA failover environment with MC/SG that works quite well. So, even when we have issues, systems failover and are back up (via MC/SG) in just a few minutes, so Mgmt sees it's back up and is happy.

Even though there are only two (2) HPUX Admins here, so far it takes a major storm with power outages, for us to come a running from home.
Set up stable systems, with good alert planning, test things at regular scheduled downtimes for HA - and the calls reduce significantly !

Regards,
Rita

[Peter Leddy_1]

Hi,

1) Thankfully at the moment I'm not on call ever but in a previous job I was on call every week for an extended period and then every 2nd week.

2) Response was expected within 15 minutes or else the operators would call again and so on every 15 minutes.

3) Yes, a percentage of weekly salary was paid for just being on-call and then if I was actually called then I would also receive my hourly rate for each hour spent fixing the problem. Also depending on what time of day and day of the week it was the hourly rate could be normal, 1.5 or 2 times the rate.

4) Being on-call every week or every 2nd week is too often in my opinion, it has the likelihood of interfering too much with your private life. From a business and personal perspective I think on-call every 3/4 weeks is fair for both the company and the person. Being on-call every week/2nd week can have a serious impact on your life which is definitely not the ideal scenario(I feel for you Rita & Francois - it's not nice!!)

Cheers,

Peter

[Anshumali]

1. Availability is ensured 24 x 7, blame to technical advancements and they keep on morphing their face to meet availability and stability of connection. Generally, in a month, i get almost 2-3 requests to be on call support.

2. ASAP, who wants to wait ;) But it takes time when someone alternate required to be arranged. Response times are eventually low when the activity is planned in advance and we know that a call might be expected after hours. there is a process if a 1st on-call person doesnt respond within 15 minutes, secondary person and the escalation matrix therefore.

3. zzzzzzzzz

4. For a country like where i live and support, the IT infrastructure sometimes proves to be problematic. We do have alternate connectivity methods to overcome this. As said above, Being on-call every week/2nd week can have a serious impact on your life which is definitely not the ideal scenario... i agree... sometimes i couldn't sleep properly as i kept in my mind that i have to pay attention to my cell-ring even when i am sleeping....unconscious mind keeps running........run run ring tring

[Jan van den Ende]

Ty,

1) Until recently, I was part of full coverage with 3 persons -- every 3rd week, with shorter intervals if one of is was on holidays, or ill.
2) Reaction within 10 minutes. Action cased upon info, usually dependent on our own judgement.
3)1 hours's pay per day of duty; saturday 1.5 and sunday 2 hours. If called, action hours (same weekend factors), with minimum of 1/2 jour per call.
4)Check for legal regulations. In the Netherlands, if you are in action more than 1 hour at night, you are NOT allowed regular work for 8 hours after the end of night-work. If those "mandatory resting" hours overlap standard daytime working hours, you must receive normal pay for those.
I know of countries that also have some, but of course different, ruling on such issues.

Proost.

Have one on me.

jpe

[Rob Leadbeater]

Hi Ty,

In a previous job, when on call...

1) 24 x 7
2) Not defined.
3) No.
4) If I was called out of hours, then I would get a set fee for the call, and would then be paid an extra hourly rate, depending on how long the problem took to fix.

Because I was never paid an "on call allowance" then I never felt any duty to pick up the phone in the middle of the night, although I usually would. This scheme also had the benefit that nobody in the team would feel guilty about having a few beers of an evening !

I guess the scheme your company adopts will depend on how critical the environment is... A 15 minute call back time seems a little quick, but then that depends on what the problem is. If I was expected to respond in 15 minutes for a user account lockout, for example, the pager/mobile would soon get turned off !

Hope this helps,

Regards,

Rob