Just an idea ...

[Volker Borowski]

Hello friends,

the speed of response to most questions in this forum is really great. Since the members are located all around the world, response is guaranteed nearly 24 hours a day.

That fact made me think about some sort of 24 hour support in general. You have big companies like SAP giving 24 hours support through diffrent locations from diffrent places on the globe. Now the problem is, this is mostly available either only for "priority one" problems (system is completly down) or it costs a whole lot of money.

But what do you do for very common problems like enabling a printer, enlarging a filesystem. Does this have to wait until tomorrow ? Is there really need to wake up a poor admin for this in the middle of the night ?

Would you as an admin consider to give this part of administration out of your hands. I mean we are in 2002. Connectivity is no problem at all. So there should be no problem to have someone from the next continent fix your archiver stuck of your database. I thought this might be a good business idea, because this service would be service in normal office hours (on the next continent) but I was always out-voted for security-reasons, 'don't likes' and anything like that.

OK, I see that there has to be a certain amount of trust for this, and may be some technical equipment around security, and agreed and commited documented procedures.

But if you assume the technical stuff could be aranged in a way that you could accept, would your belly allow you to do it ?

I am curious about your views.
Volker

[Bill McNAMARA_1]

You're right in a way Volker, but it's really all down to education and learning.

The forums could be a good way for an operator (as oposed to an administrator) to do the night shift.. but in a 24x7 environment where the stakes are high, I'd prefer an experienced sysadmin who knows the system well around..

remember also, too many cooks spoil the broth ;)

Later,
Bill

[harry d brown jr]

Volker,

If I had the need for it, it would be a great idea, but we have about 20 HPux system admins (some have a slight knowledge of our sun servers), ~15 DBA's, ~15 web admins, ~12 PC lan admins, ~200 1st and 2nd level support people (mostly apps, but some simple problem resolution).

Now as for can it be done securely, you bet! I can work from home, over the internet, have access to all of my network, including by NT file shares. I use Cisco's VPN on a pc at home, using Secure ID cards for 1st authentication, then ldap for the 2nd authentication, of course all systems have their login security. I have cable modem access to the internet from home. I can use Exceed, which I do, and it's quite fast. I have servers that have either HP's WebConsole, or Arula's WebConsole (they make the webconsoles for HP - of which they used to be part of HP). So there isn't much I can't do, other than physically touching the machines, which is why we have operators :-))

Not too long ago, I supported over a hundred client's from California to Rhode Island. These clients were banks, and they ran their bank using our software. I remotely administered their machines when they had "issues". I was bonded by my company so the auditors didn't have a problem with it. Now I didn't have their root passwd's, they had them, and they gave them to me when they needed my services.

live free or die
harry

[Volker Borowski]

Hi Bill, Harry,

that was not my intention.
For a tight 7x24 shop, there is no way out to do this with own people, no question.

But if you consider a normal 5x10 shop, with some employees working in the late evening and some two or five guys also working on weekends. Would you set up some On-Call- procedures for those few people ?

Of course, if you have 20 own admins, you have no problems to set up some on-call procedures. What if you have only one or two? These poor guys may be on call 365 days a year :-)

Volker

[harry d brown jr]

Volker,

When I was at that banking software company, I was on call 24(hours)x7(days)x52(weeks)x9(years) and there was usually just ME, and this was the day's before cell phones where everywhere and before modems faster than 19K.

But I understand where you are coming from! It's a great idea, especially for small companies. I'd say $150us per call, with 2-3 hr minimum!

live free or die
harry

[Wodisch · Accepted solution]

Hello Volker, CEO of ITRC.com,

nice idea, and worth talking about.
Do you think about "ITRC.com", the whell-known, world-wide-working virtual company, employing ~16000 admins all over the globe? :-)

The difficulties are not wether *we* would trust the persons (persons, not companies) doing the work, but our C?Os (? like E,T,I...)!
And those seem to "trust" something as soon as it has been proven by other (successful, big) companies for some years!
Some support-vendors kind of do this, already. IIRC then SAP has a support subsidiary in australia, doing support for germany (6 to 8 hours off).

The hardware costs are pretty low, in deed: a modem, ISDN, or *DSL dial-in, perhaps with some kind of call-back to a central pool of "ITRC.com-POPs" (points-of-presence, if this is not (TM) by somebody ;-).
And VPN is plain software, today, isn't it? Not so certain about "more-than-128bit" in the USA, though - is it permitted, there?
And the "certification" for the admins would be the "hat" and perhaps a minimum amount of time *working* at the forums?

Hence I believe *we* would need some "roof" organization, hiding the details from those C?Os, and *we* would need some of the "mighty wizards" or "royals" or "pharaos" as "dispatcher", and for the "premium" customers and the most-difficult cases even the "olympian" ;-)

To bring it down to a single statement: legal and trust problems!

Just my $0.00000002,
Wodisch

[Volker Borowski]

:-)

No Wodisch,
intentionally I did not think of...
But now, as you put it....

CEO of ITRC.com with 16.000 employees **smile**

That is a very charming idea, and finaly will deserve a bunny. But I guess there may be some protest on this.

Volker

[Rita C Workman]

Hi Volker,

As an Admin, in a shop that for the past couple years, had just been me...(finally I got 1 person for a backup I'm training)...Well..in a word..NOPE.
I have no problem delegating out certain admin tasks using restricted SAM. A Help Desk and developers so they can reset certain things, etc. Don't need an outside vendor for this.
BUT...past certain very basic tasks--that's it. I agree 110% with Bill.."too many cooks spoil the pot"...and I can't guarantee (experience has proven to me) they would tell me how much salt they put in or anything else.
Nobody knows your system as well as you (the person who set it up..).
If you open the door to outsourcing 'some' tasks, believe me your one heartbeat away from outsourcing your job.
I work for a government agency, that is always playing politics, and outsourcing was one. You think you have headaches now..wait till vendors are allowed to get hold of your servers. And the first thing they want is sole ownership of 'root'.
It's not pretty ..

Just my 2cents,
Rit

[Krishna Prasad]

Hello Volker,

I have to agree with the majority. You can allow access to the most knowledgable system admin....but that doesn't know he/she will know the little things about your environment/company/network/applications etc...

I think this kind of knowledge is one of the most under rated aspects in all of IT. The so called just get a consult when we need one view of the world doesn't always work.

How many times have we all had to train the consult...not becuase he/she was not knowledgable about the overall subject matter but needed to know the "details" of the environment,application etc. I understand you were basically talking about the simple problems but how many times had simple problems grow into big ones? Sometimes you just don't know what it will lead to.

[Carlos Fernandez Riera]

It is not a bad idea, not at all... in facts HP is taking goals ( and wins prices) with THIS ( this forum).

But by my experience, there is not lots of traffic on nigth ( my nigths, my working nigths), neither weekends.

In the other hand, all you dont need to now detailed hardware, neither passwords to remote help-admin boxes. We do it all the day.

(HP, is this a message for you?)

Regards.

[Darrell Allen]

Hi Volker,

Yes, it would be wonderful to have some in perhaps Australia to cover for me during my nights and vice versa. Much is possible today with VPN and web consoles, etc.

Another issue is the question of trust, accountability, and liability. I can't just allow someone access to my systems without legal means of holding them accountable. I don't think my stockholders would be appreciative of that either. I'd think they would have to be an employee or under contract.

Bill's "too many cooks" issue and Ron's about knowledge of the environment are right on. A resounding ditto on Rita's 2 cents as well.

Darrell

[Paula J Frazer-Campbell]

Volker

Very good idea, but accountability would be the problem!

If it goes wrong who takes the blame?

Also our server are unique on their configuration to do what our companies require and interdependency within then can be very convoluted.

I would not trust fully a newly employed sys admin on my site with full control of my servers because off the above.

Paula

[Robert Gamble]

I thought I would add my comments here as well.
Commercial IT Service companies do utilize your concept Volker.

In fact, in my department of a huge IT Services company, we do administrate a huge number of servers/workstations (10,000+) with less than 200 admins (located throughout the world).

We are able to do this by:
- Brand name services company (TRUST)
- Use ISO standardized processes for all systems
- High standards for hiring, continual evaluation of skills and abilities, continous training
- All client systems are availiable on a common infranet

As expected, overhead is high, but so is the profit margin.

[Sandip Ghosh]

Hi Volker,

Your idea is nice. But after some time you will find that your C?O also get this idea and instead of keeping you an employee he will tell that I will take out the help from ITRC.COM whenever it is required and that will be cost effective also.

Just another idea.

Sandip

[Jim Turner]

Hi Volker,

I can't deny the idea is cool, but I have to echo what others have indicated. For us it would be both impractical and undesirable.

There are five of us that do Tier 3 support for 80 HP-UX boxes (K's, L's, N's mostly). A problem that has dropped through the Hell Desk (24x7) *and* UNIX Tech Support (also 24x7) down to us is one that I'm going to want to handle myself. I don't mind getting woke up occasionally for those. We're all on call 24x7x365, and it's not too bad.

Cheers,
Jim

[John Payne_2]

Volker,

We are set up with ~10 admins, and we rotate on-call, but if there is an HPUX problem, I'm the guy, whether I am on call or not. That pretty much means 24x7x365... Lucky for me, I only get woken up out of bed 2-3 times a month. (Sometimes less.)

I have delegated stuff I really do not want to be woken up for out to our Operations staff. (Mostly student employees, we are a University) Things like printer setup, what to do when the filesystem that contains 3rd party logs fill up, etc. They have documented procedures for what to do in certain situations, but for most everything else, we try to keep them blissfully ignorant. That way, they are sure to call me when they need to. We have about 50 HPUX boxes, and plently of other problems for Operations to deal with. (Including just about every main OS.)

I figure that at least that way, I always know what is going on and do not get blindsided by someone coming and asking why a service has been down for a week...

John