how many sys admins does it take .... ?

[John Kittel]

I'm a forums novice, - I hope I'm not violating forums etiquette here by asking a (possibly) frivolous question.

In the collective experience of forums participants, how many system administrators per system and per company is typical?

For example, we have one sys admin (me), for approx 60 VMS, 10 linux, 5 Digital Unix, 5 HP-UX, and a dozen or so other various legacy systems. There are a few individuals here who are able to pinch hit in various areas when I'm absent, but they normally have other job responsibilities and so don't contribute to sys admin on a daily basis. ( I'm feeling overworked).

I don't know how I should assign points to responses to this question. Suggestions?

- John

[Patrick Wallek · Accepted solution]

We have approximately 40 HP-UX systems and there are 2 of us unix admins.

We have some other Windows / Netware systems and our Network equipment and there is a team of 3 other folks that manage those.

It just depends on your environment and how much of a glutton for punishment you are. ;)

[A. Clay Stephenson]

The answer is, of course, "it depends". I can't really give a correct "score" for 60 VMS boxes but if 60 VMS boxes ~ 30 UNIX boxes then you probably have your hands full. It's made more difficult in that you are supporting several flavors of UNIX. Your job could (and almost certainly should) be shared between two competant admins.

Absolutely essential to your job is having an automated, completely hand-off backup system (e.g. OmniBack/Data Protector with a Library or two). Another "must have" is a product like VantagePoint/Operation (VP/O) so that you see problems and can often automatically deal with them as (or even before) they happen.

With that many boxes, I would certainly have an automated way of dealing with logins, home directories, services, hostnames, etc. -- read into that NIS/NIS+/LDAP.

[Sridhar Bhaskarla]

John

I agree with it depends opinion. In some sites, you will not be allowed to maintenance during the office hours. So, even to install a commands patch, you will need to wait until a maintenance window in the middle of the night. But you are expected to be there during the day also because that's when the production happens. So, even if you have one such system, you will need to have two administrators.

If you feel you are too overworked, then you may consider buying softwares like Tivoli, Patrol, IT/A, VPO etc., that can allow you to do centralized administration.

For me, I will be happy to handle any number of systems provided if I am allowed to do maintenance during the business hours. But that's not reality.

-Sri

[Carla Breuer]

Hi John,
I am the only sys admin / programmer / analyst at my account. There is nobody when I go on vacation, call in sick or anything else that might come up to fill in the gaps. They called me to work from my Grandma's funeral. What's wrong with this picture? I needed that little bit of stress relief.

Thanx,

Carla

PS: Nothing like blowing off a little steam to relieve a little stress.

[Con O'Kelly]

Hi John

As everyone has said there is no correct answer to the question and it will very much depend on how the systems have been setup, the user generated workload such as requests from DBA's and the type of environment (mission critical 24x&?). Certainly in that I'd look at having at least 2 Sys Admins.

In your situation I'd put as much effort as possible in to automating as many sysdmin tasks as is possible (eg through scripts), so that you are automatically notified of filesystems nearing capacity, performance issues, key processes not running, logfile monitoring etc.

I'd also try & train other people to perform basic sysadmin duties such as adding new users, reseting passwords, dns updates, printer setup etc. For example you could then set a procedure in place (with management approval) that all User Admin etc is handled by another person in which they follow a predefined set of steps.

Cheers
Con

[Tim Adamson_1]

I agree with the other replies in that "it depends".

I have found that typically 1 sys admin should be able to effectively administer somewhere between 10 and 20 Unix servers.

However, if major projects are underway and/or the site actively upgrades software, applications, Operating Systems, etc then I think it can be too much for one person.

It is also essential to ensure you have effective tools and/or scripts to assist with common problems.



Tim.

[Steven E. Protter]

I think a little too much is expected of you. If the legacy systems are stable and require little day to dday care, you're probably okay, but let me tell you about my schedule for HP-UX, which I consider the minimum for staying current.

BTW, I manage five HP-UX systems(1 educational at home), two Linux production systems for my webhosting business. I also have a few private clients with a few Windows Servers and about 20 workstations.

Daily tasks:
make sure all production oracle databases are up and healthy with regards to disk space.
make sure legacy databases and apps are up and working.
Check the email logs for critical system alerts that go out by email
scan the log files for possible security violations and stuff
Deal with telnet users that Operations can't handle(overflow).
Project work(currently planning an oracle upgrade).
Manage quality control on oracle applications improvement project(admit thats unique to here)

Weekly tasks:
Download and prepare installation of security patches required by security_patch_check
Check software.hp.com and download and prepare for installation.
Test patches and applications previously installed during prior maintenance period.
Validate Ignite backups and prepare for tests.

Thats just what comes to mind this week. There are dozens of other things I do without thinking.

If you're going to stay on top of things and keep your systems current, its tough to do with that many OS's and boxes.

What happens if there is a major problem like B
laster getting inside your shop?(Thank you Cisco consultant). That could require a log check and network configuration changes on all 100 boxes if you're unlucky.

Obviously managment is stretching you too thin to possbily maintain quality control. I hope they at least pay you well.

SEP

[Glen Trevino]

I work for a large nationally recognized hospital and our "UNIX Team" consists of 4 admins (3 experienced 1 green) which support:

20 Prod/25 Test HP-UX servers
1 Prod/1 Test Tru-64 servers
1 Prod/1 Test AIX servers
2 Prod/2 Test VMS Servers (Just Backups and Disaster Recovery, application vendors handle the rest on the VMS boxen)

In addition, we manage and code the HL/7 interfaces (about 40), manage and code the enterprise Single Sign-On application, admin our external DNS and provide Unix support for all application upgrades (PeopleSoft, Lawson, several HBOC apps, GE Medical apps).

We are never NOT busy and there is never enough time in the day to get done what must. If we weren't salary we'd all be very wealthy from overtime. But it looks like you have a lot on your plate for one admin. 60 VMS servers!!! That's a situation I'm going to have nightmares about tonight!

[aparna challagulla]

hi steven,

when u say that u check the health of oracle database by checking the disk space. can u pl tell me exacltly how do u do it. we have oracle 9i RAC on Hp-ux 11i MC serviceguard cluster.

regards,
aparna

[Jeff Schussele]

...To change a light bulb....
Just one, but it takes three days of fighting Change Control to do it & then Mgmnt questions whether the ticket was in bypass at the time.

Tongue in cheek,
Jeff (No I've *never* been theeeerrrreee)

Sorry, It just had to come out......

[Tom Geudens]

Hi John,
Now why would this be a "frivolous" question ... ;-) ?

Anyhow, we are 8 SysAdmins strong for the following :
- 300 Retail (remote) HP-UX 2 server clusters
Upgrades on these servers are completely automated (they are all exactly the same).
- 25 Central (local) HP-UX servers. Upgrades on these servers are manually (working on automating this as well).
- 10 Retail (remot) Linux servers
Upgrades on these servers are completely automated (they are all exactly the same).
- 10 Central (local) Linux servers. Upgrades on these servers are manually (working on automating this as well).

Next to our group we have an 3 men strong A.I. group (Application Infrastructure) that is responsible for the softwares and an 6 men strong database group that is responsible for the databases.

I would say you need at least another SysAdmin with the same experience as yourself and one trainee SysAdmin as well.

Regards,
Tom Geudens

[Zigor Buruaga]

Hi,
We are 2 sysadmins for the following:

- 10 HP-UX Servers.
- 60 W2K Servers.
- 10 Legacy Systems.
- SYBASE and M$ SQLServer DBA's.
- 10 routers.
- Several applications ( some of them only maintenance, others build and maintenance ).
- TSM installation and maintenance for several W2K Servers.
- PLC programming projects.
- Operations support, configuring and building monitoring tools, etc.

In short, all sort of things are/will be under our responsability.
It's more funny, and you learn a lot of things ( that's the reason why you receives almost all the job! ), but is also more stressing, specially when one of us is on vacation ( I'm on vacation now ;-) ).
About your question, I think you need other sysadmin if your systems are critical. And, in the other hand, if I were your manager, I will put other sysadmin, it's not prudent that only one person support all that job, is not good for the worker and is not prudent for the company. Only my opinion.
By the way, I think you are right ... you are overworked.

Kind regards,
Zigor

[H.Merijn Brand (procura]

I'm doing sysadmin for

1 x HP-UX 11.11
3 x HP-UX 11.00
1 x HP-UX 10.20
1 x AIX 4.3.3
2 x SuSE 8.2
1 x Win2k (my own PC)

many databases, pretty simple network.

Dial-in dial-out is managed by someone else
All other PC's and the only Mac here managed by yet another

And I'm with most of the above: it all depends. Mainly also on the user demands. If you have stupid users, who what you to restore `lost' data day after day, you will obviously have no time left for maintainance, analysis, precautions, and updates. Let alone for the time spend on the phone explaining why something does not work, or why things fail.

Another thing is the management. If they allow you to tell the users: "System down between 13:00 and 14:00 for maintainance", and you don't pass these borders, you're off much easier than management that tells you to do maintainance in off-hours.

Enjoy, have FUN! H.Merijn

[Keith Bevan_1]

John,

Same as previous responses.

It really depends on the user base, number of systems, company IT Infastructure, configuration, staff skills, training, documentaion and procedures etc etc.

We are lucky and have one for one.
One server one admin ratio !

10 servers 10 admin staff and plenty of cross over and responsibility.

Keith

[Geoff Wild]

It depends...how much change in your environment happens, what kind of Applications, size, etc...

We have:

30 HP-UX, 8 in 4 x MC/SG clusters.
11 AIX
13 Solaris
4 EMC DMX Frames

Supported by:
8 Sys Admins (3 Seniors, 1 SAN, 3 Intermediate, 1 Junior) in a 7x24 environment.

Applications:
Point of sale
SAP (one of if not the largest in Canada)
Approx 700 printers, 5000 users
Retail System
Points System
OVO
Netbackup
DNS
NFS
Samba
etc...etc...etc...


Jeff - 3 day Change Control? lucky - we have a 5 business day change control....

Rgds...Geoff

[Fred Martin_1]

We have 1 HP-UX server, 6 NT servers, 140 PC desktops, about 20 text-only terminal users. There are 2 of us here....

Yeah we're busy but it's the desktops that take all our time.

[doug mielke]

Our shop is undermanned as well, and 24 x 7 'follow the Sun' but it's tolorable when we're not making any changes. ( 1 sysadm, 12 Unix and 20 or so central Wintel boxes, hundreds more Wintels scattered about the globe, with some local folk to help with those)


Dept Rules:
Automate everything
Change nothing
Redundantly back up your redundant backups.

[GK_5]

"It depends" on so many factors.
1. How static/dynamic the enviorment is.
2. How conservative the site is about patch management.
3. How consious the site is about security issues
4. what is the policy about audting, upgrade etc.
5. How critical the systems/application are

Sorry dude, no confirm answer.

[Geoff Wild]

PS: here's some info on "points":

http://us-support2.external.hp.com/estaff/bin/doc.pl/forward/screen=estaffAssistance?Page=file0002#forpoints


Rgds...Geoff

[Kent Ostby]

Well, its been touched, but it depends on what they are asking.

One thing you might do is write down for a week or month, what you spend your time on.

If you are going though an upgrade or they are asking that you do additional things to the system (i.e. apply a new patch bundle every 6 months) then you can use this data to demonstrate time spent.

It might also give you a feel for tasks that you can automate or come up with a sleeker process to accomplish.

Finally, keep an eye out for someone who wants to learn more about the types of systems you work on and see if you can get some free work out of them in exchange for them picking your brain about how the OS works, that might be a way to offload some of your work if management can't or won't staff you an extra person.

Best regards,

Kent M Ostby

PS As for assigning points, I've seen people use 3, 5, 7, or 10 on these types of threads.

I'm generally a "5" kind of person myself for something like this, but whatever will be find.

[John Kittel]

Wow! What a great bunch of responses! I've been somewhat out of touch with other professional system administrators for so long it's difficult to gauge what is reasonable, etc. Thanks for the reality check everyone, as well as the sage advice. The forums are wonderful.

I've decided all responses deserve 10 points! ( for multiple responses, maybe a few points for the 2nd, 3rd, etc.)

It sounds like I might be a little on the overworked side, though it seems several people have it worse. I don't really feel comfortable that all the bases are covered adequately to prevent a catastrophe someday, particularly with regard to staying current with patches, and monitoring current health. As a number of people suggested, I'm already working on automating and offloading as much as possible. I think the biggest reason for at least 2 sys admins is of course succession planning ( i.e. what if I leave?, or a truck runs over me? )

In the interest of brevity, in my original post I neglected to mention that we have a pile of Windows servers and numerous Windows desktop systems that I don't have much to do with managed by a team of 2 administrators plus 2 technicians ( their descriptions of themselves, not mine), and a pile of network equipment managed by one other person, and we have two DBAs for Oracle and Ingres. Of course they're all busy too, so there's not a lot of excess manpower to which I can offload my tasks. Our VMS servers are pretty stable, and management infrastructure was set up years ago by my former colleague who was not replaced when he left ( 4 years ago? hard to remember).

I'd like to blow off more steam, but in the interest of self preservation I'll leave it at that ( never know who might end up reading this...).

- John

[Pete Randall]

John,

1 Sysadmin/backup DBA
1 DBA/backup Sysadmin
for
6 HP-UX Servers/Workstations

3 Network/PC people
for
100 Wintel/Novell platforms
and the network itself

24x7x365


Pete

[John Kittel]

Tom Geudens, - ooops, somehow I clicked 9 instead of 10 points for your response. Very sorry. Just a mistake I made, not an intentional slight toward the quality of your response...

- John

[Steven E. Protter]

Someone asked how I checked oracle space utilization.

Here is the sql.

No points needed.




ttitle left ' Free Space By Tablespace' skip 1
prompt
set wrap off
column dummy noprint
column pct_used format 999.9 heading "%|Used"
column "Tablespace Name" format a16
column bytes format 9,999,999,999,999 heading "Bytes"
column used format 99,999,999,999 heading "Used"
column free format 999,999,999,999 heading "Free"
break on report
compute sum of bytes on report
compute sum of free on report
compute sum of used on report
select a.tablespace_name "Tablespace Name",
b.tablespace_name dummy,
sum(b.bytes)/count( distinct a.file_id||'.'||a.block_id ) bytes,
sum(b.bytes)/count( distinct a.file_id||'.'||a.block_id ) -
sum(a.bytes)/count( distinct b.file_id ) used,
sum(a.bytes)/count( distinct b.file_id ) free,
100 * ( (sum(b.bytes)/count( distinct a.file_id||'.'||a.block_id )) -
(sum(a.bytes)/count( distinct b.file_id ) )) /
(sum(b.bytes)/count( distinct a.file_id||'.'||a.block_id )) pct_used
from sys.dba_free_space a, sys.dba_data_files b
where a.tablespace_name = b.tablespace_name
group by a.tablespace_name, b.tablespace_name;
clear break
clear compute
clear column