Tasks should not belong to SA, but how to tell the management?

[Hanry Zhou]

Very often, we got requests from Application, and ask to create large scale of directories, with kind of complicated permissions and ownerships. All these are owned by application id's, and the application team could absolutely do them by themself. I believe this should not belong to SA work, and they could do themself. Plus, they know what they want, and it will be less mistakes.

But, the problem is, this has been the way for long time, to change the process is not so easy. Our management is kind of used to it

My question is, how to tell the management, this is should not our job, and should be applicaton team's job?

Thanks,

[Patrick Wallek]

It sounds to me like you've got a good start on it already.

Explain the issues to them -- There could be mistakes in typing up requests. Mistakes in understanding the request of verbal. Problem reading someones writing if handwritten. Also explain the time issues -- If you have questions, you have to go back to the application team and ask, delaying the directory creation.

Tell them that everyone could be better served by the app team doing it themselves. They would probably have a faster response time if they don't have to ask SA's and this then leaves more time for SA's to work on other things.

Don't just say -- It's not my job. Put some thought into your reasoning and justification.

[Jeff_Traigle]

Well, I'd explain it pretty much as you did to your manager and have him/her work it out with the Applications manager as he/she deems appropriate. Seems more than reasonable, if they have the permissions on the system, they should be creating their own directory structures. The only plausible reason I could see for involving the Sys Admins would be to validate that the ownership and permissions they want to create are reasonable. (Applications people have a very bad habit of making everything 777 in my experience. Sounds like maybe your Apps people have a better grasp of permissions than those I've dealt with in the past.) The sad truth is that you may not win this battle for various reasons. The law of inertia applies to organizations as well as physical objects. Change is resisted in many cases for no reason other than "this is the way it's always been." If management can't or won't acknowledge and act on a reason to change policy, then change to the policy will not occur.

[A. Clay Stephenson]

It's not uncommon for requests like these to be made and often it requires super-user permissions to actually do some of the steps; in which case, the applications guys could not do all the tasks.

A reasonable approach is to require that you be supplied with a script to run BUT while that approach does make your task easier to perform, it potentially opens up a big security hole. Anytime I am asked to run something like that, I carefully inspect the script and really watch out for anything that does a setuid operation.

[Bill Hassell]

I look at the sysadmin job from a different perspective. The sysadmin is ultimately responsible for the state of the filesystems, whether they have right permissions, have run out of space, are properly backed up, etc. An application group is familiar with programming and debugging their code but as many sysadmins will agree, very, very poor training about Unix ownerships and permissions. The majority of app writers and DBAs solve their file and directory problems (almost always "permission denied") by using the disastrous commands chmod -R 777 and chown -R dbauser:users. Management should rightfully hold me responsible for unsafe or inappropriate permissions and ownerships.

Proper change control would include a review (not just a do-it-now order) by both apps/dbas and sysadmins before going into production. I ALWAYS set permissions to the tightest setting (ie, 600 for files, 700 for directories unless there is supporting documentation as to what other groups and users need access. umask is always 077 for root, 027 for users. Yes, it often means an extra step to open up the permissions, but it keeps everyone aware of the importance of security.

[Richard Darling]

Hanry,
Bill is absolutely right. You are responsible for the filesystems...period. If anything goes wrong, which it can; and probably will if you force this issue, it falls on your shoulders. Typically, application groups know little about the issues of ownerships and permissions.

[Pete Randall]

Well, I'm going to take the middle ground then. While I agree somewhat with both sides (let them do it vs. it's your responsibility), what we have done here is to set up what we call "work volumes". They have the "disastrous" 777 permissions and the App folks have pretty much free reign over them. The only thing I ask is that they set up some method of cleaning out the crap they leave behind. The can either provide a method programmatically as part of the application, or they can ask me to add the cleanup to one of my periodic scripts.

What doesn't get caught by either one of these methods is fair game and may disappear at any time. I regularly threaten to delete everything that they haven't either identified as necessary or that is older than some specific period. That seems to get them moving.

Whether this would help in your case only you can decide (obviously).


Pete

[A. Clay Stephenson]

It seems to my poor ole brain that requiring the developers to write the deployment script solves all of these problems IFF (if and only if) you do your part in carefully reviewing the script. When you see lamebrain file mode assignments and loose umasks, you tell the developer's and they (not you) modify the script -- that way there is feedback in the loop. Eventually, they will begin to catch on to proper file permissions and ownership. The real beauty of this approach is that there is now a repeatable set of commands that can reproduce your filetrees so that any other deployments are easy.

Now the other "gotcha" is that at some time in the future, these guys can "improve" the file modes on an existing filetree in order to "fix" any problems that your silly meddling caused. That is why it is still essential to periodically check the entire system for changes like that --- and that IS your job (and you should already have cronjobs in place which do just that task).

[Hanry Zhou]

Here is a few more facts.

The parent directory which all dir/file trees created on is completed owned and grouped by the application id. Only the file system is owned by root. So, they have the full control of the directory and everything underneath.

Also, the application user knows exactly what permissions(owner, group, modes) should be creaetd, and how they are should be creatd. So, we are dealing with people who has pretty good knowledge on directories/permissions. As a mater of fact, they provide us detailed work instructions, and include everything you need.

I asked question why don't we let the user do these themself, I was just simply answered that we don't want the user to maintain directories.

I understand two different sides on this issue, but, it seems to me, this entire directory tree is just part of their applications, they have better knowlege to maintain, we (sa) could provide technical assistance, monitor them, or put appropriate limitations on them, but should not do the job for them.

[A. Clay Stephenson]

In that case, writing a set of detailed instructions on how directories/files should be organized is no more difficult that writing a script to do those assignments --- and that is the approach that you should take. In fact, the method you are now using is roughly twice as error-prone as running a script would be because not only do the instructions have to be correct but they must also be followed correctly. I would trust a script to do that much better than I would myself.

Having said all of this, it's still your responsibility to monitor the files and permissions not only when first created but also as an ongoing task.

You should approach your management and developer's with the idea that it is their responsibility to provide a script and to ask a super-user to execute it if needed (create users, create groups, modify /etc/profile, etc.) and it is the responsibility of the SA's to make sure that the filetrees remain in compliance with good practices.

[Bill Hassell]

I have exactly thiss situation: developers create a simple spreadsheet with the names of the files and/or directories on their development boxes, followed by the two checksum numbers, then permission numbers, ownerships and the destination full pathname. My sysadmin script reads each line, and then with 100% paranoia, checks EVERY item on the line. Things like: does the original file exist? Does the checksum still match? Are the permissions valid octal numbers? (777 and 666 are flagged for further review and once confirmed, an override in the script allows wide open values) Then the remote system is checked for the path, creates it if necessary, verifies if the file exists, and if so, renames the old file and moves it to a ./old directory, then uses scp to copy the new file followed by cksum to verify the copied file.

The entire process is printed to stdout so it can be copied into the call tracking system. By using the script, I can throw in all sorts of controls like no root ownership (unless discussed) and prevent a lot of typos with extra checking.

SO I get the developers to do the front-end work and send me a work order, then I let the script make all the checks and decisions while I copy paste the results.

[Heironimus]

"I asked question why don't we let the user do these themself, I was just simply answered that we don't want the user to maintain directories."

Why don't you just write all the apps so the developers won't to have to maintain code, either.

But seriously, there is no "right" answer here, it's largely a matter of philosophy and specific environments. How much responsibility can you delegate? If you have application people who know enough and ask questions as needed you can delegate a lot. If you only have developers and no dedicated application admin group you won't be able to delegate anything.

I usually push for having application admins maintain their own directories (at least their data directories, I sometimes keep their binaries). It minimizes the time application people spend waiting and lets the systems people focus on maintaining a solid infrastructure - nobody ever hired me for my amazing skill with "mkdir".

However, that requires that you have either qualified staff or regular auditing of the filesystem (which can be scripted). Many places don't do either. It also requires that you be able to delegate the responsibility along with the authority, which can turn in to a battle between managers - everybody wants the power, nobody wants to have to fix the problems or face the blame.