Quality Control on a Server rollout.

[Khalid A. Al-Tayaran]

Hi,

A few things to look at:

1- sendmail + security fixes..

2- sam -r

3- ioscan -fn | more (all devices should be claimed)

4- kmtune (kernel parameters for Oracle..)

5- Documentation

6- Run CFG2HTML and print the HTML file as reference. A must have. Also run nickel if you have it.

7- Remember Rock Ridge extensions for 11i.

[Steven E. Protter]

Khalid

Can you explain items 6 and 7 in a little more detail?

I'm leaving your duplicate post unassigned so I can give you a solution rabbit in case this leads anywhere.

Items 1-5 are covered thanks.

You have the best points to questions ratio I have ever seen.

Peace be with you.

Thanks.

SEP

[Khalid A. Al-Tayaran]

Hi,

Meaning:

6- Run The famous CFG2HTML script. It is very useful for me. I printed for all my servers and schedule a monthly update. Attached for conv.

7- install the Rock Ridge Extensions for HP-UX 11i to read your CD ROMS properly instead of PFS_MOUNT. Search the patch database.

[Steven E. Protter]

Well,

I'm still standing.

The system is in production.

I kept my job by not snapping at the 40% of our user base that could not handle following policy and shutting down their pc, followed by reading the password change instructions.

The move was not perfect, there are a few broken admin scripts, but all in all it went well.

Thanks for all of your wonderful suggestions.

itrc at its best.

SEP

[Steven E. Protter]

Oh yeah, and our legacy app does batch 30 times faster on this box than on a D380 Two way.

zzzzz.

Is that steam or smoke coming out of the computer room?

The smell of burning rubber?

Whoa!

rp5450's have horsepower.

SEP

[Steven E. Protter]

We have had a bump when we hit 60 telnet users.

Found an itrc post to increase the number of psuedo devices and am following it.

All and all smooth as silk.

Yes, I agreed to the removal of the website name thread. If and when there is contact on that site relavent to members, I will leave it to others to link in. The thread was a distraction and had served its purpose.

SEP

[Pete Randall · Accepted solution]

Steven,

Hearty congratulations to you. Any rollout, no matter how well planned and tested, will have a few hiccups. When you've done a good job, there's only a few minor hiccups. It would seem readily apparent to me that you've done an outstanding job. Well done, sir!


Pete

[Steven E. Protter]

Yeah Pete,

Tell me is having the box suddenly unable to connect to the SAN, the SAN administrator not understanding that I backed out the kernel change and still can't see the SAN a little hiccup.

I am a victim of non-HP technology.

One little change.

You can't imagine the swear words I'm uttering right now. 9 hours into a 25 hour fast(no food, no water), unluckiest day of the year, and the SAN admin can't believe theres a problem on the SAN. "What did you do to the box, he suggests.

SEP

[Massimo Bianchi]

Hi SEP,
some thing to check for the SAN:

- any error in the syslog ?
- what do you see from the "ioscan -fn", devices with unclaimed/unknown/no_hw ?
- maybe the kernel change and the reboot changed the device files
- any security/zoning added to the fc switches ?
- fcmsutil in the fc card tells something ?

Good Luck (and Pizza, i work better with a little food)

Massimo

[Steven E. Protter]

The SAN is the problem. We booted another server and it hung as well. Waiting for the SAN admin to get here collect data and boot the box.

Advice to others. HP hardware is worth the extra money. HP Support as well.

SEP

[Chris Vail]

Here's an easy one: prevent CDE logins using root.

Create /etc/dt/config/Xsession.d/0001.noroot
#!/usr/bin/sh
if [ $USER = root ]; then
/usr/dt/bin/dterror.ds 'Root Access Denied\n
Login as a non-root user, \n
or as root through Command Line Login' 'title' 'OK'
exit 0
fi

Do the same for oracle or any other account that may be used by more than one person. That forces them to log on using their private accounts, then su'ing to root or oracle. This records the transaction in /var/adm/syslog/syslog.log.

[Tim Sanko]

SEP,

I read the entire Post and have these additional suggestions.

1. Do a full disaster recovery scenario. Ignite through the end of your Enterprise Backups. You will probably never get the chance once server is in production (or test).

2. Time each step of the recovery and document each keystroke (except backspaces). Even the screen dumps of Veritas or EDM helps when sweat is running in your eyes.

3. Print out this system recovery manual and have it in a safe as well as by the servers. (We had a trip on our Fire Suppression system. It was nice to know when we'd be up exactly.)

4. Check all hd paths. If running EMC run
"powermt display dev=all | grep dead"

5. Try to get the DBAs to actually quit managing space. Work with performance tools to help them distribute the IO. Small changes such as a striped meta can reap big rewards.

Tim

[Tim Sanko]

SEP,

I read the entire Post and have these additional suggestions.

1. Do a full disaster recovery scenario. Ignite through the end of your Enterprise Backups. You will probably never get the chance once server is in production (or test).

2. Time each step of the recovery and document each keystroke (except backspaces). Even the screen dumps of Veritas or EDM helps when sweat is running in your eyes.

3. Print out this system recovery manual and have it in a safe as well as by the servers. (We had a trip on our Fire Suppression system. It was nice to know when we'd be up exactly.)

4. Check all hd paths. If running EMC run
"powermt display dev=all | grep dead"

5. Try to get the DBAs to actually quit managing space. Work with performance tools to help them distribute the IO. Small changes such as a striped meta can reap big rewards.

Tim

[Tim Sanko]

SEP,

I read the entire Post and have these additional suggestions.

1. Do a full disaster recovery scenario. Ignite through the end of your Enterprise Backups. You will probably never get the chance once server is in production (or test).

2. Time each step of the recovery and document each keystroke (except backspaces). Even the screen dumps of Veritas or EDM helps when sweat is running in your eyes.

3. Print out this system recovery manual and have it in a safe as well as by the servers. (We had a trip on our Fire Suppression system. It was nice to know when we'd be up exactly.)

4. Check all hd paths. If running EMC run
"powermt display dev=all | grep dead"

5. Try to get the DBAs to actually quit managing space. Work with performance tools to help them distribute the IO. Small changes such as a striped meta can reap big rewards.

Tim

[Tim Sanko]

Steve,

I remembered one more. Bump Maxuprcs up. I bumped them to 4096 from 75. With three oracle instances the maxuprcs caused process failure when oracle tried to run cron Jobs.

Tim

[Tim Sanko]

SEP,

Our ex-SAN admin changed ISLs without notifying anyone else and left on vacation. When all HPs and Enterprise TAPE BACKUPS failed he was on a plane to Aruba.

Some things kinda worked others didn't.

I then had to "vgexport -p" all 192 VGs and reimport them after suitable work (It was a major "EX-Wife") and then reconfigure all the disk and tape drives for our netbackup servers. (I decided to clean everything up at the time and removed all the devices in ioconfig and ioscanned and insf -e then the disk/tape work began. )

Our production box is directly attatched to the Symmetrix 8530. I refused to put it on a SAN after the above mentioned fiasco.

The other thing you can do is use like to like technology where possible. EMC's EDM backs up EMCs really well. I also think that OMNIBACK works for XP512s fairly well. I prefer either to other options in the emterprise environment.

The other issue is that fibre is fabulous, but if you have lots of drives in a TAN there can be a mess when/if you have fabric changes. when that happens removing the tape drives from the ioconfig is the best way to restart.

I hate my hard drives moving on me. It ruins my whole day. You mentioned a Fast is it for a spiritual reason. I avoid fasting, religiously,(ha!)but I believe it is great for personal discipline.

Tim

Tim

[Robert Gamble]

I've seen alot of very detailed how-to, here is something different

From a Process perspective:

1. The Request
1a. is the request valid ?
1b. verify requirements
1c. determine scope for all systems/applications involved
2. Analyze Requirements and Maintenance
2a. any templates/standards to adhere to ?
2b. determine if BOM is correct
2c. verify infrastructure requirements
3. Install and configure
3a. install OS
3b. patch OS
3c. install other OS pkg's
3d. mirror root
3e. configure non-root disks
3f. implement security features
3g. implement data backup solution
3h. create and test OS backup
4. Document and Review
4a. update inventory records
4b. update Disaster Recovery Plans
5. Sign-Off
5a. obtain customer/client signoff
5b. notify support staff

Help this helps you and others!