Booting RHEL 5.6 from SAN - Pros/Cons?

[jfonz25]

I'm just looking for a little feedback on bfs.    I currently have 4 BL460c's booting from the SAN.   The machines are strictly Oracle DBs (using RAC/ASM) running RHEL 5.6.    I don't really see any of the benefits in this type of configuration (however I do see the benefits in say VM environement).   Just hoping somebody can change my opinion before I rebuild and configure the OS on internal disks.

Thanks in advance.

[Steven E. Protter]

Shalom,

Can you do it? Yes.

Is it something you may need to learn how to do some day? yes.

Is it something I would like to do? No.

I prefer to boot off raid 1 pair of local boot disks.

I would not volunteer to start doing this. Booting off SAN leaves you no diagnostics in the event of SAN trouble.

SEP

[jfonz25]

Thanks Steve.

I actually have it up and running and well aware of the issue that could come from a SAN failure.   I also see many issues w/ mapper when adding/removing storage (escpecially removing storage).  Seems as though a reboot is needed a lot of the time to "clean things up".   I don't have these issues on my machines that are local boot.   I'm not sure if they are related or just some of the many "features" of RHEL.  

Still investigating...

[Alzhy]

I could not think of anything but PROS. All of my clients -- HP-UX, SOlaris, Linux have been on SAN or iSCSI boot - UNIX since 2004, Linux - since RHEL 5 debut'd.

The unfunded issue usually is "what if SAN fails"?

Well Duh!? -- If your SAN fails (your switch/storage array, etc) -- then there is erally nothng to troubleshoot anyway!

Most SAN failures are due to HBAs or Cables going bad.. That is why you have redundant FC connections. Other "SAN Failures" -- I still do not understand what they are or could be and what to troubleshoot for.

SAN boot (or its cousing iSCSI boo) -- IS best practice these days with the following advanatages:

- easy recovery of your system on to a diffrent server

- no more local SCSI, SAS or IDE drives that can FAIL

- faster boot ups and OS binary, temp and swap areas

- easier OS backups and fallbacks/restores (use your SAN array's snapshot features to create a periodic backup of your OS/Boot LUN prior to maintenance or patching

- easier to clone systems onto new hardware/servers

In short, FUD on SAN Boots are UNFOUNDED..... you're doing the right thing sir.

[Alzhy]

" I also see many issues w/ mapper when adding/removing storage (escpecially removing storage).  Seems as though a reboot is needed a lot of the time to "clean things up"

If you follow procedures with handing SAN storage in a multipather environment - you should have no issues and no reboots to "clean" things up good sir.  This is particularly true with removing/deprsenting SAN storage -- which there are "strict" guidlines to wit:

1.) stop usage of SAN disk from LVM, Oracle (ASM), VxVM, etc -- as your case may be

2.) make note of all the /dev/sdXXX device instances of the SAN disk (multipath -ll)

3.) remove the multipath device:  multipath -f

4.) remove entry in /etc/multipath.conf (if you use friendlynames of multipath aliases)

3.) remove the /dev/sdXXX instances of each multipath device (echo 1 > /sys/block/sdXX/device/delete

Follow the Above and you shoud have NO problems with Linux On-Line Storage Management....

It is really ALL just a matter of awareness and education. Unfortunately - this area of Linux is not that widely documented but due diligence on you part and Google will yield you practically all solutions and howtos to evey Linux issue good sir.

[Steven E. Protter]

Thanks for your reply.

I'd rather be able to produce an error message on a booted box, which proves the SAN is down versus being able to do nothing.

Call it old fashioned, parse it as you will, if given a choice, I'd rather not boot off SAN.

This is coming from the experience of a 10 month working experience in a shop that booted all its HP-UX boxes off of SAN. Nothing bad ever happened there except the time the SAN admin gave me a data disk from system b on which to build system a.

The system I build worked well but eventually had to be rebuilt on a clean disk. System b wasn't doing so well after its data got overwritten.

SEP

[Alzhy]

Shalom!

To each his own eh chap?

It's common sense really... SAN Dead or faulty? -- What is there to troubleshoot on your servers? Big Nuthin really.

Local boot Mechanical Disks are Dead.. it's so 20th century.

"ept if the OS Admin is overzealous 'nuf to inist his end needs to troubleshoot a SAN...

[UVK]

I agree with SEP.

SAN boot will only make it complex so I would prefer local.

but now that i read and think I have a question

How effective solution is SAN boot as DR solution. If I have BCV's configured can we use the BCV to boot at my DR site, given i have the same hardware ?

[jfonz25]

Alzhy/Protter -

Thanks for both of your responses.   I guess this is what I asked for...(Pros/Cons)!  

The RHEL environment is comprised of 8 blades, 4 are being used standalone for the DBs (but clustered for RAC/ASM), the other 4 are being used as Application in a VM environment running RHEL.   What I have decided to do is rebuild the DB machines to boot from the internals and have App machines boot from SAN.  

Again...I really appreciate the responses. 

Thanks!

[jfonz25]

Alzhy -

Also, thanks for input on the disks issues.   It seems like you can do things 99 times the same way on RHEL, and then on the 100th, it steps on itself.   It appears that disk maintenance is a bit of a bear on RHEL.

I was pretty close to what you recommend, but as you say, I got most of my input from google.   I am going to try your method next time I have to remove some storage.

This is what I was doing (keep in mind new to RHEL, my background is HPUX, Sun, NSK).

1. Stop I/O on VG and/or ASM/RAC disks

2. Remove VG from system

3. Remove multipath.conf entry (for ASM/RAC only)

4. Remove SAN attached storage

5. run #rescan-scsi-bus.sh -r (remove devices)

6. service multipathd reload

That probably worked 75% of time.