I've found it best to put the virtual addresses and the real addresses on two different vlans, although we've made it work without it. This prevents people from finding out that they can have their favorite server that they like to get to, and bypassing your hard-won load balancing system. With this, you can allow users into one vlan (with the virtual IP's), and deny them access to the actual VLANS that have the servers.
That switch has a few load-balancing options, including round-robin and process-counter basis (there are others too).
Round robin has the downside of not really keeping everything fully balanced, it just allocates in rr fashion, and doesn't care what the load is, or how many processes are currently running, but it is robust. By the end of the day, things can at times get pretty much out of whack in number of users and cpu/disk loads at times.
Process-counting does a handsome job of keeping all of the number of processes correctly balanced - BUT - if you end up with a server process "partially up", then you can get into a bit of problem. We've seen the app servers die from time to time (though rarely), but not fully die (which would be preferred). What happens is you can set the port probes to check to see if a) http server, b) forms server or c) java servlet(s) are up, but you can't set the CSM to know if they are up enough to really do work for the users that are trying to connect (or are already connected to this server). What this means is that any of these servers are up enough to respond to port probes, but can't get you any meaningful data to your requests. What happens then is the users on that box, start backing out and the server farm knows that this box suddenly becomes the low-numbered proc count box, so... it starts sending ALL new requests to this partially up/down box. Meaning that, no-one can get logged in, etc. until the server processes on this server are bounced, or the offending server is taken inactive from the farm.
Which one to choose - you're probably OK with either, unless you're tight on cpu resources at peak periods of activity, in which case that kind of leaves you with process-counting only as your method.
Also, Oracle Apps processes are kind of funny in that they seem to want to bring components up and have them running on the box before bringing up other parts. And, it seems to want to know that it is working that way from its own box. Solution after many weeks of diagnosis: Set the real hostname in the /etc/hosts file of each machine to have the ip address of 127.0.0.1 instead of having it be the IP of one of the Lan cards (some types of UNIX flavors do this by default). You'll then have to go and fix the security (deny,allow) sections of the Apache and Jserv configs to accept connections from the loopback connector (127.0.0.1).
You'll also have to consider something we call "The Sticky" (we've now fully forgotten the real name for this, and now all agree on calling it this by this name). This denotes a client workstations affinity to connect to the same client server for work. This is a time based setting and you will probably want a minimum of 10 minutes or so, and up to one hour, depending on what works best for you.
This allows your client workstation to keep connecting to the same forms server (or java servlet) which is doing work on your behalf, and contains results that need to be pushed back to you. Setting this too short will result in nonsensical connections if someone stops working in a form and talks on the phone for a while (longer than the time setting). Setting this too long adversely affects how well you load balance.
Newest versions of the s/w for this also offer coookie injection as alternative to "The Sticky". Newer versions also offer load balancing based on response time from a dedicated process call that you setup and designate from the server and integrate into the CSM.
Proper usage of the features of the CSM module require the latest software releases in the switch - which require the latest OS software, which require the latest firmware in the supervisor blades, which can require later versions of sup blades themselves. With that in mind, the amount of memory in the switch to hold the newest versions of all software may not be big enough, and the older versions of the OS may not support the larger memory cards (meaning you can't half step your way through the upgrade). So - MAKE SURE that your network guy has asked and reasked everyone that he needs to, and then triple checked compatibility across all of these fronts, or you'll possibly be making multiple attempts at getting the hardware configured and started up.
Going through all of this to make it work is actually going to take more time than you are probably thinking, just because conceptually it just sounds so neat and simple. So allocate more than the "day-or-two" that you may be thinking of for this aspect of the project.
We are the people our parents warned us about --Jimmy Buffett
