You can't. You have two VIPs, but only one iSCSI gateway gets assigned per target/volume per storage array cluster. Once the first ESX server (or Linux server) comes up, scans whatever VIP in lands on first and is told what storage node to use as the iSCSI gateway for that target/volume, that is the target/volume that ALL of the servers accessing that volume will use. This is a big point of confusion, the docs aren't 100% clear, and I'll admit to not being 100% happy with how pre-sales described the behavior.
My best guess, confirmed informally with lefthand support, is this has to do with SCSI-2 reservations, lock contention and making sure the writes happen in the correct order.
Windows, if you're using the Lefthand multipathing solution, is the only platform that works as you described above, since it doesn't have a single gateway for any given iSCSI target. Note in the multisite SAN docs they refer to "Application Servers" in certain places. That's Windows.
You can get yourself in a trap. Given that the VMware iSCSI initiator doesn't support routing, if you only have one adapter in each iSCSI subnet, a link failure (cable, NIC, switch reboot) will cause that server to lose the storage in one subnet or the other. You'd need to configure failover adapters on each vmk iscsi portgroup - if you've got two subnets in a single vlan, you could easily use the primary vmknic for one portgroup as a standby adapter for the other. Or you can double your vmknic count and use two per subnet, each connected to a different switch.
Or you can manually define gateways in the service console OS on an ESX server so the iSCSI traffic can be routed. That may not work in ESX 5, and if they don't support having multiple gateways for vmkernel interfaces, you'd be sunk. Also, there's a lot of advice out there about not routing iSCSI.
I just went through a series of calls with HP support working this all out over the past few weeks when I realized it wasn't quite working the way I expected it to. They've actually recommended that I use a single subnet/single VIP configuration, since almost every other failure mode (servers, links, switches, etc) is more likely than an entire site failure. And a site failure is only a problem if the VIP happens to be on that site at any given point. I've noticed in practice that if you change a storage node's IP config (I had to change the gateways at one point), the VIP seems to move very quickly.
Since I already have the storage nodes configured and they're in A/B/A/B/A/B/A/B order in the cluster, my replication pairs are already established. I should be able to just edit the IP address and gateway on each storage node I want to change, and delete the second VIP. I just need to shut everything down to do it.
