Array Setup and Networking
Showing results for 
Search instead for 
Did you mean: 

Re: Need help setting up Zimbra email system on Ubuntu Linux/vSphere 5.5

Go to solution
Occasional Contributor

Need help setting up Zimbra email system on Ubuntu Linux/vSphere 5.5

Hi All,

I could really use some help in designing a solid architecture for a Zimbra email system, hosted by Ubuntu Linux (64bit) running on vSphere 5.5, that will allow rapid recovery from failures, and can later be expanded to allow HA.  The plan from the beginning is to enable Vmotion for guests and storage, and to use Nimble snapshots for DR.

Our Nimble system is a CS460G-X2 with an ES1-H65 expansion shelf, and we're not using the Nimble exclusively for our email.  We're using it for many different application, database, and file server purposes as well.  The unit is dual-connected into a pair of 10Gb switches using iSCSI.  We have roughly1400 users on our Zimbra system, and once this migration/upgrade project has completed, we will have a 3 VM guest email system, where each guest will be providing ldap, mta, and mailbox services.

Each of the VM guests will be set up with 3 VMFS datastores, each on its own Nimble volume.  Those 3 datastores will be used for:

a) "Root & boot" linux volume - a small ext2 /boot partition, and on an LVM partition, the swap device and an ext4 root file system

b) VMware will create 3 "hard drives" on this Nimble volume, and these 3 drives will make up a striped LVM, ext4 file system.  This file system will host the Zimbra email system

c) A "backup" volume, an ext4 file system which will hold 1 week of Zimbra email system backups

Do we need to set up the VMware hard disks as independant?  IIRC, independent disks don't participate in  VMware shapshots.  If we do make the disks independant, should we just avoid VMware snapshots, instead using Nimble snapstots on all the Nimble volumes to allow us to roll changes back (like from a failed patch attempt?)  Or should we just leave it as default, use VMware snapshots, and delete the snaps once the change proves successful, or roll-back to the earlier snap?  In our testing, we were not able to revert to an earlier Nimble volume snapshot if there were any VMware snapshots on the guest.

For DR purposes, we're planning on using the Nimble snapshot capability and will take a snapshot of the "root & boot" and "backup" Nimble volumes once a week, but the Zimbra volume will have hourly snapshots.  (The Zimbra backups are also archived to a different storage system, where they are held for 3 weeks, but that is outside the scope of this question.)

Unfortunately, the VM guests can only have a maximum of 4 VMware Paravirtual scsi adapters (I was really wishing for 5).  So here is one part of my dilema:

Do I use 3 SCSI adapters, 1 for the "root & boot," 1 for the LVM Zimbra system (all three VMware hard disks on a single HBA,) and 1 for the backup system,


Do I use 4 SCSI adapters, 1 for the "root & boot," as well as the backup file system, and the other 3 for each of the 3 VM "hard disks" I'll be creating on the data store for the Zimbra system.  The reason for keeping the Zimbra system and backup file systems on different HBAs  is that otherwise, backups will take forever.

or is there a better way altogether?

I know that Linux creates a separate queue for each physical block device (Is this still true if the disk i/o scheduler is controlled by VMware vs Linux, as per setting the elevator setting to "noop?")  Are there any advantages or disadvantages to having a single disk device on its own channel in a striped LVM environment, as opposed to having all 3 drives on a single disk channel when you're working in a virtual environment?  From past experience, there were great performance advantages in having multiple channels for the disk i/o in hardware server settings.

As I'm something of a noob when it comes to vSphere, and a total newbie when it comes to the Nimble storage, I'm hoping that someone might be able to give me some helpful hints here, or say that I'm just plain wrong (and if so, please make a suggestion! :-) .)  Please feel free to ask any questions if I wasn't clear on anything.

Finally, is there a good reference or references in coming up to speed in working with vSphere and SANs, in particular, with the Nimble.  I just downloaded the "VMware on Nimble Storage-BPG."  I've got a long history with working on networks and servers (Windows, Linux, & other *ix), and I've been in IT since 1989.  Most of my experience with VMware over the years has been with VMware Workstation.

Thanks in advance,


Esteemed Contributor

Re: Need help setting up Zimbra email system on Ubuntu Linux/vSphere 5.5

Hey Mark,

     There are a ton of suggestions and options around this type of deployment. 

1)     The easy and most effective way would be to make every volume a datastore and have VMware handle HA for machine failure.  You can still partition and do three LUNs across three datastore, but it just as simple to use one.

2)      The second thing to add would be SRA to enable site failover using Nimble's amazing replication and integration with SRA.  This would enable the ability to fail an entire site over at the push of a button or event.

3)     Putting everything into a hyper visor allow you to move hardware.  What your going to find is at some point the hardware is out of date, failed, old, too much to maintain, and such. SO lets make it a VM and a simple migrate live for the future.

4)     Talk directly with one of Storage Experts - I have forwards your post to the Principle architects at Nimble and they hopefully will either contact you directly or via this post shortly.

If you don't use VMware to house the entire environment, then each LUN should have at least two paths and on two subnets.  This is a requirement for direct Linux connections to used both paths for load balancing. If you use a single subnet, Linux will only use them as a fail-over pair.


Occasional Contributor

Re: Need help setting up Zimbra email system on Ubuntu Linux/vSphere 5.5

Thank you very much!

One thing I played around with was loosely based on a BPG I read from Nimble, which involved using iSCSI initiators within the VM, and "talking" directly to the Nimble, but that brought up issues with Vmotion, and VMware flat-out told me that I'd be completely on my own if I tried doing it, but that I should let them know if it worked! :-)

Once we have the VMware infrastructure available at our "hot site" we will be using SRM/SRA.  Thanks for the recommendation.

Thank you for forwarding this on to the architects, I really appreciate it.  As I said, I'm still pretty new when it comes to Nimble Storage.

Thanks again!


Trusted Contributor

Re: Need help setting up Zimbra email system on Ubuntu Linux/vSphere 5.5

Mark, I used to administer a Zimbra system with over 1,000 mailboxes in a VM environment, so I'm pretty familiar with what you are trying to do.

Now to the VMware and storage discussion. I don't believe it's necessary to have multiple SCSI adapters. Each virtual SCSI HBA can support up to 15 devices. Also for the OS, I would use just a single VMDK and use LVM to partition off the base filesystems. The backup should be done via NFS to a central repo for all mailbox nodes, so that just leaves the disks for the repo. This is how I've done it in the past and we never had a problem with backup windows even with over 1,000 mailboxes that were large, some 10s of gigs in size. It is true that you can get better performance with the linux IO scheduler by having multiple physical devices as mentioned in our BPG. This can be done simply with VMDKs just as easily as the iSCSI initiators. Since you plan on using SRM, you will need to keep these all as VMDKs or RDMs for SRM to understand the volumes. But for clarity sake using in guest iSCSI initiators does not interfere with vMotioning. The iSCSI traffic will move between hosts during a vMotion just like any other network traffic.

I would recommend that you use Nimble snapshots for data protection, but that doesn't mean that you can't use VMware snapshots too. You mentioned having problems mounting a VMDK if there were VMware snapshots existing. When VMware takes snapshots it creates additional VMDK files with a serial number. That is the disk that contains the delta changes since the snapshot was taken and the one you should attach when recovering from a clone. If you run into issues around this again you can contact Nimble support to have them walk you through the steps.

One more point, last time I checked Zimbra's HA deployment design was only supported with RHEL clustering and not Ubuntu. So if you plan on doing native HA outside of VMware in the future you may want to think about that now.