It took around 65 hours to create 1000 VxFS file systems. Any scope for improvement?

[Sushobhan Das_1]

The following setup was required to test performance of an agent that run on HP-UX.

Basically I created a HP-UX host (rx2620, with 2 X 1.6 GHz I2; 8GB RAM; 4 X A6826A) with 1000 LUNs from a EVA8k (one path from each of the two HSV210 controller) with 4 paths for each LUN. On those LUNs, I have created 1000 stripped volumes with each volume spread across 4 LUNs. I thought that I can create the whole setup in a day but it took around 5 days plus to run "/etc/vx/bin/vxdisksetup -f -i cXtXdX format=cdsdisk" on each disk. I did the whole setup using script (I mean creation of LUN, presenting them to the host, initializing the disk using VxVM, creating vxVM VG, etc.) and found that "initializing the disk using VxVM" is the bottleneck (took more time than I even imagined) during the whole process.

Does anyone know what I could have done to make to faster?

[Pete Randall]

Are you saying that you ran /etc/vx/bin/vxdisksetup 1000 times in a row? How about if you ran them all at the same time by putting them in the background?


Pete

[Sushobhan Das_1]

Yes, I run the "vxdisksetup" command 1000 time through a script.

[Pete Randall]

And each execution of the command had to wait for the previous one to finish? It makes sense to me that it would take quite some time in a scenario like this. I would try to run the commands simultaneously to see if that would speed things up.


Pete

[Sushobhan Das_1]

Actually I too thought so and tried with so result. I tried running "vxdisksetup" for two device files from two telnet sessions but time taken is equal to what it whould have take if i had run them sequentially.

[Ralph Grothe]

In your subject line it says creation of VxFS took such long.
But in the thread you refer to disk initialisation by vxdisksetup.
I can only compare from VxVM use on Solaris boxes.
There the initialization (i.e. the vxdisksetup) only takes a tick per disk.
Could it be your script was dangling in some timeout break?
But even the creation of the filesystem shouldn't take much longer.
But I have to admit that I never used the format=cdsdisk attribute but always sticked to format=sliced since we don't need to swap these disks between say Solaris and HP-UX hosts, and the sliced layout is the preferred for Solaris (maybe some BSD labelling legacy?)

[Duncan Edmonstone]

IS there any particular reason why you created 1000 LUNs?

Why not just 1 or 2 LUNs, and then 1000 volumes?

The EVA is going to be striping across all the disks in its disk group anyway, so striping at the OS level gains you nothing. 2 LUNs would make sense as you can prefer one to each controller and ensure IOs go to each, but more than this doesn't make much sense to me.

HTH

Duncan

[Pete Randall]

Must be I/O contention then. The man page says that vxdisksetup is not generally called directly because it is called by the vxdiskadd command. Did you already do vsdiskadd? Are you sure you need to run vxdisksetup?


Pete

[Ralph Grothe]

Btw, in your script did you signal the VxVM config daemon of the existence of new VxVM disks after you ran the vxdisksetup but before you ran say vxdg adddisk?
Usually one needs to re-issue a "vxdctl enable" to make the daemon aware of the new disks.

[Sushobhan Das_1]

Let me reply in one shot -

My ultimate goal is to create 1000 VxFS file system created out of 1000 VxVM stripped volumes, each volume stripped across 4 EVA disks. This is meant to simulate a customer scenario. The customer has 16k LUNs exposed to a HP-UX Superdome from an XP12k each LUN have 4 paths. I was asked to create what I created so that we can reach close to customer's config. I created 1000 from a single EVA 8k and all LUNs were RAID 0 vdisk. This is because of space constraint in my EVA. So...that's the backgroud. This is a test config and I am not interested in security/availablily of the data.

I wrote a very simple "vxdisksetup" script and is attached here.

I didn't run "vxdiskadd" command. I am not particular about "vxdisksetup" command. Any command that can initialize (make it ready) the disk for further configuration will do.

[Ralph Grothe]

Maybe as I suspected,
a bit too haevy for the VxVM config daemon at one shot.
Try inetrspersing the vxdctl enable call maybe after every 100th vxdisksetup.

[Sushobhan Das_1]

Can you please tell me the command that I should be running for this?

[IT_2007]

You don't have to call "-f" option in the vxdisksetup command unless it was used previously. If it is a new LUN then use

vxdctl enable

and then run for loop with the following:

vxdisksetup -i /dev/dsk/cXXtYYdZZ

woule be enough to initialize the LUNs and wouldn't take much time. For testing purpose, run for loop for 10 LUNs and see how it goes.

[IT_2007]

To make it fast access to SAN devices whether to initialize or something else, you can issue the following command:

vxdctl cacheenable

cacheenable
Requests that caching of device opens are enabled in the
VxVM kernel module.

At the VxVM configuration discovery time a disk device may
be opened and closed multiple times. Depending on the disk
array(s) being used, multiple opens of the same device could
delay the startup time of the vxconfigd significantly. To
minimize the delay in multiple opens, the first open of the
device is cached and used for subsequent opens.

When the machine is booting, the caching is automatically
enabled to speed up the boot process. Once the machine has
been booted, the caching is disabled via the VxVM startup
script.

If caching is enabled and the vxconfigd has been started and
you do not plan to restart the vxconfigd or re-enable the
VxVM configuration, then the device caching should be
disabled by using the cachedisable.

[Srikanth Arunachalam]

Hi,

You can put it in loop. Just take note of the following information as well.
(1) If it is cluster group of system accessing common area, ensure that you pause after each creation of file system and the vxvm is enabled only from one system.

(2) Before you create file system have pre-checks to find out if the mount points are already created.

(3) Take a vxvm backup using dgcfgbackup and
vxconfigbackup after you perform in one system. So that restoration of backup will be sufficient to create the filesystem.

Thanks,
Srikanth

[Alzhy]

I find it hard to believe it takes that long to carve your EVA8K under VxVM. Can you please post the commands you used it its entirety to wit:

1. Initialisation of the disks
2. Creation and addition of these disks to a Diskgroup
3. Creation of your so called "1000 striped volumes"

and Drumroll...

4. Creation of your "Filesystems"?


What I could have done is as follows:

1. vxdctl enable (if the disks are newly presented)

2. create a list of all these disks (primary path of course -- which are the lowest cX number of the 4 paths for each lun)

3. Initialize them:
for disk in `cat DISLIST`;do
/etc/vx/bin/vxdisksetup -i $disk
done

4. Create DG

vxdg init perfdg disk1=cXtYdZ (from the list)

5. Add the disks to the DG
n=1
for disk in `cat DISKLIST|grep -v cXtYDZ`;do
n=n+1
vxdg -g perfdg adddisk disk${n}=$disk
done


6. Create your striped volumes:

n=0
until n=1000;do
n=n+1
vxassist -g perfdg create vol$n XXXg layout=striped ncols=4
mkfs -F vxfs -o bsize=nnnn,largefiles /dev/vx/rdsk/perfdg/vol$n
done

I reckon all these should be done in under 2-3 hours.

[Sushobhan Das_1]

Honestly, I also didn't believe that it took so much time.

I am going get the setup back sometime next week. I will redo the whole thing once again, or may be couple of times. There are couple of suggestions. I would like to try so that I don't face this issue in future.

[Alzhy · Accepted solution]

Das,

One possible contributory factor to your "slow" experience is you probably do not have the Array Support Library for EVA Installed..

Can you check?

# swlist |grep "Array Support"

[Sushobhan Das_1]

Forgot mention the OS. Its 11.23 PI and VxVM version is 4.1.

With this config, as per Veritas support ASL comes in-built into VxVM. Isn't it?

[Alzhy]

I honestly do not know.. we are stuck qith VxVM 3.5 as we cannot go beyond HP-UX 11.11 yet...

In your system startup, does VxVM spew out messages tho the effect that it "recognizes" a licensed array or something?