PC to VMS File Transfer

[ama_3]

We have an OpenVMS 8 that’s connected to four Windows XP machines over a gigabyte switch.

We use TCPIP to transfer files from our Windows XP systems over to the VMS box via WS FTP Pro (an FTP program)

Our date size ranges around 100GB-150GB of 100-200Kb files, which is close to about a million files on each transfer. We don't want to zip/tar these files.

Transfer rate is 350KB/sec, takes 3-4 weeks to fully transfer (assuming nothing goes wrong)

We tried to increase RMS pre-allocation size:

TCPIP$FTP_FILE_ALQ 20000
TCPIP$FTP_FILE_DEQ 20000

Transfer rate went down to 25KB/sec.

Where do we go from here?

[Hein van den Heuvel]

There have been a few prior writeup on FTP performance. Google! +FTP +OPENVMS

Check out the other TCPIP$FTP... logicals.
Notably WINDOWSIZE or some such, but that's for the network side.

Admittedly it sounds like a disk size issue.

See if (HWM) High-Water-Marking is enabled.

If the FTP used does not set the RMS Option FOP=SQO then HWM will may explain some slowness. Try with SET VOLUME/NOHIGH ...

You may also want to try using a RAMDISK to stage the incoming files, at least for a TIMING test.

$ mcr sysman io connect MDA0: /driver=sys$mddriver /noadapter
$ init/size ...

It may be illuminating to use
$ ANAL/SYS
> SET PROC/ID=
> SHOW PROC/RMS=(FAB,RAB,BDBSUM)

hth,
Hein.

[Richard W Hunt]

Your transfer process is no faster than the slowest link. Your links to check are:

1. Disk transfer rate, XP disk to XP RAM

2. XP/PC to Gbit switch, which depends on the settings of the XP system NIC

3. Actual transfer rate over the Gbit switch.

4. GBit switch to VMS, which depends on the settings of the VMS NIC.

5. Disk transfer rate, VMS RAM to VMS disk

Of these, the two places most likely to have screwed the pooch are the NIC settings, but I also wonder about how the destination disk or disks are laid out, geometry-wise and otherwise.

You don't say whether the GBit switch is basically used just like a fast Ethernet, but if it is, verify Full Duplex ops and verify that the NICs are set to the maximum speed.

For the destination disk(s), what is the cluster factor and is anything else to be stored on the disk(s)? A relatively small cluster factor with a large file on a fragmented disk will only SEEM to take nearly forever. In that case, it would be because you are thrashing through the bitmap allocation routines. A quick look at performance might help. Try

$ MON FILE_SYS

and look at the BITMAP rate.

$ MON IO

and look at the SPLIT TRANSFER rate.

If the disk is totally (and I do mean TOTALLY) dedicated to these files to the exclusion of all else, re-init the volume before you start the transfer process. This eliminates fragmentation issues as much as is humanly (machinely?) possible. If the disk is not dedicated, see if you can make it so. In which case you could init the volume with a larger cluster size than the defaults, which are oriented towards a lot of smaller files. When you know you will have really large files, you can make the cluster factor a lot larger.

[marsh_1]

hi,

3-4 weeks ! transfer to a cifs share ? why don't you want to zip the files ? do the files have to go over to the vms box - have you thought about using nfs ?
tar on windows to ftp or to tape ?

fwiw

[Hein van den Heuvel]

>> Our date size ranges around 100GB-150GB of 100-200Kb

So that's 200 - 400 blocks.
:
TCPIP$FTP_FILE_ALQ 20000
TCPIP$FTP_FILE_DEQ 20000

What possessed you to do that?!
If the typical output file is known to be 100x smaller?
Combined with HWM that would nicely explain the drop in performance:
- Gather 20,000 blocks
- Scatter 10MB of zeros to the disks
- Write 300 blocks
- Give back 19,700 blocks.

>> is close to about a million files on each transfer. We don't want to zip/tar these files.

Well, you may have to if performance is more important than convenience.

The way to make FTP perform is by NOT ACKing every block/file. You have to allow it to stream, which probably means zipping it up.

For that much volume, I would probably like my output to remain zipped up, but that's just me.

Zip it up in 500MB - 2000MB chunck. Transfer to RAMdrive, Unzip to final target. This will take the disk performance pretty much out of the picture. Both Zipping (except for the first file) and Unzipping (except for the last file) can be happening while a file is transferring over the wire.

>> >> assuming nothing goes wrong)

IMHO tranferring 100 large clumps would appear to be easier to monitor and recover from errors, than millions of little things.

Good luck!
Hein

[Andy Bustamante]

Are the 4 XP systems transfering to same target directory on the VMS system? Are the 4 XP systems transfering to same target disk on the VMS system? Are new file versions ;2, ;3 ;... being created? Does the target directory(s) contain other files?

OpenVMS stores directory listings in sorted ascending order. If you're adding files to a very large directory in random order, it will take longer than adding files sorted to append to end of the directory listing. Having 4 transfers into same directory could create a bottleneck.

Prior to VMS 7.2 very large was directory file over 128 blocks, with your version 8, very large will vary.

The set volume/nohighwater as previously pointed out can increase write performance. Setting this may allow a user to allocate and view the contents of deleted blocks. Your security policy will determine whether or not this is an acceptable option. VMS defaults to the more secure option.

Andy Bustamante

[Thomas Ritter]

Just to add my little bit, we perform similar amounts for transfer. We find zipping reduces files sizes by at least 90% ! Big difference. We prefer to use filezilla.