Data migration: Windows to HP-UX

[George Nimmer]

Greetings,
I have a need to move ~1TB of files from a Windows server to an HP-UX server. Has anyone here performed this kind of data migration? If so, what methods or tools did you employ? Both systems are SAN connected, so goal is to avoid using the Ethernet if possible. Something like Tivoli SANergy seems promising, has anyone used it? Thanks for your thoughts on this!!

[George Nimmer]

Oops I meant to post this question under 'System Administration'. Moderators, feel free to move it. Thanks!

[Tim Nelson]

Other than finding an NTFS to HPUX converter your only bet is either FTP or a SAMBA share.

Either way both are going to be over Ethernet.

[Bill Hassell]

Although there are complex SAN-to-SAN data mirroring options, these are not for the timid... But the connection method may be trivial compared to the effort needed to convert the data between two unrelated operating systems. If your data files are primarily ASCII, you'll need to convert the Windows file format into HP-UX format (see dos2ux man page and the ASCII option for ftp). If the data is managed by a database, the internal format may or may not be portable to the same database engine on HP-UX so you'd be looking at a db-unload and reload.

[Steven Schweda]

"files" is really not an adequate description
of the data being moved. 1TB might be more
than I'd try to move at one time, if I were
using conventional methods (FTP, ...).

ASCII FTP (or one of the X2Y programs) lets
you adjust line endings on text files, but
you need to know on which files to use it,
and which not to wreck that way.

Zip+UnZip has some text-file recognition
and conversion capability, which could be
useful in some cases. Zip 3.0 and UnZip 6.0
offer large-file capability, too, but I'd
tend to bite off less than 1TB at a time when
using those programs (especially if temporary
storage is constrained).

A more detailed description of the
requirements would be a good place to start,
however.

[George Nimmer]

Thanks all for the replies. I'm not concerned about file translation issues. It's all compressed binary data and has already been verified as good when copied to the target system.

The files are all small (~500 kb) and there are literally millions of them. ~1 TB worth.

Going over Ethernet is too slow and not an option. We are looking at utilizing the higher bandwidth of the Fibre Channel SAN to facilitate this. Some options we are evaluating:

1. Tivoli SANergy. It's a cross-platform NFS driver that routes the data traffic over fibre channel for much higher throughput than traditional NFS-over-ethernet.

2. Inserting some intermediate server that is able to read/write filesystems understood by both Windows and HP-UX. Perhaps a SAN attached Linux server.

3. Some solution based on SAMBA

Thoughts?

[Heironimus]

#2 would require one server that can handle the disk structures for both source and target. If you use VxVM on HP-UX, don't use dynamic disk on Windows, and license Veritas Storage Foundation for Linux, then Linux might be able to play middleman - but that's a lot of "if" and I'd want to spend some time talking to Veritas before trying it. I suppose you could write a GNU tar archive of the data to a raw LUN, but the higher speed of fibre might be countered by the extra copying on the SAN.

I don't see how #3 would help. Samba doesn't really do anything with your underlying storage. File copies would still go over the network.

I have never heard of the Tivoli product before.

[Rita C Workman]

I didn't notice what kind of "disk" they are, so I might ask, Can you simply attach similar SAN disk to the Windows disk by doing a disk-sync or mirror-disk and then split them back off and mount them up on the HPUX box?

For myself, using EMC storage, I'd just use the 'associate' command to the join the disks and use a symmir command to mirror or sync the data on them. Then run a symmir split command and voila your data is copied (disk-2-disk, no network traffic) and then take those new disks and mount them up where I need them.

If you have different SAN storage, see if they have similar options.

Rgrds,
Rita

[George Nimmer]

Heironimus, I like your tar-to-raw-device idea, I'll do some testing on that one...

Rita, SAN disk is an EMC Clariion. Yes, we can present storage from this array to either HP-UX or Windows. The problem with cloning the volume is that HP-UX cannot mount an NTFS filesystem.

[Olivier Masse]

Under Linux or one of the BSDs, I'd consider mounting the NTFS filesystem read-only and copy to another native filesystem. But unfortunately, AFAIK there's no third party software that lets you mount NTFS volumes on HP-UX.

If the servers are near one from another, personally I would just plug a crossover ethernet cable between two cards, set them to a private network address and use FTP. If that's not possible and there's just a switch between them, it doesn't matter much as long as your portgroups are not too much oversubscribed.

FTP runs almost at wire speed, so at 1Gb/s we're talking about around 3 hours to transfer 1TB which is not that long (if my calculations are correct), when compared to another fancier solution that involves installing a third party software.

Since you're copying many files, rsync might be an interesting option... if will take a while for the transfer to start initially, but if it is interrupted, you will at least be able to restart where you were.

Good luck

[Bill Hassell]

I would agree on a crossover cable with 1Gbit LAN cards. The wire speed is 100 MBytes but with acknowledgments and typical packet overhead, you can expect about 70 MB/sec. But be sure to set Jumbo frames (9000 byte packet) if the Windows card supports it. 3-5 hours should get the job done. SAMBA/CIFS is just another network filesystem with a *lot* more overhead than ftp. Be sure to select BINARY for the transfers.

[Heironimus]

If you do the crossover cable method then tar + netcat might be faster than FTP with a very large number of small files. But it's hard to say without actually testing.

[George Nimmer]

The problem with FTP is that with millions of small files, it becomes very slow. We've already tried FTP (on dedicated gig-e interfaces) and at that rate it would take ~6 days to copy all the data. If it were large files, I'm certain FTP would perform much better.

Our tar to raw SAN lun test worked nicely. We used cygwin on the windows server and simply tar'd a large test directory tree to the raw SAN lun (no partitions, no filesystems). Then we changed the presentation of the lun to the HP-UX server and tar xvf the device file and sure enough, there was all our data!! Even though it requires copying everything twice, it was still *much* faster than FTP.

I think we're going to take the tar-to-SAN approach. We can write some simple shell scripts on the HP-UX box to set all the ownership and permissions and whatnot.

Thanks all for your ideas and input!