Read Open VMS Filesystem on HP-UX

[Cory Gaetz(1)]

Hello All,

We have a project that needs to pull data (~300GB) from a VMS (v7.???) file system to an HP-UX (11.23) box.

We have attempted to use FTP but have encountered challenges around this method.

Are there any file systems that can be written to in VMS and read in HP-UX? I know NFS will work, but we want to see if there is anything that would not involve transfer over the network.

Barring that, are there any methods that others have used to accomplish this task? We have tried FTP, sFTP, scp.

Thanks for the time and help, let me know if I haven't profided sufficient detail.

[Steven E. Protter]

Shalom,

If the openvms system has NFS server, make the file system a NFS share pull the data like this.

This much data, however you are better off taring the data up and transferring it that way.

scp -rp might work, depending on if openssh server is installed on the VMS system.

SEP

[Cory Gaetz(1)]

Hi Steven,

Thanks for the response. NFS is an option, although one that we are trying to avoid.

With the data transfer we need to start moving individual files (1MB - 1GB) as soon as they are created on the VMS side, so tar unfortunately is not an option.

We tried scp but the overhead that it added caused the data transfer to take too long (think 4x-5x longer).

Thanks for the suggestions though. Anyone else?

[Steven Schweda]

I'm easily confused, but ...

> [...] we want to see if there is anything
> that would not involve transfer over the
> network.

Like what, tape?

> [...] VMS (v7.???) [...]

Perhaps:

TCPIP SHOW VERSION

> [...] we need to start moving individual
> files (1MB - 1GB) as soon as they are
> created on the VMS side [...]

Is this a one-time exercise, or repeated?

> We have tried FTP, sFTP, scp.

And didn't like what about each?

[Cory Gaetz(1)]

Thanks for the reply.

Both systems exist on the same XP12000. If there is a File System that both OS can read then we can simply mount the LUNs to both systems. Sorry for not explaining that earlier.

Unfortunately I do not have access to our VMS systems. So getting the exact version is challenging.

This job will have to run every night, starting at 11pm local and going until all files are copied.

The "s" protocols all added too much overhead to the copy, resulting in us running over the window to get the work done in.

We are currently using FTP but because of the complexity of the job (multiple files arriving at different times every night, pre-reqs that must be met, etc.) are looking for an alternate solution.

[Steven Schweda]

> [...] If there is a File System that both
> OS can read then we can simply mount the
> LUNs to both systems. [...]

I think that you may be doomed on that front.

> We are currently using FTP [...]

Should be relatively simple, hence quick.

> [...] the complexity of the job (multiple
> files arriving at different times every
> night, pre-reqs that must be met, etc.)
> [...]

It's not immediately obvious to me how a
different file transfer scheme would simplify
any of that.

If you're pulling the files from the HP-UX
side, but the VMS side is the one who knows
when they're ready, then it might make more
sense to push them from the VMS side,
instead.

> [...] I do not have access to our VMS
> systems. [...]

That doesn't exactly add flexibility.

[Bill Hassell]

The simple answer is that HP-UX cannot read a VMS filesystem, just like a PC can't read an AIX filesystem or Solaris can't read a MAC's disk. The filesystems must be translated into a common format which is what NFS (or CIFS/Samba) does. But NFS has a very high overhead and less than perfect network connections will cause endless problems.

ftp is by far the most efficient method as it provides adaptive and overlapping packet exchanges. But the standard ftp program does not handle recursion into subdirectories nor does it provide a skip feature when the destination file is already there.

A good solution is to use rsync which has all the features you might need including a link to use ssh rather than rsync connections. rsync also has compression capabilities.

Is the network local? For WAN connections, even rsync (and certainly NFS) will really bog down but ftp will run quite well. (WANs have long turnaround delays) If you use NCftp (does cost a few $$) then you inherit a lot of features including file-in-progress flags so a remote system monitor will know when the file is completely transferred.

[Steven Schweda]

> [...] the standard ftp program does not
> handle recursion into subdirectories nor
> does it provide a skip feature when the
> destination file is already there.

Wget can help with at least some of that, but
I don't know if any of that would actually
help in this case, because I don't know
enough about what's being transferred when,
why, whence, whither, and so on.

> A good solution is to use rsync [...]

It's a good solution to _some_ problems.
Even if it would do this job in principle,
finding rsync for VMS (or installing it
without "access to our VMS systems") could
prove problematic.


> The simple answer is that HP-UX cannot read
> a VMS filesystem, just like a PC can't read
> an AIX filesystem or Solaris can't read a
> MAC's disk.

All you need is software. Using mtools, my
VMS system can read/write a FAT32 file
system, for example. (Sometimes the simple
answer is the too-simple answer.)

[Cory Gaetz(1)]

Thanks for the replies so far folks.

Bill - unfortunately, like many companies, the "r" commands are forbidden here. I have looked at NCftp, as notification of a complete file transfer (or failure) would make our lives much easier. Thank you for that suggestion. Have you used these flags successfully?

Steven - I have been able to find an ODS2/5 reader for various flavours of Linux (and even BSD). This could lead to an intermediary step of mounting the ODS file system on a Linux box, then copying that to another file system and mounting that on HP-UX. Not ideal, but certainly getting closer.

As for no access to our VMS systems, those are handled by another group. Segregation of duties and all that jazz. So while I don't have direct access if something needed to get done it could be.

Thanks again.

[Mel Burslan]

Cory,

Do not discount rsync as another r-command. It is very flrxible and can operate on many different transports. We are successfully using it with an underlying ssh/scp connection. You might want to invest some time reading at least the man page and see if you can configure it in your environment using ftp or any other method, suitable to your needs.

I understand you wish that there was a way to mount the VMS created SAN volume onto your HPUX server but due to disk data structures being completely different, it is out of question. So, you will have to choose the lesser of evils for your task.

My 2 cents.

[Steven Schweda]

> Do not discount rsync as another r-command.
> It is very flrxible and can operate on many
> different transports. [...]

Are _you_ running an rsync server on VMS?
(Is anyone?)

> [...] I have been able to find an ODS2/5
> reader [...]

I always have a VMS system available, so I've
never felt the need, so I've never tried one.
(Is the existing code is large-file-capable?)

I'd guess that you'd want to grab a whole
disk image when the VMS system is idle, and
work with that. This sounds sub-optimal if
the VMS files are changing frequently. (And
I may be old, but 300GB still sounds like a
lot of bits to me.)

> It's not immediately obvious to me how a
> different file transfer scheme would
> simplify any of that.

Still true.

[Cory Gaetz(1)]

Here is a link to the ODS reader I found. Look under the various ODS2 directories.

http://www.digiater.nl/openvms/freeware/v80/

Thanks for the suggestion Mel, I have now read the rsync man page and it looks like a powerful tool. I will bring it up as an option.

Steven - You are right, 300GB is a lot of bits. This is part of an intermediary step in removing an old legacy system and modernizing it. You are also correct that grabbing an entire disk image is sub-optimal for the task at hand.

One of my co-workers is currently attempting a different method of using FTP for the transfer. The problem is not in the FTP transfer itself, it is in the correct communication of the ~1000 dependencies and sub-dependecies on when the original files are available, when they can be moved, and what happens after the move. Being able to mount the same file system to both OS eliminates many of the complexities of the current solution.

I'm not trying to be purposefully vague. There are parts of this I don't understand myself. I have been asked by the powers that be to look for alternate solutions.

Thanks again for all the responses.

[Bill Hassell]

NCftp is indeed quite powerful (and very fast). We transfer about 1-2 terabytes every day to a Windows box which connects to an OC12 (600 MBit link). The file-complete flag is either a prefix or suffix that you choose (such as .tmp). The remote end ignores any file with .tmp but forwards the renamed files as they are completed. NCftp will probably be a good fit and much, much faster than rsync. I could get reasonable performance with rsync on a local APA link (4 each GigE LAN ports using crossover cables) but I had to use jumbo frames (9000 bytes each) and at least 5-8 copies of rsync working on different files. With 2 or 3 NCftp sessions, I could hit almost 3 GB/sec throughput with the same 4 APA links.

Since I deal with several hundred TB of data every day, a few hundred GB is no big deal. But as your description of the problem has lots of restrictions, once you have the transport program working, the rest will be file coordination.

[Hein van den Heuvel]

>> Are there any file systems that can be written to in VMS and read in HP-UX? I know NFS will work, but we want to see if there is anything that would not involve transfer over the network.

It is possible to get something going in this space but it will be tricky. Some thoughts:
1) An ODS-2 reader like you already mention.
2) Use a DOS/FAT structured disk. On OpenVMS you can write those with 'mtools' software and/or the EXCHANGE system tool.
3) ZIP and RAW devices.! I'm surprised Steven did not mention this. ZIP (3.0) on OpenVMS refuses to write to RAW device, so first ZIP to a file (on RAMdisk in memory?) and then COPY the file to a disk mounted /FOREIGN. Unmount when done. On the HPUX side UNZIP from the raw device, if need be by piping from 'dd' specifying the appropriate block count matching the OpenVMS ZIP file.
4) Use an LD device on a file (also on http://www.digiater.nl/) to copy files onto, transfer underlying file with FTP, binary, use that ODS reader.. if you can get that to work.
5) BACKUP the target, transfer, Use a unix tool to read OpenVMS BACKUP files.
6) NFS

I suspect the trying to use an intermediate re-mountable disk is more throuble than it is worth. Just use (dedicated?) local Gigabit connection.

But really before you do any of that, it needs to be 100 % clear exactly what the files mean, hwo they hang together (pointers, directories), and how they can be used on the HPUX box.

A) Are they all simple ascii 'text' files? There is good hope! If they are 'STREAM_LF' record fomt, then they are directly useable. The default OpenVMS format is 'variable length records'. which can be converted via a tool, or just lest FTP ASCII do it?

B) Are they binary image files (PDF, JPG,..) there is some hope! FTP Transfer in BIN mode.

C) RMS Indexed files? Convert to flat text first.

D) Binary files with OpenVMS native floats and integers? Now it gets tricky.

Hope this helps some,
Regards,
Hein van den Heuvel ( at gmail dot com )
HvdH Performance Consulting

Note to 3), I just tried this but only VMS - VMS over raw device. Since OpenVMS has no native 'dd' with a max record count I just copied all block and manual set the end to make unzip (6.0) a little more happy.
I also used a quick DCL script to copy the data to a file to unzip. For a production tool you would of course write a quick RMS READ/WRITE tool in the language of your choice.

$mcr sysman io connect mda1: /driver=sys$mddriver /noadapter
$init/size=5000 mda1 hein ! RAMdrive
$mount/foreign mda1 ! Raw device ready.

$zip x.zip *.com ! Some sample files
$copy x.zip mda1:/log ! Out
$ conv/stat/fdl="rec; form fix; size 512" mda1: y.zip ! And back
$ set file/att=ebk=150 u.zip ! trim
$ unzip -l y.zip ! Presto


Sample DCL script to copy from raw to file:
$ type [-]copy.com
$i = 0
$if p3.eqs."" then p3=9999
$open/read/error=done in 'p1
$crea/fdl="rec; form fix; size 512" 'p2
$open/appen ou 'p2
$loop:
$read /end=done in record
$writ /symb ou record
$i = i + 1
$if i.lt.'p3 then goto loop
$
$close in
$close ou
$
$done:
$exit '$status

[Steven Schweda]

> [...] I'm surprised Steven did not mention
> this. [...]

Perhaps he's waiting for a clear explanation
of the requirements and constraints before
doing much more than complaining about the
suggestions made by others.


> [...] I have been able to find an ODS2/5
> reader [...]

Does it do ODS5? (Do you use ODS5?)

[Hein van den Heuvel]

In an earlier response I wrote:
3) ZIP and RAW devices.!

I got around to test this, and it works.
I don't think it is practical or useful, because try as I might I could get neither zip to write to a raw device or pipe ( --out, -, ... ) nor was unzip on hpux willing to read from a pipe (for me).
But it can be done without network transfers but still using staging files. IMHO, If the data is in a file anyway, then you might as well transfer that over a gigabit network.

Anyway...

I did no use a SAN drive, but a hot-swap brick.
On the sending (OpenVMS) side I used zip 3.0 staged in memory (mda1:)

$zip mda1:test [...]*.c [...]*.txt [...]*.com
$mount dka100: /foreign ! Real disk
$copy/log mda1:test.zip dka100: ! 3361 blocks
$dismount dka100:

Remove tray from my OpenVMS system, plug into my HPUX system. On HPUX i had to find unzip first. I downloaded and grabbed the executable from http://hpux.connect.org.uk/hppd/hpux/Misc/unzip-5.52/


$ioscan (actually used smh, to discover new drive)
$ioscan -funC disk
:
disk 6 0/1/1/0.1.0 sdisk CLAIMED DEVICE HP 36.4GST336704LC
/dev/dsk/c2t1d0 /dev/rdsk/c2t1d0
:

Check for reasonable data:

dd if=/dev/rdsk/c2t1d0 count=1 bs=512 | xd -c
0000000 P K 03 04 14 \0 \0 \0 \b \0 da bc F 5 e bd
0000010 w } b7 02 \0 \0 ae 05 \0 \0 \r \0 \t \0 C A
0000020 C H E _ F L U S H . C U T 05 \0 01
:
Looks good! ( :-) )
Transfer in reasonable sized chunks (100*512 bytes), with size as indicated by OpenVMS copy log, divided by 100 rounding up.

# dd if=/dev/rdsk/c2t1d0 bs=51200 count=34 of=vms.zip
# unzip vms.zip
Archive: tmp.zip
inflating: CACHE_FLUSH.C
:
inflating: UNZIP60/VMS/mod_dep.com

# ls -R
# cat

Presto! Files as there, in hierarchy, and readable.


For bonus points I doublechecked I could put a poor-man 'filing' system on the raw device.
That worked as well:

OpenVMS:

$ld create lda1:
$ld connect dka100: lda1:/lbn=(start=10000,count=10000)
$mount/for lda1:
$copy mda1:test.zip lda1: /log

Hpux:

# dd if=/dev/rdsk/c2t1d0 bs=51200 count=34 iseek=100 of=xxx.zip
# unzip xxx.zip


Like I said... I don't think it is useful, but it can be done.
If someone happens to know how to feed unzip from a pipe, or a raw device, then let me know I think I need the pipe, and a simple tool like DD in order to limit the block count. Without it you get:

# unzip -l /dev/rdsk/c2t1d0
Archive: /dev/rdsk/c2t1d0
End-of-central-directory signature not found. Either this file is not
a zipfile....

Cheers,
Hein.

[Steven Schweda]

> [...] I could get neither [...]

The basic Zip archive format/method typically
involves/requires seeking in the archive, so
pipes cause trouble. With Zip 3.0 and up,
there's a "-fd" option to enable the use of
"data descriptors", which might help. I've
tried not to think about this stuff when I
could, and if I ever knew anything
substantial about it, I seem to have
forgotten it. I'd need to study the source
and/or the archive format appnote to see
what's true. I'm even less sure about what
UnZip does. When desperate, the Info-ZIP
forums might turn up something (or someone
who's better informed).

http://www.info-zip.org/board/board.pl?