Read Open VMS Filesystem on HP-UX

[Steven Schweda]

> Do not discount rsync as another r-command.
> It is very flrxible and can operate on many
> different transports. [...]

Are _you_ running an rsync server on VMS?
(Is anyone?)

> [...] I have been able to find an ODS2/5
> reader [...]

I always have a VMS system available, so I've
never felt the need, so I've never tried one.
(Is the existing code is large-file-capable?)

I'd guess that you'd want to grab a whole
disk image when the VMS system is idle, and
work with that. This sounds sub-optimal if
the VMS files are changing frequently. (And
I may be old, but 300GB still sounds like a
lot of bits to me.)

> It's not immediately obvious to me how a
> different file transfer scheme would
> simplify any of that.

Still true.

[Cory Gaetz(1)]

Here is a link to the ODS reader I found. Look under the various ODS2 directories.

http://www.digiater.nl/openvms/freeware/v80/

Thanks for the suggestion Mel, I have now read the rsync man page and it looks like a powerful tool. I will bring it up as an option.

Steven - You are right, 300GB is a lot of bits. This is part of an intermediary step in removing an old legacy system and modernizing it. You are also correct that grabbing an entire disk image is sub-optimal for the task at hand.

One of my co-workers is currently attempting a different method of using FTP for the transfer. The problem is not in the FTP transfer itself, it is in the correct communication of the ~1000 dependencies and sub-dependecies on when the original files are available, when they can be moved, and what happens after the move. Being able to mount the same file system to both OS eliminates many of the complexities of the current solution.

I'm not trying to be purposefully vague. There are parts of this I don't understand myself. I have been asked by the powers that be to look for alternate solutions.

Thanks again for all the responses.

[Bill Hassell]

NCftp is indeed quite powerful (and very fast). We transfer about 1-2 terabytes every day to a Windows box which connects to an OC12 (600 MBit link). The file-complete flag is either a prefix or suffix that you choose (such as .tmp). The remote end ignores any file with .tmp but forwards the renamed files as they are completed. NCftp will probably be a good fit and much, much faster than rsync. I could get reasonable performance with rsync on a local APA link (4 each GigE LAN ports using crossover cables) but I had to use jumbo frames (9000 bytes each) and at least 5-8 copies of rsync working on different files. With 2 or 3 NCftp sessions, I could hit almost 3 GB/sec throughput with the same 4 APA links.

Since I deal with several hundred TB of data every day, a few hundred GB is no big deal. But as your description of the problem has lots of restrictions, once you have the transport program working, the rest will be file coordination.

[Hein van den Heuvel]

>> Are there any file systems that can be written to in VMS and read in HP-UX? I know NFS will work, but we want to see if there is anything that would not involve transfer over the network.

It is possible to get something going in this space but it will be tricky. Some thoughts:
1) An ODS-2 reader like you already mention.
2) Use a DOS/FAT structured disk. On OpenVMS you can write those with 'mtools' software and/or the EXCHANGE system tool.
3) ZIP and RAW devices.! I'm surprised Steven did not mention this. ZIP (3.0) on OpenVMS refuses to write to RAW device, so first ZIP to a file (on RAMdisk in memory?) and then COPY the file to a disk mounted /FOREIGN. Unmount when done. On the HPUX side UNZIP from the raw device, if need be by piping from 'dd' specifying the appropriate block count matching the OpenVMS ZIP file.
4) Use an LD device on a file (also on http://www.digiater.nl/) to copy files onto, transfer underlying file with FTP, binary, use that ODS reader.. if you can get that to work.
5) BACKUP the target, transfer, Use a unix tool to read OpenVMS BACKUP files.
6) NFS

I suspect the trying to use an intermediate re-mountable disk is more throuble than it is worth. Just use (dedicated?) local Gigabit connection.

But really before you do any of that, it needs to be 100 % clear exactly what the files mean, hwo they hang together (pointers, directories), and how they can be used on the HPUX box.

A) Are they all simple ascii 'text' files? There is good hope! If they are 'STREAM_LF' record fomt, then they are directly useable. The default OpenVMS format is 'variable length records'. which can be converted via a tool, or just lest FTP ASCII do it?

B) Are they binary image files (PDF, JPG,..) there is some hope! FTP Transfer in BIN mode.

C) RMS Indexed files? Convert to flat text first.

D) Binary files with OpenVMS native floats and integers? Now it gets tricky.

Hope this helps some,
Regards,
Hein van den Heuvel ( at gmail dot com )
HvdH Performance Consulting

Note to 3), I just tried this but only VMS - VMS over raw device. Since OpenVMS has no native 'dd' with a max record count I just copied all block and manual set the end to make unzip (6.0) a little more happy.
I also used a quick DCL script to copy the data to a file to unzip. For a production tool you would of course write a quick RMS READ/WRITE tool in the language of your choice.

$mcr sysman io connect mda1: /driver=sys$mddriver /noadapter
$init/size=5000 mda1 hein ! RAMdrive
$mount/foreign mda1 ! Raw device ready.

$zip x.zip *.com ! Some sample files
$copy x.zip mda1:/log ! Out
$ conv/stat/fdl="rec; form fix; size 512" mda1: y.zip ! And back
$ set file/att=ebk=150 u.zip ! trim
$ unzip -l y.zip ! Presto


Sample DCL script to copy from raw to file:
$ type [-]copy.com
$i = 0
$if p3.eqs."" then p3=9999
$open/read/error=done in 'p1
$crea/fdl="rec; form fix; size 512" 'p2
$open/appen ou 'p2
$loop:
$read /end=done in record
$writ /symb ou record
$i = i + 1
$if i.lt.'p3 then goto loop
$
$close in
$close ou
$
$done:
$exit '$status

[Steven Schweda]

> [...] I'm surprised Steven did not mention
> this. [...]

Perhaps he's waiting for a clear explanation
of the requirements and constraints before
doing much more than complaining about the
suggestions made by others.


> [...] I have been able to find an ODS2/5
> reader [...]

Does it do ODS5? (Do you use ODS5?)

[Hein van den Heuvel]

In an earlier response I wrote:
3) ZIP and RAW devices.!

I got around to test this, and it works.
I don't think it is practical or useful, because try as I might I could get neither zip to write to a raw device or pipe ( --out, -, ... ) nor was unzip on hpux willing to read from a pipe (for me).
But it can be done without network transfers but still using staging files. IMHO, If the data is in a file anyway, then you might as well transfer that over a gigabit network.

Anyway...

I did no use a SAN drive, but a hot-swap brick.
On the sending (OpenVMS) side I used zip 3.0 staged in memory (mda1:)

$zip mda1:test [...]*.c [...]*.txt [...]*.com
$mount dka100: /foreign ! Real disk
$copy/log mda1:test.zip dka100: ! 3361 blocks
$dismount dka100:

Remove tray from my OpenVMS system, plug into my HPUX system. On HPUX i had to find unzip first. I downloaded and grabbed the executable from http://hpux.connect.org.uk/hppd/hpux/Misc/unzip-5.52/


$ioscan (actually used smh, to discover new drive)
$ioscan -funC disk
:
disk 6 0/1/1/0.1.0 sdisk CLAIMED DEVICE HP 36.4GST336704LC
/dev/dsk/c2t1d0 /dev/rdsk/c2t1d0
:

Check for reasonable data:

dd if=/dev/rdsk/c2t1d0 count=1 bs=512 | xd -c
0000000 P K 03 04 14 \0 \0 \0 \b \0 da bc F 5 e bd
0000010 w } b7 02 \0 \0 ae 05 \0 \0 \r \0 \t \0 C A
0000020 C H E _ F L U S H . C U T 05 \0 01
:
Looks good! ( :-) )
Transfer in reasonable sized chunks (100*512 bytes), with size as indicated by OpenVMS copy log, divided by 100 rounding up.

# dd if=/dev/rdsk/c2t1d0 bs=51200 count=34 of=vms.zip
# unzip vms.zip
Archive: tmp.zip
inflating: CACHE_FLUSH.C
:
inflating: UNZIP60/VMS/mod_dep.com

# ls -R
# cat

Presto! Files as there, in hierarchy, and readable.


For bonus points I doublechecked I could put a poor-man 'filing' system on the raw device.
That worked as well:

OpenVMS:

$ld create lda1:
$ld connect dka100: lda1:/lbn=(start=10000,count=10000)
$mount/for lda1:
$copy mda1:test.zip lda1: /log

Hpux:

# dd if=/dev/rdsk/c2t1d0 bs=51200 count=34 iseek=100 of=xxx.zip
# unzip xxx.zip


Like I said... I don't think it is useful, but it can be done.
If someone happens to know how to feed unzip from a pipe, or a raw device, then let me know I think I need the pipe, and a simple tool like DD in order to limit the block count. Without it you get:

# unzip -l /dev/rdsk/c2t1d0
Archive: /dev/rdsk/c2t1d0
End-of-central-directory signature not found. Either this file is not
a zipfile....

Cheers,
Hein.

[Steven Schweda]

> [...] I could get neither [...]

The basic Zip archive format/method typically
involves/requires seeking in the archive, so
pipes cause trouble. With Zip 3.0 and up,
there's a "-fd" option to enable the use of
"data descriptors", which might help. I've
tried not to think about this stuff when I
could, and if I ever knew anything
substantial about it, I seem to have
forgotten it. I'd need to study the source
and/or the archive format appnote to see
what's true. I'm even less sure about what
UnZip does. When desperate, the Info-ZIP
forums might turn up something (or someone
who's better informed).

http://www.info-zip.org/board/board.pl?