Read Mainframe data directly from (?shared) disk

[Massimo Bianchi]

HI All,
one of my customer has a nice architecture:

ERP on Mainframe, Datawarehouse on HPUX.

Every day it downloads 2TB of data from Host to HP to upload new content for analisys.
There are almost 1.000 sequential files.

He's asking on how to speed up this process, currently taking 20hours.

We had a couple of ideas:
1) create a zlinux partition, have the data transfer internally on MF, using FICON directy to some rade/write disks, and present these disks read only to HPUX.

QUESTION: is there a common fs readable by hpux ? maybe a driver for ext3 for hpux ?

2) Do you know of any FICON plug connector for HPUX ?

3) other ideas ?

Thanks and regards,
Massimo

[Steven E. Protter]

Shalom,

I don't see the mainframe model or operating system. Not that this would help me very much.

I am told that there is some form of NFS for mainframes, also Samba ports.

In such a scenario, the shares could be made available for mount.

The cause of the slow transfer should be looked at, perhaps its the network speed between the systems.

These links are shots in the dark:
http://www.linuxquestions.org/questions/linux-general-1/recover-tape-in-hpux-fbackup-format-587854/

http://www.zdnetasia.com/itlibrary/servers-and-server-os/0,3800009948,41175742p,00.htm

SEP

[Massimo Bianchi]

Hi,
Should be z/OS 1.8 / 1.9 . I have no information regarding model.

I looked at the links, but there were no usefull hints. fback/frecover or tar are not viable....

[OldSchool]

how is the data being transmitted today? you need to be aware that some ibm hardware used to (and maybe still does) use EBCDIC encoding instead of ASCII.

if that's true, and the transport mechanism used today does the translation (like ftp might / should) then a shared disk might be problematic.

also, are these ~1000 files all new data? or are only some of the files new / modified and they are all getting sent?

if the later, look at something like rsync to transfer only the changes.

without a better understanding of what's being done today & why its done the way it is, I've got nothing more to offer

[Massimo Bianchi]

HI,
all the 1k files are created as new every day. Up to now we have no way to change this, Host is very conservative on their procedure. If they work, they won't change it.

Now the transfer is done via FTP, that is slow. This is the reason they are going towards shared disks.

Speaking the truth there are a couple of solutions, but the vendor (EMC Infomover, HDS UPS) is not promising a long time support, so we alooking for alternatives.

Rgards,
Massimo

[Olivier Masse]

That's a fascinating discussion. I think that even though FTP is a long process, it's the easiest one that can be done, and furthermore the knowledge transfer will be much simpler. Concentrate on increasing the bandwidth between the two servers first, to lower your window.

I don't see how you could easily do this using your SAN. HP-UX doesn't support many filesystems except vxfs and hfs -- and I don't think they are production-level implementations on Linux. Maybe vxfs, but that would be a long shot on zlinux. And there could be some endianess or other compatibility issues that you might encounter.

I might be crazy but could it be possible to use mkisofs to make a 2Tb image file directly on a raw disk, assuming the standard allows it? Then present that disk read-only to HP-UX, assuming the endianess is the same (I don't know). But the hard part will be to trick HP-UX into mounting a big iso filesystem like this one, on a disk instead of a DVD. In all cases you won't be supported.

Another alternative, as some have said above, is to use your zLinux with NFS as your storage medium. But expect some performance issues woth your Data Warehouse.

Good luck

[Bill Hassell]

2TB in 20 hours?

You have a network problem. FTP is way faster than Samba or NFS. No network disk sharing will ever be as fast as FTP because of the huge number of small packets needed to see the list, get status, seek to a record, open/read/write/close, etc.

Start with typical network data rates (80% of wire speed, 10 bits/byte for encapsulation):

10Mb/sec = 0.8MB/sec = 2.8GB/hr
100Mb/sec = 8MB/sec = 28GB/hr
1000Mb/sec = 80MB/sec = 288GB/hr

These are optimistic throughput numbers -- 60-70% of wire speed is probably closer. So the link to the mainframe is not 1Gbit (or you have network setup problems). Change the connection speed to 1Gbit and if your switches (and the mainframe) support Jumbo frames, change the packet size from 1500 to 9000.

I would also split the group of files into 4 to 6 groups and start a separate ftp job for each file. And make sure you are using one copy of ftp per group. Starting and stopping ftp for each file is extremely slow. Even if you change nothing on the network, multiple ftp sessions will definitely improve throughput on your existing link.

Additionally, you can add 3 more network cards and split the files into 4 groups and then split each group into 4-6 groups (as above) to transmit in parallel over each link.

It is unlikely that the mainframe supports link aggregation (linking or teaming) but if it does, you could add several more LAN cards, create an aggregate with 4 or 6 cards on both ends, then start 8 to 15 ftp sessions at the same time (with different file groups for each ftp session). I created 4 link aggregate (with jumbo frames) between two HP-UX boxes using 1Gbit ports and found that the sweet spot was about 8-10 ftp sessions at the same time. That link was passing 400MB to 600MB per second! Since that is way faster than my internal disks could write, I used PowerPath to an EMC array with 4 paths. Any additional links to the aggregate would probably hit the EMC disk limits.

NOTE: a 4 link aggregate has no advantage if only one process uses the link.

[Massimo Bianchi]

Hi Bill,
you are correct, we have many parallel FTPs over many ethernet channel

If we had just one, 2000Gb/288Gb/h would be 7 hour, but we have difficulty in the efficiency. Getting 50% in neither a very bad results.

Not all files are available togheter, so data transfer is not so efficient. But it might be that there are possible enhancement in zOS networking, I'll gave that a shot.

Thanks,
Massimo

[Steven E. Protter]

Is it possible to connect the mainframe and HP-UX to a SAN?

Maybe an EVA-8000?

Perhaps speeds would be better if both systems wrote to the same disk group.

SEP

[Massimo Bianchi]

Hi SEP,
that is my question: how to connect to same disk group and exchange data ?

It's not a problem of making the LUN visible, it's to find a common layout ....

Massimo