Radiology Information System running OpenVMS is Dying

[deuterium]

Hi there,

Got a slight challenge for you all, which could potentially help thousands of patients at a hospital. We have an ancient digital OpenVMS sytems which includes the following pieces of hardware: a MicroVAX3100 server, a digital StorageWorks archive, two digital DECserver 300 terminal servers, and an external 2Gig drive. This system used to be the 'Radiology Information System' at our hospital many years ago, and holds a bunch of patient records which we are obligated, by law, to keep for 10 years after the last usage. We decomissioned this system nearly 5 years ago, and have another 5 years to go. We have backups of the patient data on DAT tape. Oh, the system used to run an application called 'MaxiFile' which was a Radiology Information System, written in the M/MUMPS database language.

And we're running OpenVMS 7.1. Well, we used to be running OpenVMS 7.1. We're not really running anything anymore, and that's what we need some help with. ;)

So, last week we attempted to physically move the equipment from one rack to another within our computer room, in order to clear space. We powered down all of the components of the OpenVMS system, and moved them to the new rack, and everything was plugged in as before. The system came back up and ran for about week, and then started having intermittent system crashes. First, one of the DECserver 300 terminal servers never powered back on. But the system kept running. About the same time, it started having kernal errors and memory dumps, and rebooting itself. After about a half dozen system halts and memory dumps, spaced roughly one day apart each, the system started giving memory dumps and kernel errors during startup. We've been able to boot it up once or twice since this started, although it really appears to be on it's last legs. Even ff we can get it to boot into the operating system again, I'm not sure if we'll be able to get the application running again.

The reason I come to this forum is that the vendor who sold us this system went out of business like 5 years ago, and the company that was providing maintenance decided not to offer to extend the contract because they, too, were going out of business. So, we don't have anybody to take care of this stuff, and I've got to figure out a way to either repair the system or migrate the data.

So, the question is one of strategies in regards to retrieve the patient data. We have a dead and/or dying OpenVMS operating system, swappable harddrives and storage arrays which we can move to another machine if need be, and a bunch of data backup DAT tapes. We also have a modern Windows 2000/XP hospital network, with a couple spare machines around the department that we can work with. Question becomes, how can we retrieve this patient data?

Ideas I've come up with:
1. Try to reinstall or repair OpenVMS on this hardware (may or may not work if memory dumps and crashes are caused by hardware failure).
2. Try to find replacement hardware (not likely). Migrate patient data on disks to new hardware. Try to migrate MaxiFile application.
3. Try to emulate this system on a Windows 2000 or linux box. Access patient files on DAT tapes from OpenVMS running on a Windows emulator.
4. Try to access patient records on DAT tapes directly Windows. Probably have to use MUMPS and/or knowledge of filesystems used by OpenVMS to write to tape.

Any thoughts on these ideas on repair vs. migrate? Our hospital is loathe to spend any money on this particular problem, so it's going to be up to me to do most of the work. Personally, I've been investigating options 3 and 4 as I think they offer the best chance of success considering the circumstances and environment. Any experience or suggestions regarding data retrieval from DAT tapes written with OpenVMS/MUMPS would be an invaluable help right now.

Anyhow, I'll monitor this thread and respond with additional information as needed. Thanks for your time in advance!
Robert Watson
Radiology Information Systems Manager
raw9015@nyp.org




FYI: In case you're wondering, patient records older than three year tend to only be used for patient histories and legal reference. So, there is no clinical urgency to this request, although there is a legal necessity to try to find a solution.

[Ian Miller.]

Its going to cost some money and time and the hospital will have to live with that as the price for legal compliance.

MV3100 class vaxes can be acquired for small amounts of money. It may be possible to take the disks out of the old system and put them in the replacement system. At a guess it shoulds like you have a cpu or memory hardware fault.
If not restoring the tapes to a replacement system.

Emulation would cost money as you would have to get CHARON-VAX or similar and install it on a suitably fast windoze system. Could be this would be more money.

[deuterium]

Well, the hospital is willing to spend some money, although we are a not-for-profit institution, so money doesn't exactly grow on trees around here.

>>MV3100 class vaxes can be acquired for small amounts of money. It may be possible to take the disks out of the old system and put them in the replacement system. At a guess it shoulds like you have a cpu or memory hardware fault.

Where? On EBay? MicroVax-R-Us?

FYI, as far as I can tell, it does seem to be a CPU fault. The memory dump lists a bunch of registers and then the images which were loaded at the time. The last line of the dump lists SYSTEM_PRIMITIVES_MIN.EXE and a couple strings of data. I'm in agreement that the CPU may be bad.

>> Emulation would cost money as you would have to get CHARON-VAX or similar and install it on a suitably fast windoze system. Could be this would be more money.

How expensive is the CHARON-VAX emulator? Do you have a ballpark price?

[Volker Halle]

Robert,

I would also suggest to follow both the MicroVAX 3100 HW replacement and/or the CHARON-VAX emulation path.

I may also be able to give some advise about the underlying problem for the crashes and halts.

OpenVMS VAX includes the CLUE utility, which keeps a history and crash information for the last 100 crashes and shutdowns. And there is also ERRLOG.SYS.

The first thing you can do is this:

$ CLUE:==$CLUE
$ CLUE/DISPLAY
CLUE_DISPLAY > DIR/OUT=filename

and then post the contents of the ASCII file filename as an attachment in this thread. This should give an overview of the crash history of this system.

Information about each crash could then be extracted with:

CLUE_DISPLAY > EXTRACT/OUT=filename n ALL

where n is the crash number in column 1 from the crash history display.

You can get contact information of CHARON-VAX resellers or representatives from:

http://www.softresint.com/charon-vax/CHARON_partners.htm

Note there is a CHARON-VAX training in Nashua, NH on June 14/15.

Volker.

[Doug Phillips]

Robert,

Something happened during the move, but not to OpenVMS so there's no need to reinstall it. (there seldom is!)

First, make sure the fan is running and the openings are clear. I've seen intermittant CPU errors and such caused by overheating.

Regardless, I would do a $shutdown, power it off, unplug everything, take off the cover and see what's inside. A box that's been around for this long has probably grown a few dust-bunnies (even in a very clean environment, you'd be surprised!) and you could have jarred something loose during the move. While you're in there, check the connections and cables. Be carefull about static electricity.

If the problem is overheatiing, the longer you let it run the more chance there is that something will be permanently damaged, and it might not be yet.

It's just a computer and if you've ever opened up a PC it won't look all that strange to you.

-Doug

[John Gillings]

Robert,

Please contact your local HP Customer Support Centre. There are many possible solutions to your problem.

[Andy Bustamante]

In addition to the many good answers so far. I agree that something probably happened during the equipment move.

Option 1 is extremely unlikely, something probably got jiggered in the move.

Option 2 Many resellers out there. Check the web wander over to OpenVMS.org for some listings. If you have a requirement that these records be available, you should have a service contract in place. Next day service will save you a buck or two.

Option 3 Possible. Charon-VAX is the way to go, but this will also require funding.

My approach would be check cabling, make sure all cables are seated, including SCSI termination. Open the MicroVAX and reseat all cards on the motherboard. Reboot and monitor. If you have another outage, reseat the memory, monitor again.

[comarow]

Even if you do not have a support contract with HP, you can get a Per Call event, or even obtain a consumable number of calls.

There are lots of used 3100s in the marketplace, usually fully refurbished, from
fully authorized VAARs.

[John Donovan_4]

Robert,
I agree with Doug. These are hearty little systems and it takes a lot to put them out of commision. I worked for a local school which had one sitting in a closet for 2 years WITHOUT and A/C and it continued working like a champ.

As Doug said, power down, disconnect everything, open her up, reseat the cards that you can and reconnect everything. Make SURE that you haven't bent any cable pins because this can cause erratic behaviour.
IS the StorageWorks hardware reporting nay issues? Can you still see all the drives at the boot prompt?

[deuterium]

Hey, thanks for all the great responses so far! So, I opened up the MicroVAX3100 and fought off all the dust bunnies with a can of air. So far so good. All the obvious connections seemed in place, although I didn't do a throughough check (yet).

I booted up again, and got the following error:

83 BOOT SYS
-DKA300
%SYSBOOT-F-Unexpected Machine Check

then it went back to the >>> prompt, and I told it to 'boot' a couple more times. After about three or four tries, it started into OpenVMS and started going through it's system startup. Then it halted again. The screen was going by too quickly for me to write down everything, but I caught a 'SYSTEM-F-MCHECK' error and a Register Dump. It also paused at a line which said 'Improperly handled condition, image exit forced', and then proceeded to dump more registers, and then a list of all the images that were currently loaded at the time of the error. The last listed image was a 'SYSTEM_PRIMITIVES_MIN.EXE' I believe.

Anyhow, I powered everything down and brought it back up. Now we're getting:

-DKA300
?4F SCBINT, DKA300
84 FAIL

I would think that DKA300 is a hardware pointer perhaps? Do any of you happen to know if this relates to a SCSI cable or anything of the sort? Also, would those register dumps be caused by a unplugged connection?

Anyhow, I'm going to go through a take another look at our poor MicoVAX machine later this afternoon. Gotta go work on getting our MRI Scanner hooked up to the Archive in the meantime.

Thanks so much for all of your input again!
Robert