Radiology Information System running OpenVMS is Dying

[deuterium]

Hi there,

Got a slight challenge for you all, which could potentially help thousands of patients at a hospital. We have an ancient digital OpenVMS sytems which includes the following pieces of hardware: a MicroVAX3100 server, a digital StorageWorks archive, two digital DECserver 300 terminal servers, and an external 2Gig drive. This system used to be the 'Radiology Information System' at our hospital many years ago, and holds a bunch of patient records which we are obligated, by law, to keep for 10 years after the last usage. We decomissioned this system nearly 5 years ago, and have another 5 years to go. We have backups of the patient data on DAT tape. Oh, the system used to run an application called 'MaxiFile' which was a Radiology Information System, written in the M/MUMPS database language.

And we're running OpenVMS 7.1. Well, we used to be running OpenVMS 7.1. We're not really running anything anymore, and that's what we need some help with. ;)

So, last week we attempted to physically move the equipment from one rack to another within our computer room, in order to clear space. We powered down all of the components of the OpenVMS system, and moved them to the new rack, and everything was plugged in as before. The system came back up and ran for about week, and then started having intermittent system crashes. First, one of the DECserver 300 terminal servers never powered back on. But the system kept running. About the same time, it started having kernal errors and memory dumps, and rebooting itself. After about a half dozen system halts and memory dumps, spaced roughly one day apart each, the system started giving memory dumps and kernel errors during startup. We've been able to boot it up once or twice since this started, although it really appears to be on it's last legs. Even ff we can get it to boot into the operating system again, I'm not sure if we'll be able to get the application running again.

The reason I come to this forum is that the vendor who sold us this system went out of business like 5 years ago, and the company that was providing maintenance decided not to offer to extend the contract because they, too, were going out of business. So, we don't have anybody to take care of this stuff, and I've got to figure out a way to either repair the system or migrate the data.

So, the question is one of strategies in regards to retrieve the patient data. We have a dead and/or dying OpenVMS operating system, swappable harddrives and storage arrays which we can move to another machine if need be, and a bunch of data backup DAT tapes. We also have a modern Windows 2000/XP hospital network, with a couple spare machines around the department that we can work with. Question becomes, how can we retrieve this patient data?

Ideas I've come up with:
1. Try to reinstall or repair OpenVMS on this hardware (may or may not work if memory dumps and crashes are caused by hardware failure).
2. Try to find replacement hardware (not likely). Migrate patient data on disks to new hardware. Try to migrate MaxiFile application.
3. Try to emulate this system on a Windows 2000 or linux box. Access patient files on DAT tapes from OpenVMS running on a Windows emulator.
4. Try to access patient records on DAT tapes directly Windows. Probably have to use MUMPS and/or knowledge of filesystems used by OpenVMS to write to tape.

Any thoughts on these ideas on repair vs. migrate? Our hospital is loathe to spend any money on this particular problem, so it's going to be up to me to do most of the work. Personally, I've been investigating options 3 and 4 as I think they offer the best chance of success considering the circumstances and environment. Any experience or suggestions regarding data retrieval from DAT tapes written with OpenVMS/MUMPS would be an invaluable help right now.

Anyhow, I'll monitor this thread and respond with additional information as needed. Thanks for your time in advance!
Robert Watson
Radiology Information Systems Manager
raw9015@nyp.org




FYI: In case you're wondering, patient records older than three year tend to only be used for patient histories and legal reference. So, there is no clinical urgency to this request, although there is a legal necessity to try to find a solution.

[Ian Miller.]

Its going to cost some money and time and the hospital will have to live with that as the price for legal compliance.

MV3100 class vaxes can be acquired for small amounts of money. It may be possible to take the disks out of the old system and put them in the replacement system. At a guess it shoulds like you have a cpu or memory hardware fault.
If not restoring the tapes to a replacement system.

Emulation would cost money as you would have to get CHARON-VAX or similar and install it on a suitably fast windoze system. Could be this would be more money.

[deuterium]

Well, the hospital is willing to spend some money, although we are a not-for-profit institution, so money doesn't exactly grow on trees around here.

>>MV3100 class vaxes can be acquired for small amounts of money. It may be possible to take the disks out of the old system and put them in the replacement system. At a guess it shoulds like you have a cpu or memory hardware fault.

Where? On EBay? MicroVax-R-Us?

FYI, as far as I can tell, it does seem to be a CPU fault. The memory dump lists a bunch of registers and then the images which were loaded at the time. The last line of the dump lists SYSTEM_PRIMITIVES_MIN.EXE and a couple strings of data. I'm in agreement that the CPU may be bad.

>> Emulation would cost money as you would have to get CHARON-VAX or similar and install it on a suitably fast windoze system. Could be this would be more money.

How expensive is the CHARON-VAX emulator? Do you have a ballpark price?

[Volker Halle]

Robert,

I would also suggest to follow both the MicroVAX 3100 HW replacement and/or the CHARON-VAX emulation path.

I may also be able to give some advise about the underlying problem for the crashes and halts.

OpenVMS VAX includes the CLUE utility, which keeps a history and crash information for the last 100 crashes and shutdowns. And there is also ERRLOG.SYS.

The first thing you can do is this:

$ CLUE:==$CLUE
$ CLUE/DISPLAY
CLUE_DISPLAY > DIR/OUT=filename

and then post the contents of the ASCII file filename as an attachment in this thread. This should give an overview of the crash history of this system.

Information about each crash could then be extracted with:

CLUE_DISPLAY > EXTRACT/OUT=filename n ALL

where n is the crash number in column 1 from the crash history display.

You can get contact information of CHARON-VAX resellers or representatives from:

http://www.softresint.com/charon-vax/CHARON_partners.htm

Note there is a CHARON-VAX training in Nashua, NH on June 14/15.

Volker.

[Doug Phillips]

Robert,

Something happened during the move, but not to OpenVMS so there's no need to reinstall it. (there seldom is!)

First, make sure the fan is running and the openings are clear. I've seen intermittant CPU errors and such caused by overheating.

Regardless, I would do a $shutdown, power it off, unplug everything, take off the cover and see what's inside. A box that's been around for this long has probably grown a few dust-bunnies (even in a very clean environment, you'd be surprised!) and you could have jarred something loose during the move. While you're in there, check the connections and cables. Be carefull about static electricity.

If the problem is overheatiing, the longer you let it run the more chance there is that something will be permanently damaged, and it might not be yet.

It's just a computer and if you've ever opened up a PC it won't look all that strange to you.

-Doug

[John Gillings]

Robert,

Please contact your local HP Customer Support Centre. There are many possible solutions to your problem.

[Andy Bustamante]

In addition to the many good answers so far. I agree that something probably happened during the equipment move.

Option 1 is extremely unlikely, something probably got jiggered in the move.

Option 2 Many resellers out there. Check the web wander over to OpenVMS.org for some listings. If you have a requirement that these records be available, you should have a service contract in place. Next day service will save you a buck or two.

Option 3 Possible. Charon-VAX is the way to go, but this will also require funding.

My approach would be check cabling, make sure all cables are seated, including SCSI termination. Open the MicroVAX and reseat all cards on the motherboard. Reboot and monitor. If you have another outage, reseat the memory, monitor again.

[comarow]

Even if you do not have a support contract with HP, you can get a Per Call event, or even obtain a consumable number of calls.

There are lots of used 3100s in the marketplace, usually fully refurbished, from
fully authorized VAARs.

[John Donovan_4]

Robert,
I agree with Doug. These are hearty little systems and it takes a lot to put them out of commision. I worked for a local school which had one sitting in a closet for 2 years WITHOUT and A/C and it continued working like a champ.

As Doug said, power down, disconnect everything, open her up, reseat the cards that you can and reconnect everything. Make SURE that you haven't bent any cable pins because this can cause erratic behaviour.
IS the StorageWorks hardware reporting nay issues? Can you still see all the drives at the boot prompt?

[deuterium]

Hey, thanks for all the great responses so far! So, I opened up the MicroVAX3100 and fought off all the dust bunnies with a can of air. So far so good. All the obvious connections seemed in place, although I didn't do a throughough check (yet).

I booted up again, and got the following error:

83 BOOT SYS
-DKA300
%SYSBOOT-F-Unexpected Machine Check

then it went back to the >>> prompt, and I told it to 'boot' a couple more times. After about three or four tries, it started into OpenVMS and started going through it's system startup. Then it halted again. The screen was going by too quickly for me to write down everything, but I caught a 'SYSTEM-F-MCHECK' error and a Register Dump. It also paused at a line which said 'Improperly handled condition, image exit forced', and then proceeded to dump more registers, and then a list of all the images that were currently loaded at the time of the error. The last listed image was a 'SYSTEM_PRIMITIVES_MIN.EXE' I believe.

Anyhow, I powered everything down and brought it back up. Now we're getting:

-DKA300
?4F SCBINT, DKA300
84 FAIL

I would think that DKA300 is a hardware pointer perhaps? Do any of you happen to know if this relates to a SCSI cable or anything of the sort? Also, would those register dumps be caused by a unplugged connection?

Anyhow, I'm going to go through a take another look at our poor MicoVAX machine later this afternoon. Gotta go work on getting our MRI Scanner hooked up to the Archive in the meantime.

Thanks so much for all of your input again!
Robert

[deuterium]

>>> Even if you do not have a support contract with HP, you can get a Per Call event, or even obtain a consumable number of calls.

Thanks for the info, comarow. Didn't know that they had a Per Call option, and it hadn't really occured to us that HP might be an option. Management will like to hear that there's a vendor we can contact to help take care of this problem.

[Volker Halle]

Robert,

machine checks indicate hardware errors, most typically memory or CPU. So some hardware may need to be fixed.

You may be able to run memory diagnostics from the console prompt. Try >>> HELP to see, if you can get some help.

Diagnostics test are generally started with a

>>> T n

command. You may need to look up details in the appropriate system manual for your machine.

Volker.

[Volker Halle]

Robert,

to capture all console output, it would be good to connect a PC/Laptop to the MicroVAX 3100 console port using a serial null modem cable and use a terminal emulator (PowerTerm, Hyperterm etc.) and configure the emulator to capture all output from the MicroVAX into a file (Powertem: Communication -> Receive File).

Volker.

[John Donovan_4]

It looks like some sort of hardware failure(sorry). When you're at the >>> prompt enter SHOW DEVICE to see the devices out there. I don't recall whether the MV had SHOW CONFIG or not but try it too. Your boot disk or SCSI card might show an error.
The 4F SCBINT error is (from old DEC page)
"4F SCBINT Unexpected SCB exception or machine check."

Which basically translates to:
"This appears to be a hardware error -- please contact your hardware
service organization for assistance."
SCB by the way means "System Control Block"

All of my research points to disks, controller or IO bus. Do you have an IMAGE backup of your system disk & the other disk? If so pick up a MV from somewhere (cheap!) and restore the IMAGES back to disk. With these errors it looks hardware related & not VMS operating system.

[Antoniov.]

Hi Deuterium,
my own idea is:
1.If trouble is hardware obviously you can't solve reinstalling vms. It appears a hardware bug check.
2.Hardware replacement (without advertising):
Island computer http://www.islandco.com/
Partfinder http://www.compaqrepair.com/DEC/
Trident http://www.tridentusa.com/service/support/dec/hardware/
In Europe
Hammer http://www.hammer.co.uk/
MIT http://www.mitlimited.com/
3.Emulation
Best emulation is Charon VAX http://www.charonvax.com/
There is a freeware emulator called simh
http://simh.trailing-edge.com/

You can also find other informations on http://www.openvms.org/

HTH
Antonio Vigliotti

[Jan van den Ende]

Robert,

As you found out by now, most of us are giving educated guesses based on previous experience as well as your report.

My 2 (EUR) cents:
First, you ask about the meaning of DKA300.
Well, it is the hardware ID of a SCSI disk, and, since you (try to) boot from it, this is the system disk.

(the next is assuming your system consists of more than one single box; ie., you have one or more "external" boxes, connected with thick cabling. If not, what follows below does not apply)

The sequence
- physically moving the system
- boot problems
- unable to even begin booting
- errors about even adressing DKA300
is (to me) making the SCSI cabling the prime suspect. And believe me: it _IS_ much too easy to bend some pin in the connectors.

Disconnect the cable(s) carefully, and meticulously scan the pins for any being bent.
If found, the problem boils down to getting a new cable.

-- hope this helps.

Proost.

Have one on me.

jpe

[Robert Gezelter]

Robert,

I have encountered this situation on several situations. This is most likely a hardware problem, not a problem with OpenVMS or the applications software.

It is not unusual for a system to be jostled when it is moved. The connectors and cards can be dislodged from their connections. Also, elderly ribbon cables can develop small cracks in their conductors, making them unreliable.

Another possibility is that you have a problem with the power supply. A malfunctioning power supply can have all forms of strange side effects. At one client, a small VAX that had been running for almost 15 years without a problem started behaving erratically, generating machine checks and other strange symptoms. A change of the CPU and memory did nothing. A change in the power supply completely resolved the problem. If the CPU can be considered the heart of the system, the power supply probably should be the circulatory system. The bad power supply, in effect, caused a circulatory collapse.

In my experience, most damage is caused in such situations when actions are taken in haste. Working carefully, I can't ever recall losing a system or data in this state. There have been close calls when somebody immediately resorted to reinstallations or restores from backups. Most likely, the disks are intact, and it is the rest of the system that is having problems.

To summarize:
- check and reseat all cables and boards
- consider using a spare system
- if possible, clone the system and work with the clone

I hope that the above is helpful. If I can be of further assistance, please let me know (Ironically, I happen to be in and out of meetings in Manhattan today).

- Bob Gezelter, http://www.rlgsc.com

[John Donovan_4]

One other item: if you know how (and there are articles around on how to do this) the SCB can be rebuilt.

[comarow]

If you have a CD and an operating system
CD, you can boot to it. That would be a great way to determine if it's the system or the disk.

If you don't have a cd, you could try booting the stand alone backup tape.

[Jan van den Ende]

Re Comarov:

if you have NOT got the CD, it will definitely be possible to get one, if only on a load basis.

Anyone in that geographic region?

Proost.

Have one on me.

jpe

[Jan van den Ende]

Sorry,

"load" read "loan"
(I need typing lessons! :-(

Proost.

Have one on me.

jpe

[Willem Grooters]

Given it's age and probably (over)heating due to all the dust inside, I simply have to agree it's an hardware error, either with the system disk (DKA300), the controller it's connected to, or the cabling in between. And, seen what came accross your screen when booting, that won't be all. Even if you were able to fix this, other parts may (will) fail in due time. IMHO, it's beyond repair.

Although a refurbished Vax 3100 (or other) may be cheaper, I would advise to use a PC with CharonVax, since you already use Wintel boxes. The extra advantage will be you can then use technology that cannot be used on VAX. All your disks can be transferred to your Wintel storage; it's been done before several times.
Your VMS license can be transferred - at no, or very low cost. The only thing you will need is your system disk backup to be restored, or media to built a new one from scratch. But the first is the easy option.

Willem

[Carl Couric · Accepted solution]

Hi Deuterium,

If your in the USA, I've got 3 MV3100s just sitting here in my lab doing nada, nothing, collecting dust.... I can give you one, just pay for shipping :) .

Based on everyones response, you have either:

1) Lost a SCSI bus controller
2) Lost a SCSI disk on the bus (to test, you would take each drive off one at a time and see if the bus behaves, but if you pull the system drive, well, kind of hard to test then, you need a backup SCSI disk).

Also, if it is just a disk (like a 1Gb or 2Gb [RZ-26 or RZ-28]) I may have a few of those around as well... I think I may even have a box of 4Gb RZ-29s around here somewhere!

And to test a disk (BIG DISCLAIMER HERE: USE AT YOUR OWN RISK, IF IT DELETES YOUR DISKS DATA, DON'T EVEN THINK OF BLAMING US!), you can from the >>> prompt "T 50" and "T 51" if I recall. One will run a format on a new/replacement SCSI disk and other does a verify. It will completely erase the disk but if you getting failures in there, you know you have hardware issues.

Carl
http://www.carlc.com - OpenVMS webhosting since 1999

[deuterium]

wow. i just want to thank everybody again for the great help and advice that you've offered during the past week. here's the update:

so, i opened up the box a second time, as per Jan and Robert's suggestion, and sorted through the cables, maticulously looking for broken pins and whatnot. didn't really find anything though, so the SCSI cable seems relatively intact. but who know. i plugged the SCSI cable abck in, making sure that it was securely connected on both ends.

When I rebooted, the machine started to boot into the OpenVMS operating system again, which was a good sign. I did notice an error which I didn't notice before saying something to the affect of '%SYSTEM_F_FULL Unable to write to dev'. It did manage to get past that stage, however, and proceeded to the regular system start up. Then it crashed again with the register dumps.

I rebooted, and again got "%SYSBOOT-F-Unexpected Machine Check error" .

Out of curiosity, is it common for OpenVMS to have register dumps if a drive is completely filled?

For the record, the idea that the error is being caused by hardware failure is amongst the most easily believable suggestions I've ever run into. You simply have no idea what kind of horrid condition this system was in. When I came into this position about a year ago, it was covered in grime, dirt, dust, and water stains. Parts of the plastic are turning yellow-brown from over-heating and/or chemicals or something. And you should have seen the state of the terminal server and fax-server data lines. The morons who hooked it up had no clue about cable management, and over the past 15 years the cables became a giant 10 square-foot spider-web-knot of data lines. It was literally the kind of mess that I wouldn't have been surprised if a couple bugs or rodents had made a nest inside of it.

After about 9 months on the job, I decided I was ready to decomission the fax and terminal servers so that we could move it to our server room or something. Everything was going along fairly smoothly until I physically moved it from it's rack to a new rack. That's when it had it's first bad heart attack, I think.

Anyhow, we have plenty of data backups, so we're not worried that critical patient data is lost. But we do need to get this thing back up and running, as we got another subpeona yesterday.

[deuterium]

So, after a week of responses, the overwelming response seems to be to investigate replacing the hardware; an option I hadn't initially considered seriously.

Currently, my thinking is that I would like to follow up with Carl Couric's offer. Carl, if you could send me your email, I'd like to discuss with you about obtaining one of your MV3100s. As a not-for-profit organization, we can at least write up one of those receipts-of-donation-thingers with our tax-code, so you should be able to get an honest to goodness legit tax write-off for the donation. We'll send a thank you note, letter of appreciation, and all that jazz too. And we will, of course, pay for shipping and handling. =)

If that doesn't work, then we'll go on to investigate the possibility of accessing the the via Windows somehow as per Willem's suggestion. There are definately some possible benefits to doing that, and given the condition of the equipment, I have to agree that future hardware errors are a matter of when, not if. However, the rest of the equipment and the new controller might be able to muck along for another couple years until our legal requirements expire. That, and it would be far easier to migrate the data with a working MicroVax3100 (e.g. being able to FTP the database to a new machine).

As per camarow's question, the system doesn't have a CD drive anywhere. In fact, it doesn't seem to have a floppy drive of any sort anywhere. It did used to have an honest-to-goodness tape reel storage system though (6250 BPI, 3/4" inch tape, about 18" in diameter reels). The tape reel system was decommissioned about 10 years ago when they got the 2Gig hard-drive though. =)

If Carl's offer doesn't work out, we'll have to look at some of the solutions that Antiniov offered. Thank you for the list of of suppliers, Antoniov!

And good call on the terminal emulator, Volker. That hadn't quite occured to me, although it actually solves a number of minor concerns that we had been thinking about, in the case of console failure.

As per John's question, I'm not sure if we have an IMAGE backup of our system disk. That could be a problem. However, we do have lots of data backup sitting around here, so I'll sort through them and try to get an answer to that question. We may very well have something.

Next week, I'm also going to switch out the UPS on the system also. It's been having some battery problems, and Robert may be onto something regarding the fluctuating power supply. I know that any rechargeable battery has only so many recharges before it goes dead. ALthough a UPS seems to be always charged, and doesn't have the same problems of charge/drain that normal rechargeable batteries have, I can only assume that ours might be on the fritz.

Anyhow, that's it for this week. I'm sure I'll be posting updates in the upcoming weeks as we get the machine restored or the data migrated.