Database Locking

[Tony Marron]

Hi

I have a problem with an old Vax 3100 running an ERP system based on a DBMS Rdb database. Following a disk crash and recovery users can access the database for reads but the system locks up when anyone attempts a write. I am guessing this is a DBMS problem (I am speaking to Oracle) but normally DBMS simply rejects the transaction but does not lock the whole system.
I would appreciate any suggestions as to if there is anything I should check in VMS.

Thanks in advance.

[Ian Miller.]

Can you be more specific about 'the system locks up'.

Can do you mean the users of the application can't do anything? Can you login to VMS (not using the application) successfully?

[Tony Marron]

Hi Ian,

The whole system seems to hangs and it is not possible to access anything, including a DCL prompt. Hence the question as to whether this is bigger than just a DMBS issue. The only action I can take is a switch off/on !

[Hoff · Accepted solution]

There are various "3100" models -- the MicroVAX and VAXstation and even a few VAXserver 3100 series boxes, all slightly different. These are of various vintages and various limits. The older boxes will usually have older disks and older disks can be more prone to failures. On no particular evidence, I'd be looking at the disks first.

Oracle Rdb is pretty good at locking out the whole database when something bad happens, as you've noted. But it should not lock out the whole OpenVMS VAX system, as others have noted.

You may well have a low-level disk corruption or a failing or failed disk, or a core hardware problem. Or you might have a software problem that's triggering a system-wide problem. That the database transaction triggers it implies either an I/O hardware problem, or a software lock-up of some sort.

Assuming you can get the box rebooted, check the hardware. Check for any hardware errors reported on-line. Check the error log.

If this is a hardware error underneath one of the pieces of Rdb , you may need to roll in your database archives and roll forward your incrementals.

If you have a hardware and/or software service contract for OpenVMS VAX or the particular 3100 system involved, now might be a good time to call your provider(s). If you don't have a current support provider, there are organizations around that offer per-call. (Full disclosure: HoffmanLabs is one.)

The other possibility is that the database has somehow tipped over and consumed critical system resources, or grabbed critical system locks. And wedged the system or system disk.

To try to differentiate these two possibilities -- and there are others -- I'd tend to use the console to attempt to halt the system, and I'd force then a crashdump. (This assumes a dump file has been configured.) I'd then ANALYZE/CRASH the dump see what the system was doing when it crashed. If you can't halt the system when it's locked up, it's probably hardware. If you can halt the system and force the dump, it might be hardware or software. A reboot triggers a SCSI reset which can sometimes clear up a "stuck" disk, for instance.

In either case, somebody's going to dig into this and the error log, the crash dump (if you can get one), the system resources and tuning, and the system and storage hardware...

There are other cases that can cause odd and similar problems, such as a disk-full. Oracle Rdb might well encounter this and lock up the database, but I've not seen Rdb then trigger a system-wide hang. Well, actually I have, but the database hang turned out to be part of a more involved error.

If you can get the box going again, I'd first consider the status of the archives before proceeding. Now would be a good time for a full RMU/BACKUP to some other disk or out to archive media, and probably with a database verification thrown in, if you don't already have a recent/current archive you're willing to recover from, or roll forward from. This assumes you can get RMU/BACKUP to work.

As and after you regain access to your data and your applications, I'd also look at whether or not I wanted to upgrade this box to a newer (old) VAX or to a VAX emulation, or a port to a new or used Alpha, or an Integrity server.

Stephen Hoffman
HoffmanLabs

[Tony Marron]

Hi Stephen,

Thanks for the detailed response. I am beginning to suspect that their may be a disk issue here. I am going to do some further digging and will update you later - Thanks.

[Colin Butcher]

Hello Tony,

When you say "disc crash" - do you know what happened? Is there any useful information in the error logs?

What kind of hardware configuration does this system have - single disc, multiple discs, RAID array (unlikely), internal or external discs? The 3100 family varied quite a lot in scale.

With a machine of that age I would be tempted to consider alternatives, even if just newer VAX hardware that will run the same version of VMS. The biggest risk is probably the disc subsystem, so using a RAID array could well be a good idea. Your options depend on what hardware you currently have.

In general it's best to separate data storage from the OS if you can do so. It's also a good thing to automate much of the system housekeeping as well - things like tidying up error logs, audit server logs and so on, otherwise the system disc can gradually run out of space, then you end up with a hung system at some point.

Getting a snapshot of the system "as is" would be a sensible first step. One technique I use is to bring along another small VAX and boot from that, then mount the target system's discs read-only over the SCSI bus (leaving the first system powered up but not booted if there's no other way to get at the discs).

You'll find that several of the people who contribute to the ITRC are also in a position to help you if you require external assistance.

Good luck. Cheers, Colin.

[Tony Marron]

Hi Colin,

Thanks for your thoughts and suggestions. This is a very old system, 3 seperate disks (no raid) but thankfully not critical to the business. I believe that we have narrowed the problem down to a system database on the ERP package and issues with the Oracle product CDD. We are reinsalling both at the moment so I will let you know.

[Wim Van den Wyngaert]

To check which file(s) use bad blocks, try anal/disk/read and check for parity errors.

I have 1997 Alpha stations and they have bad blocks at a rate of 1 per year (average for 60 stations). I mark the bad file as "bad" (rename) and do a set file/nomov to prevent defragmenter to use it. Then I reinstall it by copying it again from another system.

Fwiw

Wim