DECWindows won't open DECterm on 2nd+later sessions

[Richard W Hunt]

h/w: Alpha ES40 mod II, 8 Gb RAM, 4xCPU, firmware is 7.3-1, console GZA0 is based on an ELSA GLoria Synergy card. It is working fine hardware-wise (see below for how I know this).

s/w: OpenVMS 8.3 freshly installed and patched through the July, 2009 patches. Significant parts: TCPIP 5.6 now at 9ECO4, MOTIF 1.6 now patched to ECO05 (and yes, we went through ECO02 to get there). We also have CDSA, KERBEROS, and SSL from the 8.3 kit but I don't think they are relevant. SYSGEN: VAXCLUSTER = 0.

We upgraded (not fresh-installed) OpenVMS last week. I was able to log in on the console using the new desktop, created a DECterm window, did a few things. The next day, performed all patches through July, 2009 including the UPDATE v10 patch.

Notes from that patch said it affected some system libraries in a way that might require a re-link or full re-install of some things. OK, noted that fact.

A couple of days after that, I logged in on the console but could not open a DECterm port. Nothing I could do would open up a new window. Not DECterm, nor FileManager, nor AppManager. And none of the stuff for screen tailoring worked either.

So I figured OK, I need to re-install because of the libraries comment as noted in UPDATE v10. Did that. Removed and re-installed MOTIF and patched it to ECO05.

Same exact symptom. First time after the re-install, I can open a DECterm session. Subsequent logins, no can do. I click double click but nothing opens. Nor can I open FileManager or AppManager windows. For those, I eventually get a pop-up that says nothing responded. Yes, right-click pops up and gives me a help screen if I ask for it, but nothing else seems to work right.

I've searched the forum first, and here are the answers to the most common questions.

Kits: Yes, the kit exited normally. Yes, it talked about it being the signed kit. No, I did not interrupt it during the install.

Configuration: All defaults, no customization for the system account or my personal account. As plain-vanilla as you can get.

SYSGEN: WINDOW_SYSTEM = 1; Global sections and global pages are very large. Memory is NOT depleted because right now, nobody else is touching this system. ORACLE isn't installed yet, either, so memory isn't a problem.

I've looked at the DECW$SERVER_0_ERROR.LOG file and it isn't particularly instructive. I can see where it starts up. The only error is that I haven't started the TCPIP service yet so it cannot attach to the TCPIP transport. No big deal at the moment. It clearly also connects to the _LOCAL transport, which is the console. I can see where the Permedia2 s/w inits screen 0. No errors in the startup. Ends with the message,

Permedia2: Exit from init with success.

I can see where it connects me when I log in through the window-style login. I see where the session removes certain paths from the fonts list because we didn't install some options and don't use anything in CDE$USER_DEFAULTS path. Some connections open and close, but no errors. Eventually Permedia2 says "Finished screen close down."

Next login doesn't look particularly different, certainly no error messages, but the DECterm window doesn't open. Yet the error log doesn't say antyhing about that.

And finally, if I use the Options button on the login to NOT use the New Desktop, I get a DECterm session, whether I use Safe mode or the "Traditional" mode. So I know the hardware is OK, it has to be a s/w problem.

So... where should I start looking?

[Hoff · Accepted solution]

When last I checked, DECwindows CDE didn't save stuff until logout. (I switched back to DECwindows Motif eons ago...)

Clean out the DT subdirectory under the login directory of whatever username is in use here; this will nuke all your settings.

As a test, try nuking the directory once you're in the locked out state, then exit the DECwindows session, login another DECwindows session, and see how far you get.

Also see if there is a local a definition of the logical name CDE$USER_DEFAULTS for your server.

And confirm your login directory ownerships are correct.

nb: The WINDOW_SYSTEM parameter is meaningless; that parameter was rendered irrelevant some time ago.

[Richard W Hunt]

Thanks, Hoff. I'll try later today to do the appropriate directory nuking.

As to CDE$DEFAULTS: My account is one that exhibits this oddity of behavior, so I figured I'd check the logicals from the personal account.

CDE$USER_DEFAULTS is a search list equal to CDE$USER_SPECIFIC and CDE$USER_COMMON in that order.

CDE$USER_SPECIFIC is $1$DGA1:[SYS1.CDE$DEFAULTS.USER.]

CDE$USER_COMMON is $1$DGA1:[SYS1.SYSCOMMON.CDE$DEFAULTS.USER.]

We are not running a cluster - but we once were, before our government security guys went ape over having test and production in the same network environment. So I split the cluster. This particular box booted through SYS1, obviously.

Each former cluster member is still the backup for the other system. Is it possible that being a former cluster but no longer being that way is mucking up the linkage? Though why it would work the first time and not any other time still eludes me. As you say, it must save something somewhere.

If this turns out to be the former cluster issue, would mucking about in DECW$LOGICALS.COM be productive or is that best done elsewhere? (Either way works for me, I have some specialized startup code where I could add a few more logicals.)

[Hoff]

You'll need to rummage through the DT and the CDE user and system roots for details such as file ownerships and file default protections and for any locally-customized DECterm default files such; if this box is truly "free" of the secure perimeter, I'd look to deinstall, then clean out all the CDE and DT roots I find still around, and try another installation. (If not simply perform my own preferred wipe and install.)

OpenVMS software installations tend to leave what some folks might consider local customizations and what others might call detritus; don't trust a deinstallation to remove all files, and don't trust a reinstallation to overwrite all files.

I'm familiar with secure environments; these all tend to be custom, and sometimes heavily customized, sometimes with SEVMS... I happen to like wiping these disks not just for security reasons that are of concern to the security officer(s) or administrator(s), but because that step also gets rid of all the weirdness these customizations can produce. (I've been suggesting the same for hobbyists for eons, too. Wipe and install, lest you end up playing through into somebody else's Adventure.)

[Richard W Hunt]

I've already re-installed once, but I can try again. I'm going to erase the CDE stuff as you suggest and flat-out whack that puppy, then start from scratch. I am in an isolated enclave so I can do at least some of that from my desktop, even if non-windows. At least I can pick up the kits from the distribution disk and plotz them down in a convenient place in case I have to repeat this experiment entirely too many times.

I'm going to catch-up on assigning points on this thread shortly, but first things first. Thanks for the help.

[Hoff]

Deinstall, then look for CDE giblets.

Use all due caution here, as some of the remaining giblets arrive with OpenVMS (via the DECwindows support files), and some arrive with the CDE (DECwindows) kit; the version-dependent and version-independent portions of that product.

I'd feel (relatively) safe deleting files in the CDE-specific roots and CDE-specific directories, but (definitely) not safe deleting DECwindows and CDE stuff in the OpenVMS system directories.

The typical application installation process and the application management model on OpenVMS is unfortunately rather intertwined with OpenVMS itself; in retrospect, it's not particularly modular. Ah, well.

[Richard W Hunt]

At this point, I'm running out of options other than to go back and "nuke" the whole disk. I.e. reinstall OpenVMS 8.3 as an original install rather than an upgrade. The amount of system reconfiguration required to do that is so incredible on our system that I'm saving that as the last resort.

As I write this, I've already done the removals. Right, wrong, or indifferent, I'm done fooling around. We've got at least two other products (not HP-related) that are also acting up. I'm beginning to see an issue in which we step back from the console (and nobody gets hurt...). At least until next fiscal year, because that's less than two months away and I have three more servers to do this to. And we are in the middle of hurricane season, just waiting for THAT particular shoe to drop, too...

I LOVE working for the government. I do, I do, I DO love working for the government...

[Richard W Hunt]

The strategic nuclear option didn't work.

I went back to my v7.3-2 system disk backup, restored it, booted same, and removed DWMOTIF 1.3-1 from that disk. Cleaned up the .DT subdirectories and as much as I felt I could get away with.

Then proceeded with the upgrade to 8.3 again. When it asked me if I was satisified with the options, I told it NO so I could manually force it to install DWMOTIF 1.6

Then did all patching. Same problem.

The first time I log in after the installation of DWMotif 1.6, joy joy. Once I log out, the next and all subsequent sessions will not open a DECterm window unless I chose the Options button >> Safe Mode

So I removed and reinstalled DWMOTIF 1.6 just to give myself something to do. (As if I didn't already have work...). Same thing. After install, first console login works. Second and subsequent console logins bring up the desktop but I cannot open anything. DECTerm is totally silent. The other things eventually give me an OK-only dialog box that says that no running task will honor the event I triggered.

[Hoff]

Short of logging into and digging around inside the box, I don't know what to tell you. Something here is definitely seriously wrong with the configuration; marvelously subtly wrong.

[Richard W Hunt]

I've been reading through the documentation on the 8.3 documentation disk, but the manuals all tell me how to customize the desktop or add new apps and icons and such as that.

I'm using the defaults for everything. I don't WANT to customize it.

Reviewing my notes from a LONG time ago, this problem first showed up when we upgraded to OpenVMS 7.3-2, but we didn't have the same problem that we do now. I didn't need to install a new version of something that really wants the GUI.

I cannot tell for certain because I wasn't always totally responsible for the machines, but I think this first reared its ugly head after an installation of CDSA because something in the authentication sequence changed as a result.

I don't know now how to make the DECW$SERVER_0 log more verbose, in hopes that it might actually tell me something. I'm still wading through the documentation and not always understanding what I'm seeing. (I'm an old CLI guy. Windows? That's what I look out of on a rainy day...)

Seriously, I've studied this until I'm blue in the face. And trust me, that ain't my color.

We are in the process of setting up an HP S/W support contract, which will be my next step before I take the tactical nuclear option. Which would be, instead of doing a PRESERVE type of install, let the INSTALL process erase and rebuild the system disk. But all of the site-specific configuration stuff will have to be rebuilt and that will take a long time to get right. I've got decent configuration documentation, but who EVER has perfect records of what they've done to the system in the last few years?

[John Gillings]

Richard,

Try logging in from another system (not at the console) and see if you can start any DECwindows applications from the command line. The example program ICO gives fairly good error messages:

$ SET DISPLAY/CREATE/TRANSPORT=LOCAL
$ RUN DECW$EXAMPLES:ICO

I'd also double check your SYLOGIN and LOGIN procedures to make sure they bypass anything interactive for non-interactive sessions.

[Richard W Hunt]

We were able to open a ticket with HP support. The bizarre behavior is probably due to something that happens after the first time you log in via the console, but here's the simple answer:

SYLOGIN has a dialog for interactive sessions. I am obviously not testing correctly for the DECWindows case in order to avoid the dialog. It appears that one of the "behind-the-scenes" DECW tasks actually goes through SYLOGIN when I didn't want it to, and my test for that case didn't stop the dialog. I will be working with HP for some sure-fire tests on how to tell when a given session can or cannot respond to a dialog.

Before anyone asks, the dialog is mandated for any session capable of interactively responding to a question. The mandate came from a level about 2 steps below the President of the USA so I won't be able to avoid having it. My "out" is that if the session isn't capable of responding, I don't have to ask in the first place. I just have to test correctly for cases that cannot respond. I thought I was doing it right but obviously I missed something.

Before I close this, I'll post what I find out from HP. Thanks for all who were trying to help.

[Richard W Hunt]

Final resolution.

For some strange reason, there are two kits with exactly the same name but different sizes and different file types for the DWMOTIF_SUPPORT for OpenVMS 8.3 on the 8.3 install kit. If somehow you copy both of these files, one is .PCSI and one is .PCSI$COMPRESSED. If you do a PROD INSTALL in a directory where both exist, you (appear) to get the .PSCI, which appears to be documentation only, instead of the compressed file with 100Kblocks+ of stuff.

I still don't know why the original install failed, but when I removed and reinstalled the product, I did so from copies of the various kits placed on a disk temporarily dedicated to the upgrae process. Somewhere along the line, the two files appeared in the same folder. When I uninstalled MOTIF from the 7.3-2 system and upgraded everything, then went back, patched all libraries, unistalled MOTIF and re-installed again (because of the STARLET.OLB patch), the wrong DWMOTIF_SUPPORT file got installed.

The fix was to go back to the original kit and grab the big SUPPORT file, remove the small file, and PROD INSTALL the support kit again. Didn't even have to reboot for it to start working correctly.

I have to believe that this was an incredibly nasty combination of a couple of old config files on the original install plus the confusion over the support kit. And yes, I verified this: The support kits differ only in the file type, .PSCI vs. .PSCI$COMPRESSED, and a PROD INSTALL by the typical approach of the name as reported by a PROD FIND picks the wrong one because it goes in alphabetic order.

Thanks to all who commented. I though I was very careful to watch my configuration, but that support file must have slipped through the cracks. Ah, well. Live and learn.

[Richard W Hunt]

Problem is now solved. I have notified HP of the potential for confusion between the two files on the 8.3 kit. They say they will consider my comments for the next release disk they have to build.