IA64 linker /export & /publish

[Tim Yeung_1]

In the VMS Guide to Porting from Alpha to IA64 manual, section 5.3.3.4 mentions /EXPORT and /PUBLISH as new features. I'm on IA64/VMS V8.3, but LINK/EXPORT and LINK/PUBLISH returns Invalid Qualifier error. The V8.3 linker reference manual does not mention them either. Are these qualifiers supported? Where can I get info on how to use them?

Thanks.

[John Gillings · Accepted solution]

Tim,
Looks to me like they were backed out at the last minute.

See SYS$UPDATE:IA64_LINK.CLD. Comments say:
!
! X-8 (15) HB Hartmut Becker 23-Jun-2005
! Remove /export and /publish
!
! X-7 (14) HB Hartmut Becker 21-Mar-2005
! Change disallow rules for /export
!
!
...
!
! X-5 (11) HB Hartmut Becker 1-Jul-2004
! Add positional /export_symbol_vector and local
! /publish_global_symbols
!

So, it sounds like there's (at least) latent support for them, but it's not yet been released into the wild. From the names, I'd guess they write extra files with the symbol vector and/or global symbol table in some kind of human readable format. Only the product managers know when or if they'll ever be enabled.

What are you trying to do? Extracting the symbol vector can be done with ANALYZE/IMAGE (or, find a copy of my FAKE_RTL.COM procedure - a side effect of building a fake shareable image is a text file listing the symbol vector - called ".VEC".

$ @fake_rtl
Defining FAKE_RTL environment
$ @fake_rtl smgshr
%COPY-S-COPIED, SYS$COMMON:[SYSLIB]SMGSHR.EXE;1 copied to DISK_CSD:[TEST.FAKE_RTL]REAL_SMGSHR.EXE;1 (1326 blocks)
Generating symbol vector for SMGSHR on IA64
%ANALYZE-I-ERRELF, DISK_CSD:[TEST.FAKE_RTL]REAL_SMGSHR.EXE;1 0 errors
Generating MACRO code for SMGSHR
Compiling FAKE_SMGSHR
Linking FAKE_SMGSHR
$ TYPE
smgshr.vec
GSMATCH=LEQUAL,1,104 !
00000000 0000000000111648 0000000000111648 SMG$CREATE_VIRTUAL_KEYBOARD (procedure)
00000008 0000000000111660 0000000000111660 SMG$DELETE_VIRTUAL_KEYBOARD (procedure)
00000010 0000000000111678 0000000000111678 SMG$READ_STRING (procedure)
00000018 0000000000111690 0000000000111690 SMG$READ_COMPOSED_LINE (procedure)
00000020 00000000001116A8 00000000001116A8 SMG$SET_KEYPAD_MODE (procedure)
00000028 00000000001116C0 00000000001116C0 SMG$CANCEL_INPUT (procedure)
00000030 00000000001116D8 00000000001116D8 SMG$CREATE_KEY_TABLE (procedure)
00000038 00000000001116F0 00000000001116F0 SMG$ADD_KEY_DEF (procedure)
00000040 0000000000111708 0000000000111708 SMG$GET_KEY_DEF (procedure)
00000048 0000000000111720 0000000000111720 SMG$DELETE_KEY_DEF (procedure)
... etc

An easy way to dump the global symbol table is to insert the object module(s) into an object library and use LIBRARY/LIST/NAMES to dump the symbol table (this can also be done for shareable images using a shareable image library).

$ lib/create/share smg sys$share:smgshr
$ lib/lis/names smg/share
Directory of ELF SHAREABLE IMAGE SYMBOL TABLE library DISK_CSD:[TEST.FAKE_RTL]SMG.OLB;1 on 9-NOV-2007 10:08:11
Creation date: 9-NOV-2007 10:07:54 Creator: Librarian I01-40
Revision date: 9-NOV-2007 10:07:54 Library format: 6.0
Number of modules: 1 Max. key length: 1024
Other entries: 129 Preallocated index blocks: 213
Recoverable deleted blocks: 0 Total index blocks used: 15
Max. Number history records: 20 Library history records: 0

Module SMGSHR
SMG$ADD_KEY_DEF
SMG$ALLOW_ESCAPE
SMG$BEGIN_DISPLAY_UPDATE
SMG$BEGIN_PASTEBOARD_UPDATE
SMG$BEGIN_USER_OUTPUT
SMG$CANCEL_INPUT
SMG$CHANGE_PBD_CHARACTERISTICS
SMG$CHANGE_RENDITION
...

[Tim Yeung_1]

We're trying to extract a bunch of global symbols from a shareaable image that contains many many data items.

I'll give FAKE_RTL.COM a try. I will also read your article in the OpenVMS Technical Journal.

Thanks again

Tim

[deleted1411]

Please read the Linker release notes and the Linker manual. The qualifiers were intended to help with creating symbol vector options, especially for programming languages with mangled names (for example C++).

However, exporting everything is not good and can not reliably work on VMS. Therefore these qualifiers were removed. There is no latent support. These qualifiers will not come back.

If you are using C++ and need to associate mangled and demangled names, please use the CXXDEMANGLE tool or with the I64 V8.3 Linker create a map file section with mangled and demangled global names. Also, check the C++ compiler documentation, which was enhanced to help with creating shareable images (/EXPORT_SYMBOLS qualifier).

The Linker qualifiers were never intended to extract universal symbols from existing shareable images. A full linker map shows the symbol vector options. A cross reference section of the linker map for shareable images contains the globals symbols and flags universal ones with "(U)". And as already mentioned, there are other ways to find the universal symbols from shareable images.

From the entry I can only guess what you are trying to do: you want to call lib$find_image_symbol (or the CRTL equivalent) to activate a shareable image and need to know the "real" universal symbols. If you need the universal symbols for something else, I would like to know.

Hartmut Becker

[Tim Yeung_1]

The application was developed starting with VMS 1.2. There are now over 7 million lines of Macro, FORTRAN, and C code. It was migrated to Alpha in the late 1990's. All of the original developers have retired. I got involved recently to migrate it to IA64.

The application consists of many shareable images of data items. When linking, they used the BASE=

command in the link options file to "hard-code" the shared data in pre-defined address locations. For example, BASE=%X100000 for shareable image A and BASE=%X200000 for shareable image B. When a process stores data in a particular data item, it passes the data item address (via mailboxes) to other processes. Other processes can access the same data item because all shared data are linked to the same addresses. Thousands of UNIVERSAL statements were maintained in the link options files to declare data item locations.

When the application was migrated to Alpha, BASE=

was no longer supported. So the addresses were converted into offsets. They also grew tired of maintaining the long lists of UNIVERSAL statements. So on Alpha, they first do a LINK/NOEXE/STB to generate a symbol table of the shareable image. Then they run a custom program called SYMVEC.EXE to read the symbol table file and create a link options file containing the long list of SYMBOL_VECTOR statements. Then they use this link options file to link again to generate the shareable image. They create many shareable images. They finally, they link programs with subsets of shareable images.

I have to either update SYMVEC to handle ELF symbol tables, or I find an alternative to export/publish all the shared data items.

Tim

[deleted1411]

It looks like exporting everything, that is all global symbols from each object, worked in your case. But there might be some knowledge inside the SYMVEC program to select only specific symbols or to suppress some symbols (that would be similar to what the C++ compiler does with its /export switch and which makes it more useful than a linker switch). If that's the case, you have to "port" that program.

As you probably know, on I64 the ELF is used for object and image files as well as for symbol table files and debug symbol files. ANALYZE/OBJECT (and /IMAGE) work on all ELF files. An STB file is neither an object nor an image, but ANALYZE/OBJECT/SECTION=SYMTAB can be used to format all the symbols. So SYMVEC may be implemented as a DCL command procedure processing this output.

It looks like you need to create the set of symbol_vector= options only once. But I may be wrong and new shareable images may be created on a regular basis. At the moment I don't see that you need to write a program. But if you need, any public source code which can read ELF64 symbols should do. Please send me mail if you need a pointer to such source code.

However, from your description I still don't understand why the developers decided to read an STB to extract just the global symbols. The same names are available in the sources and reading ascii text is usually easier. It's much easier, if all symbols are marked to be exported in the language context, like
with a dllexport. That's the preferred way over selecting everything from the object module. The simple reasons are, the compilers may have included some global symbols which can and should not be exported and there may be global symbols you don't want to share. It is obviously very late to start using a dllexport construct in the sources (which may not be available in the programming language, anyway and it may be necessary to implemented that as a commment). But at least this would clearly describe the public interface, which
is always a good idea.

Hartmut Becker

[Hein van den Heuvel]

Back in the VAX days I worked on an application (TMS) which required a double link. I don't remember the exact details, but I might be able to find them.
I seem to recall is was also to do with 'based' images which we manually dynamically activated (pre- lib$find_image_symbol).
We had a main calling to a sub, but the sub called to functions or needed data from main which we at that point did not thing we could simply capture in a shared library / common between them.
To link main we presented it with a list of dummy symbols, just to get going. With a main (and its symbol table?!) we linked the subs. Finally we relinked main against the real subs with their real addresses.
Yuck, but it was the best we could come up with at the time.

fwiw,
Hein.

[John Gillings]

Tim,

Maybe an architectural change might help?

I find "naked" data structures exported from shareable images always cause trouble!

Rather than using symbol vector entries and mailboxes to locate your data, why not have a single function (say) LOCATE_DATA in the shareable image to which you pass the name of the object you want the address of. Since the routine is inside the shareable image it knows the address (in FORTRAN you can find it with %LOC). Code can be generated from the data declarations. Depending on the number of objects and frequency you expect to use this, you could have a simple sequential search:

INTEGER FUNCTION LOCATE_DATA(name)
IMPLICIT NONE
CHARACTER*(*) name
IF (name.EQ.'A')THEN
LOCATE_DATA=%LOC(A)
ELSE IF (name.EQ.'B')THEN
LOCATE_DATA=%LOC(B)
...
ELSE
LOCATE_DATA=0
ENDIF
RETURN
END

for more intensive use, build a tree (see the LIB$*TREE routines) or binary search through an array.

An alternative would be to return an array of name/address pairs to the caller, and let the caller deal with the lookups.

This code, and code generator would be the same on VAX, Alpha and IA64.

(regarding the EXPORT and PUBLISH - I remember some discussion about this concept, to which I strongly objected, listing several reasons why it would cause far more problems than it solves, maybe they listened? Sorry ;-)

[Tim Yeung_1]

At this stage, it's too much work to change the design. There are over 7 million lines of code. We are not creating any more global sections, but the software is still being enhanced and we get new data items everytime we build.

We have downloaded the ELF formats and have made tremendous progress in fixing SYMVEC.EXE to read an ELF STB file. We should should have at least one program built for IA64 by the end of this week.

Thanks for all the suggestions.

Tim

[Willem Grooters]

Just a thought.
If you require the entrypoints and interface definitions, it might be that the OBJ2IDL program in the WSIT 2.0 toolkit could be of help. It creates a XML file you could parse for the required data. Worth a try?

WG