- Community Home
- >
- Servers and Operating Systems
- >
- Operating Systems
- >
- Operating System - OpenVMS
- >
- Re: SHOW PROCESS/CONT shows PC=2BAD2BAD.CODE2BAD?
Categories
Company
Local Language
Forums
Discussions
Forums
- Data Protection and Retention
- Entry Storage Systems
- Legacy
- Midrange and Enterprise Storage
- Storage Networking
- HPE Nimble Storage
Discussions
Discussions
Discussions
Forums
Forums
Discussions
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
- BladeSystem Infrastructure and Application Solutions
- Appliance Servers
- Alpha Servers
- BackOffice Products
- Internet Products
- HPE 9000 and HPE e3000 Servers
- Networking
- Netservers
- Secure OS Software for Linux
- Server Management (Insight Manager 7)
- Windows Server 2003
- Operating System - Tru64 Unix
- ProLiant Deployment and Provisioning
- Linux-Based Community / Regional
- Microsoft System Center Integration
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Community
Resources
Forums
Blogs
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО02-23-2010 10:54 AM
тАО02-23-2010 10:54 AM
Re: SHOW PROCESS/CONT shows PC=2BAD2BAD.CODE2BAD?
SDA> v q vcb+vcb$l_fcbfl
Queue is complete, total of 168 elements in the queue
Hein:
The system is a cluster of 3 nodes running V8.3 (no -1Hx), and was up to date with all V8.3 patches as of a year ago. The application runs on only one now, the other's mostly idle, the third is a quorum system.
Mon Mode shows 1% user mode, ~70% kernel mode, ~10% MPSYNC, on a 4 processor rx3600.
The PC at DEL_EXTFCB_C+360 matches what you have, +350 = cmp4.eq p7, p0=11,r28 and +380 = cmp4.eq p0, p8=11, r30
I'm attaching the output of the SPL script. As you can see, tons of FILSYS time.
I neglected to mention that the process is running IBM MQ Series client code. I don't see how this could relate, there are 4 channels to the fragmented disk, none seem to relate to MQ. The executable image lives there, and 3 logging files do as well. One's 1000 blocks, the other two are much smaller.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО02-24-2010 03:26 AM
тАО02-24-2010 03:26 AM
Re: SHOW PROCESS/CONT shows PC=2BAD2BAD.CODE2BAD?
interesting data. Let me give you my interpretation:
Some XQP operation is being executed around 55000 times per second. This operation involves 5 routines, which acquire and release the FILSYS spinlock during operation:
MAPVBLK, UPDATE_EOF, CREATE_FCB, UPDATE_FILESIZE, ALLOCATE
Routine BUILD_EXT_FCBS calls at least 4 of these routines directly (3) or indirectly (1).
DEL_EXTFCB acquires the FILSYS spinlock about 23 times per second, but holds it for a very LONG time (about 28 ms !), so the DEL_EXTFCB operation under the FILSYS lock seems to be causing a problem. This is being confirmed by the PCS data !
Somehow a file with many extension headers must be involved in this...
Did you look at MONITOR FCP ? This should also show some unusual numbers.
Volker.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО02-24-2010 08:44 AM
тАО02-24-2010 08:44 AM
Re: SHOW PROCESS/CONT shows PC=2BAD2BAD.CODE2BAD?
$ DFU SEARCH/MULTIPLE diskname
Purely Personal Opinion
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО02-24-2010 10:56 AM
тАО02-24-2010 10:56 AM
Re: SHOW PROCESS/CONT shows PC=2BAD2BAD.CODE2BAD?
another question comes to mind:
why is all the FILSYS spinlock activity happening on CPU 1 ? Is there a process with affinity set to CPU 1 ?
You may also want to edit SPL.COM to add
$ write outcom "spl start col/spin=FILSYS"
$ write outcom "wait 00:00:05"
$ write outcom "spl show col"
because this is really the only 'interesting' spinlock on this system.
Volker.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО02-26-2010 09:29 AM
тАО02-26-2010 09:29 AM
Re: SHOW PROCESS/CONT shows PC=2BAD2BAD.CODE2BAD?
What I don't understand is why a channel to this file doesn't show up in a SDA> SHOW PROC/CHAN, otherwise I would have found this much sooner. I don't know what IBM/MQ is doing.
Also, the drive I was discussing all along has never been more than 5% full. It is surprising that the drive was as fragmented as it was.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО02-26-2010 09:40 AM
тАО02-26-2010 09:40 AM
Re: SHOW PROCESS/CONT shows PC=2BAD2BAD.CODE2BAD?
maybe that file was not permanently open ?!
DEL_EXTFCB was being called 23 times per second to delete those 2000+ FCBs (one FCB per extension file header). This gives about 46000+ FILSYS acquisition and releases per seconds for each of all the 6 routines involved, giving about a total 276000+ acq/s. SPL.COM reported about 332000 acq/s, so this was pretty close...
So everything I concluded from the data you've provided seems to have been true ;-)
Nice problem and analysis,
Volker.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО02-26-2010 12:53 PM
тАО02-26-2010 12:53 PM
Re: SHOW PROCESS/CONT shows PC=2BAD2BAD.CODE2BAD?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО02-27-2010 06:15 AM
тАО02-27-2010 06:15 AM
Re: SHOW PROCESS/CONT shows PC=2BAD2BAD.CODE2BAD?
The high acquisition and release rate for FILSYS are in other routines for example MAP VBNs. The long waits are in MAP VBNs from other PIDs. It didn't only take much time to delete all the extension FCBs, it also blocked other applications "on this disk".
It very likely looks like MQ tried (still tries?!) to mimic Unix behavior on VMS, where you can always see what's just written at the end of a (log) file. At least for some situations there are better ways than to open/close.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО02-28-2010 03:14 PM
тАО02-28-2010 03:14 PM
Re: SHOW PROCESS/CONT shows PC=2BAD2BAD.CODE2BAD?
>It very likely looks like MQ tried (still
>tries?!) to mimic Unix behavior on VMS,
>where you can always see what's just written
>at the end of a (log) file. At least for
>some situations there are better ways than
>to open/close.
Not that it helps with your MQ code...
If you want to be able to see the end of a "live" file, rather than force an update of the file on every write (frequent and expensive), it's better to put the resource burden on the reader. So, every time you want to read the file, you do something to make sure the EOF is up to date. Since reading is likely to be much less frequent than writing, the overall cost is far less.
The simplest way to achieve this in OpenVMS is for the writer to open the file SHARE=READ+WRITE. A process wanting to read the file opens it for APPEND access SHARE=READ+WRITE then closes the file. This causes the existing writer to flush all buffers and update the EOF mark. You can now re-open the file READ access SHARE=READ and see recent output.
Note that even if you don't have access to the source code, you can usually trick any program into shared write using a process permanent file with a small DCL jacket:
$ CREATE/FDL="RECORD;FORMAT STREAM_LF"
$ OPEN/APPEND/SHARE=(READ,WRITE) LOG
$ DEFINE/USER SYS$OUTPUT LOG
$
$ CLOSE LOG
now the reader side:
$ OPEN/APPEND/SHARE=WRITE LOG
$ CLOSE LOG
$ TYPE/TAIL
- « Previous
- Next »