sd commands (except swlist) failing on permissions

[Don Ferderber]

Hmmm ... tried turing on the debug mode as suggested and here is the output ... the first part shows what happened when swagentd daemon restarted, and teh second part is the result of running swverify on one of the patches installed:

DEBUG: Removing /var/adm/sw/standalone file if it exists.

======= 08/28/09 09:43:55 EDT BEGIN swagentd (pid = 17485).

DEBUG: @(#) $Revision: SD-UX/B.11.31.00.177, DAEMON, UDL_PHCO_38226,
Optimized, Build Env /SD/VBE/11.00_990P, Built Jun 2 2008 00:35:17$
DEBUG: Daemon - initializing . . .
DEBUG: ID_InitAgentMgmt() called
DEBUG: Daemon - initializing host management
DEBUG: UTIL_ObtainLockFd: fcntl(F_SETLK) failed. Permission denied (13).
* Restarting swagentd program, process ID 1487.
DEBUG: UTIL_ObtainLockFd: fcntl(F_SETLK) failed. Permission denied (13).

======= 08/28/09 09:43:56 EDT END swagentd (pid = 1487).

* Registered RPC protocol "ncacn_ip_tcp" with endpoint "2121".
* Registered RPC protocol "ncadg_ip_udp" with endpoint "2121".
DEBUG: entering SEC_ServerInit
DEBUG: entering sec_acl_mgr_configure("nil_db")
DEBUG: entering SEC_SetLocalContext
DEBUG: Initialized rpc server, binding ncadg_ip_udp:47.140.176.7[2121]
DEBUG: UTIL_SigInit: Entered, Signals_initialized=0
DEBUG: UTIL_SigInit: sigprocmask(set, all_list)
DEBUG: util_SigHandler initializing, thread priority is 31
DEBUG: UTIL_SigAction called sig 1 act 0x1
DEBUG: UTIL_SigAction called sig 2 act 0x1
DEBUG: UTIL_SigAction called sig 3 act 0x1
DEBUG: UTIL_SigAction called sig 25 act 0x1
DEBUG: UTIL_SigAction called sig 27 act 0x1
DEBUG: UTIL_SigAction called sig 28 act 0x1
DEBUG: UTIL_SigAction called sig 18 act 0x4002ab92
DEBUG: UTIL_SigAction called sig 15 act 0x4002ab9a
DEBUG: UTIL_SigAction called sig 16 act 0x4002ab9a
DEBUG: UTIL_SigAction called sig 17 act 0x4002ab9a
DEBUG: UTIL_SigAction called sig 14 act 0x4002aba2
* Checking job queue at 1 minute intervals.
DEBUG: Stdin, stdout, and stderr flags were read and written with O_NONBLOCK
off. Necessary when SD runs in some libcma environments, harmless
otherwise. The stdin, stdout, and stderr flags were 0x2, 0x2, 0x2,
reset to 0x2, 0x2, 0x2. To disable, set env var
SDU_DEBUG_NO_RESET_NONBLOCK.
DEBUG: Caught signal #14, call func at 0x4002aba2
DEBUG: Sol: enter SOL_QueueOpen.
DEBUG: entered SEC_SetPrivMode()
DEBUG: Sol: enter SOL_FindJobs.
DEBUG: Sol: enter SOL_QueueClose.
DEBUG: RPC entering ID_RpcAgentInit.
DEBUG: sw_rpc_agent_int() called: 10 options, soc /, task 8
DEBUG: entered SEC_CheckClientAccessToObjectPath
DEBUG: UTIL_IsNetAddrLocalBvec, bvec[0]="47.140.176.7", comparing to
"47.140.176.7" and "47.140.176.7"
DEBUG: SEC_GetClientAuthn: local_evidence "(NULL)", client_is_local 1, net_add
r/net_addr_is_local "47.140.176.7"/1, user "root", group "sys.root.other.bin.adm
.daemon.mail.lp.users", localauth_already_done 0
DEBUG: Problem with local_evidence option. Error 0 (0), le="(NULL)", len=0,
filename="(NULL)", rn="(NULL)", rv="(NULL)", debug_deny_local=-1.
ERROR: Cannot authenticate local principal "root".
ERROR: Access denied to start verify agent on /. No user authenticated.
08/28/09 09:45:08 EDT
DEBUG: Caught signal #14, call func at 0x4002aba2
DEBUG: Sol: enter SOL_QueueOpen.
DEBUG: entered SEC_SetPrivMode()
DEBUG: Sol: enter SOL_FindJobs.
DEBUG: Sol: enter SOL_QueueClose.
DEBUG: Caught signal #14, call func at 0x4002aba2
DEBUG: Sol: enter SOL_QueueOpen.
DEBUG: entered SEC_SetPrivMode()
DEBUG: Sol: enter SOL_FindJobs.
DEBUG: Sol: enter SOL_QueueClose.
DEBUG: Caught signal #14, call func at 0x4002aba2
DEBUG: Sol: enter SOL_QueueOpen.
DEBUG: entered SEC_SetPrivMode()
DEBUG: Sol: enter SOL_FindJobs.
DEBUG: Sol: enter SOL_QueueClose.
DEBUG: Caught signal #14, call func at 0x4002aba2
DEBUG: Sol: enter SOL_QueueOpen.
DEBUG: entered SEC_SetPrivMode()
DEBUG: Sol: enter SOL_FindJobs.
DEBUG: Sol: enter SOL_QueueClose.
DEBUG: Caught signal #14, call func at 0x4002aba2
DEBUG: Sol: enter SOL_QueueOpen.
DEBUG: entered SEC_SetPrivMode()
DEBUG: Sol: enter SOL_FindJobs.
DEBUG: Sol: enter SOL_QueueClose.
DEBUG: Caught signal #14, call func at 0x4002aba2
DEBUG: Sol: enter SOL_QueueOpen.
DEBUG: entered SEC_SetPrivMode()
DEBUG: Sol: enter SOL_FindJobs.
DEBUG: Sol: enter SOL_QueueClose.
DEBUG: Caught signal #14, call func at 0x4002aba2
DEBUG: Sol: enter SOL_QueueOpen.
DEBUG: entered SEC_SetPrivMode()
DEBUG: Sol: enter SOL_FindJobs.
DEBUG: Sol: enter SOL_QueueClose.
DEBUG: Caught signal #14, call func at 0x4002aba2
DEBUG: Sol: enter SOL_QueueOpen.
DEBUG: entered SEC_SetPrivMode()
DEBUG: Sol: enter SOL_FindJobs.
DEBUG: Sol: enter SOL_QueueClose.
DEBUG: Caught signal #14, call func at 0x4002aba2
DEBUG: Sol: enter SOL_QueueOpen.
DEBUG: entered SEC_SetPrivMode()
DEBUG: Sol: enter SOL_FindJobs.
DEBUG: Sol: enter SOL_QueueClose.
DEBUG: Caught signal #14, call func at 0x4002aba2
DEBUG: Sol: enter SOL_QueueOpen.
DEBUG: entered SEC_SetPrivMode()
DEBUG: Sol: enter SOL_FindJobs.
DEBUG: Sol: enter SOL_QueueClose.
DEBUG: Caught signal #14, call func at 0x4002aba2
DEBUG: Sol: enter SOL_QueueOpen.
DEBUG: entered SEC_SetPrivMode()
DEBUG: Sol: enter SOL_FindJobs.
DEBUG: Sol: enter SOL_QueueClose.
DEBUG: Caught signal #14, call func at 0x4002aba2
DEBUG: Sol: enter SOL_QueueOpen.
DEBUG: entered SEC_SetPrivMode()
DEBUG: Sol: enter SOL_FindJobs.
DEBUG: Sol: enter SOL_QueueClose.
DEBUG: Caught signal #14, call func at 0x4002aba2
DEBUG: Sol: enter SOL_QueueOpen.
DEBUG: entered SEC_SetPrivMode()
DEBUG: Sol: enter SOL_FindJobs.
DEBUG: Sol: enter SOL_QueueClose.

[Bob E Campbell]

I have been told that this symptom might exist when there is a mixed installation of SD. Can you post the what strings for the files:

/usr/sbin/swagentd
/usr/sbin/swinstall
/usr/lbin/swagent

And while you are at it, swinstall should have 11 or so hard links.

[Don Ferderber]

Hey Bob ... here is the output as requested:

# what /usr/sbin/swagentd
/usr/sbin/swagentd:
$Revision: SD-UX/B.11.31.00.177, DAEMON, UDL_PHCO_38226, Optimized, Bui
ld Env /SD/VBE/11.00_990P, Built Jun 2 2008 00:35:17$
# what /usr/sbin/swinstall
/usr/sbin/swinstall:
$Revision: SD-UX/B.11.31.00.145, CONTROLLER, UDL_PHCO_35587, Optimized,
Build Env /SD/VBE/11.00_990P, Built Feb 16 2007 15:23:37$
# what /usr/lbin/swagent
/usr/lbin/swagent:
$Revision: SD-UX/B.11.31.00.177, AGENT, UDL_PHCO_38226, Optimized, Buil
d Env /SD/VBE/11.00_990P, Built Jun 1 2008 23:55:42$

# ll /usr/sbin/swagentd
-r-xr-xr-x 1 root bin 798720 Jun 2 2008 /usr/sbin/swagentd
# ll /usr/sbin/swinstall
-r-sr-xr-x 11 root bin 2478080 Feb 16 2007 /usr/sbin/swinstall
# ll /usr/lbin/swagent
-r-xr-xr-x 2 bin bin 1810432 Jun 2 2008 /usr/lbin/swagent

.... I know I sound like a broken record, but this only happened after a failed patching run ... I do not know exactly what happened, but I suspect the system was rebooted (unexpectedly) during application of patches. The last few lines of swinstall.log look like this:


# tail -8 /var/adm/sw/swagent.log
* Installing fileset "PHCO_38181.SYSCOM,r=1.0" (7 of 40).
* Installing fileset "PHCO_38226.GZIP,r=1.0" (8 of 40).
* Installing fileset "PHCO_38226.SD-AGENT,r=1.0" (9 of 40).
* File "/usr/sbin/swagentd" could not be removed. It was renamed
and added to the cleanup file "/var/adm/sw/cleanupfile".
* File "/usr/lbin/swconfig" could not be removed. It was renamed
and added to the cleanup file "/var/adm/sw/cleanupfile".
* Installing fileset "PHCO_38226.SD-CMDS,r=1.0" (10 of 40).


Odd ...

[Don Ferderber]

Sorry ..

I said the last few lines of swinstall.log, but I meant swagent.log

[mvpel]

Well there you have it, Don, looks like you were right - the swinstall binary is from PHCO_35587, while the swagentd and swagent binaries are from PHCO_38226, which superseded 35587.

The fileset that was interrupted was:

Installing fileset "PHCO_38226.SD-CMDS,r=1.0" (10 of 40).

... which would make sense.

Perhaps reapply PHCO_38226 with reinstall=true, and see what happens? Or as a quick test, copy an 11.31.00.177 swinstall binary over to /tmp from another system and try it out?

[mvpel]

Duh, you can't reinstall PHCO_38226, sorry. It's been a long day.

But if you don't have another HP you can probably use the "tar" command to pull out the binaries from the .depot file for the SD-CMD fileset and install them by hand, yes?

[Bob E Campbell]

Looks like my expert is just that. That install failed in a 5 second window of badness, kind of funny actually.

I think getting the old SW-DIST.SD-AGENT files out of the save area or putting the SW-DIST.SD-CMDS files onto the system will get you running. If you go into the depot remember they are probably compressed without extentions (cat foo | gzcat > is your friend) and the save area is a place to be careful.

Once you get the tool running install a new version of the SW-DIST product to clean up the rest. Still got my contact info or BH's? Shout if you need some help to get the rest of the way.

[Don Ferderber]

Thanks everyone ...

I will try this on Monday ... its Miller time right now

[Bob E Campbell]

Turns out there is even documentation on recovery:

http://docs.hp.com/en/5992-5875/apc.html

Not that us experts would ever read something ;-)

[mvpel]

Don - what news from the Riddermark?