SAM corrupted

[dictum9]

This is RX4640 running 11.23

SAM didn't even start and would dump core. So I downloaded PHCO_34208 (Critical) SAM patch and all dependecies, and when tried to install them, got:


ERROR: File "/var/adm/sw/products/SystemAdmin/SAM-HELP/INFO" does not xx
x x exist. xx
x x ERROR: Internal error. Could not open the catalog (INFO) for a xx
x x sparse filest. This error occurred in function xx
x x "IA_SparseDepsInstallChkAncestor", source file "a_check.c", xx
x x line 4539. Please give this information to your xx
x x Hewlett-Packard support contact. xx
x x ERROR: Cannot allocate memory. Either there is no memory available xx
x x or the memory management system is corrupted. xx
x x ERROR: Cannot continue the Analysis Phase until the previous errors x
x x are corrected.

[IT_2007 · Accepted solution]

Make sure that you have enough memory available using top or glance or swapinfo commands. Looks like memory issue.

[inventsekar_1]

Hi /etc,
Note that,
ERROR: Cannot allocate memory. Either there is no memory available or the memory management system is corrupted.

verify the disk space with bdf.

[dictum9]

thank you.

swapinfo checks out fine. It has 16GB of RAM and 16GB of swap.

BTW, SAM has never ran on this (new) system, one time it totally locked up.


PS 10 pts to all answers.

[Steven E. Protter]

Shalom,

Seems to me the patch download was incomplete or corrupt.

swconfigure \*

That may fix it.

If not, download the patch again, swremove the old try and swinstall again.

SEP

[Pupil_1]

Looks like the patch has not been downloaded properly. Try downloading the patch again !!

Also, could you give the o/p of the command file core (ie the core created when sam is run)

[dictum9]

bdf checks out fine. Nothing unusual I see in either dmesg, syslog or anything at all.


Now you meant swconfig? I ran it just like you had it, and nothing. Fromthat SAM patch bundle, nothing at all got installed, it failed analysis, so I doubt swconfig did anything anyway.

I will try to redownload it and reinstall it. I choose the .tar format, maybe that's the problem. Although I was careful to choose "bin" during the ftp process.

[Pupil_1]

Might be worth using swverify as well

[Sean Dale]

This may be a longshot, but, the Patch depot could be corrupt.

[Mel Burslan]

Is this system really doing anything ? From the question and the errors you listed, it seems like there is something deep down inside the OS which is wrong, not only SAM. Are you experiencing any other difficulties ? How does your syslog look ? Is it logging any errors ?

[inventsekar_1]

i usually download as gzip file.
and after downloading it, u can check the download as in this link:

about file fingerprints:

http://www2.itrc.hp.com/service/patch/wrap.do?pageKey=patch.html.fileFingerprintsHelp&BC=main|pdb|search|patchDetail{PHCO_34208,{hpux:11.23,}}|cart|

also
man cksum

[Jonathan Fife]

Why did the patch bundle say it failed analysis?

Seems like a good place to start looking for corruption or missing dependencies for SAM.

[dictum9]

Redownloading and reinstalling (or attempting to) did nothing. Same error. I looked at the error and manually created the following file. Then it worked fine.

# ll /var/adm/sw/products/SystemAdmin/SAM-HELP/INFO
-r--r--r-- 1 root sys 7603 Aug 15 16:24 /var/adm/sw/products/SystemAdmin/SAM-HELP/INFO

During the install, it said it would reboot, but didn't. I ran swlist and got:



#: swlist
WARNING: Incomplete definition for the product "PHKL_32602", beginning
at line 25445. The following attributes are not defined:
filesets the filesets contained in the product
WARNING: Incomplete definition for the product "ISEE-SysInfo",
beginning at line 35807. The following attributes are not
defined:
filesets the filesets contained in the product
WARNING: Incomplete definition for the product "PHCO_31670", beginning
at line 57247. The following attributes are not defined:
filesets the filesets contained in the product
WARNING: Incomplete definition for the product "PHCO_31674", beginning
at line 57565. The following attributes are not defined:
filesets the filesets contained in the product
WARNING: Incomplete definition for the product "PHCO_31657", beginning
at line 57606. The following attributes are not defined:
filesets the filesets contained in the product
ERROR: Duplicate definition for the product "PHCO_31657", beginning
at line 57606. This product defines the same values for the
same version attributes as another product or bundle contained
in the root (installation). Those attributes are

PHCO_31657,l=/,r=1.0,a=HP-UX_B.11.23_IA/PA,v=HP

The duplicate product has been marked as corrupt, and its tag
attribute changed to "_product_57606".
WARNING: Incomplete definition for the product "VRTSvmpro", beginning
at line 72870. The following attributes are not defined:
filesets the filesets contained in the product
WARNING: Incomplete definition for the product "PHCO_32519", beginning
at line 78530. The following attributes are not defined:
filesets the filesets contained in the product
WARNING: Incomplete definition for the product "PHCO_32925", beginning
at line 79831. The following attributes are not defined:
filesets the filesets contained in the product



What does it all mean?

[dictum9]

Mel Burslan


You hit the nail squarely on the head, actually. SAM is just a symptom of the problem. There was OS corruption, something about the inode. (See my other thread (weird inode error on Itanium))

I just rebuilt this system, I saved a copy of the /usr, /var and /opt filesystems in another Lvol, did newfs on these lvols, then copied the info back. However, I did not create a tarball like I should have, just used the cp command to copy the entire filesystem. I wonder if cp missed something, like links, or hidden files.

[inventsekar_1]

yes. swconfig only. not swconfigure.

The "\*" software specification selects all products.

also refer the swconfig log file:
/var/adm/sw/swconfig.log

[dictum9]

I ran it, but it's not the problem. There is some kind of filesystem corruption involved. Also it didn't reboot when it was installing that patch bundle. I am rebooting it and will see if the corruption goes away, and sam starts.

[Patrick Wallek]

It is entirely possible that something did not copy correctly. Especially if there were errors in a filesystem.

At this point I would be very tempted to recover from an Ignite tape that you ran prior to the errors starting, or if the system is not yet in production, reinstall from scratch.

[inventsekar_1]

Hi /etc,
what command u used to install the patch?

is it:
swinstall -x autoreboot=true -x patch_match_target=true -s /tmp/PHCO_34208.depot

refer the 34208 patch installation instructions:

http://www2.itrc.hp.com/service/patch/patchDetail.do?patchid=PHCO_34208&sel={hpux:11.23,}&BC=main|pdb|search|#Installation%20Instructions:

[Mel Burslan]

I second Patrick's opinion of "Resinstall from scratch" unless you have another, properly working rx4640 available to take a ignite image of.

When building systems, copying whole filesystems from other servers or previous installations, is inviting trouble. Even if you had tar'ed up and restored, you will never know if there was a piece of code that was specific to the other server's hardware configuration.

Trust the system installation procedure to build the system properly. If you need modifications on the system, perform them as they are done normally. For instance, if you need an identical kernel to another machine, parameter-wise, you get the kmtune output and modify the necessary parameters and re-compile a new kernel, instead of copying the kernel from another system and expecting it to run here. These examples can be expanded to copying patches, relevant to the source server but damaging on the current server's configuration.

Reinstall, and you will have better results I am sure.

[dictum9]

It looks like a reinstall is the solution. I just cannot believe a system got so badly corrupted for an unknown reason. I would like to at least understand what happened.

It's a new system that was installed not too long ago, so not much work has been done to it... but I hope to at least recover some files if I can boot into single user mode.

[Mel Burslan]

have you by any chance copied the whole or partial contets of /stand from another server into this one ? or /usr, /var ? Usually, abrubt changes in any of these, /stand being the biggest culprit, can hose the system this bad.

[dictum9]

Yeah, it got hosed real bad...

I did :

shutdown -r
ISL> hpux -is boot
mount /usr

/problem = the problem filesystem (in my case, /usr /var and opt)

cp /problem /temp_problem

newfs -F vxfs /problem
cp -r /tmp/problem /problem

reboot

install PHCO_34208(Critical)s700_800 11.23 cumulative SAM patch

[dictum9]

I did not copy anything from another system

[inventsekar_1]

Hi /etc,
1. u only have the root access or anybody else have root access?
2. and to identify what happend one thing is the logfiles:

/var/adm/sw/swagentd.log
/var/adm/sw/swinstall.log
/var/adm/sw/swagent.log
/var/adm/sw/swmodify.log
/var/adm/sw/swconfig.log
/var/adm/syslog/syslog.log

the next idea is that, u can contact HP and from the core dump, they can find what happend.
in this link u can find "little" information on how to analyse crash dumps.
http://docs.hp.com/en/B2355-90950/ch05s05.html#bajchebf

crashutil command can help u little bit.
its man page:
http://docs.hp.com/en/B2355-60105/crashutil.1M.html

[dictum9]

from message when trying to boot:


ERROR: Your system crashed before I/O and dump configuration was complete.
This system does not support a crash dump under these circumstances.