Repeated /dev/vg00/lvol3 -> /dev/root

[Nico van Royen]

On 2 of our systems (HPUX 10.20), the / entry is changinf from its original mount point /dev/vg00/lvol3 to /dev/root.
The box is not booted into LVM maint. mode, and after a normal reboot de / entry is /dev/vg00/lvol3 (as it should).
Also all other items seems ok, the lvlnboot -v output seems ok, root and boot configs etc.
On random times (at least, it seems random), de entry is changed to /dev/root (als the /etc/mnttab reflects /dev/root).
When fixing the entry by removing the /etc/mnttab and issueing a mount -a, the bdf entry is back to /dev/vg00/lvol3 .. for a while at least. Withing 2 days to a week the entry is again back to /dev/root.

Anyone else had this or know what could cause it?

[Clemens van Everdingen]

Hi,

As far as I recall is going to lvm maintenance mode the cause of this.

Might be single usermode as well.

C.

[Nico van Royen]

The systems where not booted into lvm mode, after normal reboots, the entry is correct (/dev/vg00/lvol3) for a short time (2 to 3 days mostly), then at some time it changes to /dev/root.

[Clemens van Everdingen]

Hi,

Forgot to mention, might also be a result of installing diagnostics.
Might be this ?

C.

[Bill Hassell]

/dev/root is an emergency device file when vg00 is corrupted and/or the boot record area is invalid. If /dev/root exists, there is a serious problem with vg00 definitions. Start with lvlnboot -v to see if it looks normal. If there are errors, you'll need to resolve them. See man lvlnboot, specifically option -R.

[Nico van Royen]

Diagnostics are installed quite a while back (Online diagnostics from the HP support bundle, B.10.20.18.13), other systems (which also have the online-diag tools do not have this problem.

The output of lvlnboot is:

# lvlnboot -v
Boot Defenitions for Volume Group /dev/vg00
/dev/dsk/c1t15d0 (10/4/4.15.0) -- Boot Disk
/dev/dsk/c2t5d0 (10/8.5.0) -- Boot Disk
Boot: lvol1 on: /dev/dsk/c1t15d0
/dev/dsk/c2t5d0
Root: lvol3 on: /dev/dsk/c1t15d0
/dev/dsk/c2t5d0
Swap: lvol2 on: /dev/dsk/c1t15d0
/dev/dsk/c2t5d0
Dump: lvol2 on: /dev/dsk/c2t5d0, 0

Wich seems ok. Also.. the systems are running with the 'problem' for several month already in a production enviroment now without any problem (just ignite complains sometimes)..

[Acer_1 · Accepted solution]

I had resolved it before.
1.change the entry in /etc/mnttab:/dev/root->/dev/vg00/lvol3
2.mv /etc/lvmtab /etc/lvmtab.org
3.vgscan
4.lvlnboot -r /dev/vg00/lvol3
lvlnboot -d /dev/vg00/lvol2
lvlnboot -s /dev/vg00/lvol2
lvlnboot -b /dev/vg00/lvol1
5.reboot it's OK!

hope it helps.

[Nico van Royen]

Its not the point of getting it fixed 'that'easy.. remounting ( (re-)moving the mnttab, mount -a ) DOES fix it temporarily.. (back to /dev/vg00/lvol3 as it should).. the problem is that in x time, the disk switches back to /dev/root for yet unknown reason.

[Clemens van Everdingen]

Hi,

Did you use the solution from Joseph ?
I agree with him, it probably will stay away.

Regards,
C.

[Nico van Royen]

In the past year that the system has this problem, have tried many things (including editing the mnttab).. and although also this fixes it for a while.. several days later.. its back to /dev/root. The same if we reboot, or not.

[Acer_1]

Hi,Nico:
I'd solved the problem using my solution.
Have you try it?I think it should work!
B.R

[Clemens van Everdingen]

Nico,

Try the solution like this (Joseph's solution) !!!

1.change the entry in /etc/mnttab:/dev/root->/dev/vg00/lvol3
2.mv /etc/lvmtab /etc/lvmtab.org
3.vgscan
4.lvlnboot -r /dev/vg00/lvol3
lvlnboot -d /dev/vg00/lvol2
lvlnboot -s /dev/vg00/lvol2
lvlnboot -b /dev/vg00/lvol1
5.reboot it's OK!

C.

[Nico van Royen]

A system reboot will be done later this day (service levels prevent reboots during daytime operation), and we'll see if the problems also stays away. Untill now.. every attempted fix was only temporarily.

[Nico van Royen]

After rebooting, its currently at vg00/lvol3.. lets see if this time it stays this way.
I'll post another message to this thread if the problem yet again occurs.

(to be honest, I wouldn't be suprised if it did).

[Nico van Royen]

Well.. as I suspected,
today April 23, 8.45am.. The same volume /dev/vg00/lvol3 (mounted as / ) is back to the /dev/root.

Any other solutions / idea's ?

[Clemens van Everdingen]

Hi Nico,

So the problem reoccurred !

Let's see:

Do you have a mirrores situation on vg00 ?

Let's check the status of the extents on /dev/vg00/lvol1:

lvdisplay -v /dev/vg00/lvol1

Any stale extents ?

C.

[Clemens van Everdingen]

Hi Nico,

Are there any error/messages in syslog regarding the moment this happend ?

Pls. Post a part from there !

C.

[Nico van Royen]

All (vg00) volumes logical extends are status 'current' and at 'available/syncd' and all have: Mirror copies 1

No wierd stuff in the syslog (see below, although some repeated messages have been left out..).

/var/adm/syslog :
Apr 22 17:59:03 as02a telnetd[1439]: recv: Connection timed out
Apr 22 18:40:01 as02a syslog: su : + tty?? root-p_as
Apr 22 18:40:02 as02a syslog: su : + tty?? root-oracle
Apr 22 23:30:03 as02a syslog: su : + tty?? root-p_as
Apr 23 02:00:11 as02a ftpd[4931]: connection from 10.7.2.130 [10.7.2.130]
Apr 23 02:00:11 as02a ftpd[4931]: FTP LOGIN FROM 10.7.2.130 [10.7.2.130], aojam03
Apr 23 02:00:12 as02a ftpd[4931]: aojam03 of 10.7.2.130 [10.7.2.130] deleted /as/aspkg03/as/jam/aojam03/ABOBS_200204230005_5ESS.dat
Apr 23 02:00:13 as02a ftpd[4931]: aojam03 of 10.7.2.130 [10.7.2.130] deleted /as/aspkg03/as/jam/aojam03/ABOBS_200204230005_AXE.dat
Apr 23 02:00:13 as02a ftpd[4931]: FTP session closed
Apr 23 04:05:53 as02a ftpd[18247]: connection from 10.1.72.1 [10.1.72.1]
Apr 23 04:05:53 as02a ftpd[18247]: FTP LOGIN FROM 10.1.72.1 [10.1.72.1], osgasd
Apr 23 04:06:02 as02a ftpd[18247]: FTP session closed

Apr 23 08:30:05 as02a syslog: su : + ttyp6 royen99-root
--- eof

[Ruediger Noack]

Only an idea...
Do you split your mirror copy because of backup? May be this causes your problem.

Ruediger

[Nico van Royen]

Nop, our backup consists of just an OmniBack backup up the entire filesystem (Weekly full and rest incremental), and once in 3 months an IgniteUX dump.

[Clemens van Everdingen]

Hi,

Did you check with lifls if both disks have same LIF info on them ?

lifls /dev/rdsk/cXtXdX (replace X for both disks).

C.

[Clemens van Everdingen]

Hi,

I am still wondering what action on the system could cause this issue ?
Did anything specific happend before teh situation is back to /dev/root ?

C.

[Nico van Royen]

Clemens,
Here I start to see a diffence now,

the lifls on the /dev/dsk/c2t5d0 (PV1) gives:
ODE MAPFILE SYSLIB CONFIGDATA SLMOD
SLDEV SLDRIVERS SLSCSI MAPPER IOTEST
PERFVER PVCU SSINFO ISL AUTO
HPUX LABEL

while /dev/dsk/c1t15d0 (PV2) has one entry extra on the last line:
HPUX PAD LABEL
(first 3 lines are identical).

- Nico

[Nico van Royen]

If anything specific happened ? Not on the hardware anyway.. The machines are operational for about 4 years now, and in the time only additional software is installed, Updated offline & online diagnostics, predictive, ignite, URM (Axent's Unix Resource Manager.. security tool), IT/Operations, ESM, ITA and MC/Service guard. (the disks used for service guard have each there own VG group).

- Nico

[Clemens van Everdingen]

Hi,

Ok ! Let's see if there are more differences between those discs and what bootstring is used.

Boot string used in kernel:

echo 'boot_string/S'|adb /stand/vmunix /dev/mem

Output looks like this:
boot_string:
boot_string: disc(8/4.5.0;0)/stand/vmunix

Now lets see boot string on disks:

lifcp /dev/dsk/cXtXdX:AUTO $(tty)

Output like this:

hpux (;0)/stand/vmunix

In mirrored situation it should be hpux -lq.

C.