/ file system mounted in /dev/root file automatically how?

[HPquestion]

Its HP UX 11.23. Today noticed that ROOT file system got mounted in /dev/root instead of
/dev/vg00/lvol3.

Some how / entry in /etc/fstab got changed ( in april 2007) .And in between we had 2 reboots and we didn't have any error at all during the reboot. Looks like OS is smart enough to use /dev/root automatically? Could anybody please explain me the possibilities.
Are we in trouble?

[Robert-Jan Goossens]

Hi,

Could you post the output of

# lvdisplay -v /dev/vg01/lvol3

Regards,
Robert-Jan

[Laurent Menase]

It uses the information positionned by lvlnboot
for /dev/root.

[Robert-Jan Goossens]

make that vg00 :-)

# lvdisplay -v /dev/vg00/lvol3

[HPquestion]

lvdisplay -v /dev/vg00/lvol3
--- Logical volumes ---
LV Name /dev/vg00/lvol3
VG Name /dev/vg00
LV Permission read/write
LV Status available/syncd
Mirror copies 1
Consistency Recovery MWC
Schedule parallel
LV Size (Mbytes) 576
Current LE 36
Allocated PE 72
Stripes 0
Stripe Size (Kbytes) 0
Bad block off
Allocation strict/contiguous
IO Timeout (Seconds) default

--- Distribution of logical volume ---
PV Name LE on PV PE on PV
/dev/dsk/c2t1d0s2 36 36
/dev/dsk/c2t0d0s2 36 36

--- Logical extents ---
LE PV1 PE1 Status 1 PV2 PE2 Status 2
00000 /dev/dsk/c2t1d0s2 00275 current /dev/dsk/c2t0d0s2 00275 current
00001 /dev/dsk/c2t1d0s2 00276 current /dev/dsk/c2t0d0s2 00276 current
00002 /dev/dsk/c2t1d0s2 00277 current /dev/dsk/c2t0d0s2 00277 current
00003 /dev/dsk/c2t1d0s2 00278 current /dev/dsk/c2t0d0s2 00278 current
00004 /dev/dsk/c2t1d0s2 00279 current /dev/dsk/c2t0d0s2 00279 current
00005 /dev/dsk/c2t1d0s2 00280 current /dev/dsk/c2t0d0s2 00280 current
00006 /dev/dsk/c2t1d0s2 00281 current /dev/dsk/c2t0d0s2 00281 current
00007 /dev/dsk/c2t1d0s2 00282 current /dev/dsk/c2t0d0s2 00282 current
00008 /dev/dsk/c2t1d0s2 00283 current /dev/dsk/c2t0d0s2 00283 current
00009 /dev/dsk/c2t1d0s2 00284 current /dev/dsk/c2t0d0s2 00284 current
00010 /dev/dsk/c2t1d0s2 00285 current /dev/dsk/c2t0d0s2 00285 current
00011 /dev/dsk/c2t1d0s2 00286 current /dev/dsk/c2t0d0s2 00286 current
00012 /dev/dsk/c2t1d0s2 00287 current /dev/dsk/c2t0d0s2 00287 current
00013 /dev/dsk/c2t1d0s2 00288 current /dev/dsk/c2t0d0s2 00288 current
00014 /dev/dsk/c2t1d0s2 00289 current /dev/dsk/c2t0d0s2 00289 current
00015 /dev/dsk/c2t1d0s2 00290 current /dev/dsk/c2t0d0s2 00290 current
00016 /dev/dsk/c2t1d0s2 00291 current /dev/dsk/c2t0d0s2 00291 current
00017 /dev/dsk/c2t1d0s2 00292 current /dev/dsk/c2t0d0s2 00292 current
00018 /dev/dsk/c2t1d0s2 00293 current /dev/dsk/c2t0d0s2 00293 current
00019 /dev/dsk/c2t1d0s2 00294 current /dev/dsk/c2t0d0s2 00294 current
00020 /dev/dsk/c2t1d0s2 00295 current /dev/dsk/c2t0d0s2 00295 current
00021 /dev/dsk/c2t1d0s2 00296 current /dev/dsk/c2t0d0s2 00296 current
00022 /dev/dsk/c2t1d0s2 00297 current /dev/dsk/c2t0d0s2 00297 current
00023 /dev/dsk/c2t1d0s2 00298 current /dev/dsk/c2t0d0s2 00298 current
00024 /dev/dsk/c2t1d0s2 00299 current /dev/dsk/c2t0d0s2 00299 current
00025 /dev/dsk/c2t1d0s2 00300 current /dev/dsk/c2t0d0s2 00300 current
00026 /dev/dsk/c2t1d0s2 00301 current /dev/dsk/c2t0d0s2 00301 current
00027 /dev/dsk/c2t1d0s2 00302 current /dev/dsk/c2t0d0s2 00302 current
00028 /dev/dsk/c2t1d0s2 00303 current /dev/dsk/c2t0d0s2 00303 current
00029 /dev/dsk/c2t1d0s2 00304 current /dev/dsk/c2t0d0s2 00304 current
00030 /dev/dsk/c2t1d0s2 00305 current /dev/dsk/c2t0d0s2 00305 current
00031 /dev/dsk/c2t1d0s2 00306 current /dev/dsk/c2t0d0s2 00306 current
00032 /dev/dsk/c2t1d0s2 00307 current /dev/dsk/c2t0d0s2 00307 current
00033 /dev/dsk/c2t1d0s2 00308 current /dev/dsk/c2t0d0s2 00308 current
00034 /dev/dsk/c2t1d0s2 00309 current /dev/dsk/c2t0d0s2 00309 current
00035 /dev/dsk/c2t1d0s2 00310 current /dev/dsk/c2t0d0s2 00310 current

[Robert-Jan Goossens]

Looks okay.

# mv /etc/mnttab /etc/mnttab.old
# mount -a
to recreate the mnttab file

# bdf

Robert-Jan

[HPquestion]

Thanks for all your help! We have entry in /etc/fstab.
/lvol3 / vxfs delaylog 0 1

And we have the befow line in /etc/mntab
/dev/root / vxfs ioerror=nodisable,log,dev=40000003 0 0 1195429301

Should I change fstab to /dev/root to match with /etc/mnttab file?

This is production box and the last 2 reboot worked correctly.

[Mridul Shrivastava]

The above answer tell you how to fix the problem.
But the usual reason for having /dev/root mounted on / is that someone completed a boot (init 3?) while the system was in LVM maintenance mode.
One should always do a reboot or shutdown -ry 0 from LVM maintenance mode.

[Robert-Jan Goossens]

You do not have to change the /etc/fstab file, you just have to recreate the /etc/mnttab file. There is no harm in doing that online on a production box.

# mv /etc/mnttab /etc/mnttab.old
# mount -a
will recreate the mnttab
# bdf
to check.

[Mridul Shrivastava]

It shouldn't be /dev/root so don't change fstab...

Please rename the mnttab file and then recreate it using mount -a
Then execute bdf it should show /dev/vg00/lvol3 mounted on root. This is the correct.

[HPquestion]

Thanks! Last two reboot (clean reboot) worked correctly with /dev/root. Problem is that our fstab has a typo.
Instead of
/dev/vg00/lv0l3 / vxfs delaylog 0 1

we have the below entry there.
/lvol3 / vxfs delaylog 0 1

We did clean reboot earlier. We didn't use init 3.

[Mridul Shrivastava]

Then I would recommend to populate the fstab with the correct information, change it to /dev/vg00/lvol3 and then follow the steps mentioned above.

It is purely safe and will not cause any harm to the system.

/dev/root means in past it has been booted in run level 3 from LVM maintenance mode. This is not a normal scenario.

[Robert-Jan Goossens]

Then you will have to edit the /etc/fstab, change lvol3 into /dev/vg00/lvol3 and then move and recreate the mnttab file.

[HPquestion]

Thanks again! As its a production system do you see any possibily of root file system corruption if I exeute the suggested changes? I am still trying to understand the pieces before I do anything in production :-) .Is that mean my / files which are located in /dev/root will be moved to /dev/vg00/lvo3 if I do the change?

Current BDF output is :
/dev/root 589824 324944 262816 55% /

root>cd /
root>ls
.gpmhp-ricr3prd home sapmnt
.hh backup lib sbin
.lsof_ricr3prd bin lost+found stand
.mozilla dbcopy net tmp
.mozilla-license dead.letter newprofile transports
.profile depot opt usr
.q4rc.pl dev oracle var
.sh_history devnull patches.txt
.ssh dvd sapbasis
.sw etc sapinst

[Robert-Jan Goossens · Accepted solution]

--
Thanks again! As its a production system do you see any possibily of root file system corruption if I exeute the suggested changes? I am still trying to understand the pieces before I do anything in production :-) .
--

There is no possibily of root filesystem corruption.

--
Is that mean my / files which are located in /dev/root will be moved to /dev/vg00/lvo3 if I do the change?
--

Well yes and no, curtain HP commands will not work properly like ignite's make_tape/net_recovery commands.

Regards,
Robert-Jan

[HPquestion]

"Well yes and no, curtain HP commands will not work properly like ignite's make_tape/net_recovery commands."

Now I am really scared. Is that means that I may loose some files and directories as it will move the existing stuff from /dev/root to /dev/vg00/lv03?

[Robert-Jan Goossens]

No :-) you will not loose files, your / root filesystem itself will not change, you just have to change the mount information/administration regarding the root filesystem.

[HPquestion]

Thanks again! Our /etc/fstab file got corrupted. I found 2 lines there where our previous admin made the typos. After that we had around 2 reboot (using sam shutdown -r option), do you think that
fstab typo/corruption can lead OS to
mount root in /dev/root?

[Mridul Shrivastava]

yes there is a possibility of that. I still think that someone had booted it to level 3 from LVM then might have noticed /dev/root instead of /dev/vg00/lvol3.
After that he was trying to change the fstab accordingly and it got corrupted ,hence it is still mounted on /dev/root even after 2 successful reboots.

[HPquestion]

Thanks again!

Questions on fixing the error:
1.As its a SAP production box , do we need to shutdown sap and database before doing "mount -a" .
2.Looks like we had /dev/root situation happened 2 months before. In between I had to increase one file system which is mounted at /usr/sap. Do you see any error if I have to do "mount -a" after fixing the errors in fstab file?

3. Trying to understand more about /etc/mnttab file. When /etc/mnttab file get updated and used ?
Is it during the reboot and "mount -a"? Is fstab feeds mnttab all the time?
If fstab file get corrupted due to typo or so,how /etc/mnttab get updated during reboot?


Questions on " Why it happened"

I did all the reboots and always used shutdown -r.Didn't use init 3 at all.

Do we have any system log other than /var/adm/syslog to check the previous reboot procedures happened in the system?

Last time we had a power crash on Nov 18th and I had to do "shutdown -h" to shutdown the box.

Is he below syslog looks normal?


Nov 18 11:50:14 guard_tap[28917]: connect: Network is unreachable
Nov 18 11:50:32 su: + console -root
Nov 18 11:54:29 HP-PRM: [9446]: prmconfig: configuration reset
Nov 18 11:55:11 /usr/sbin/envd[2665]: terminated by signal 15
Nov 18 11:55:12 diagmond[2655]: Exit due to user requested abort
Nov 18 11:55:13 FontServer[2433]: terminating
Nov 18 11:55:13 sshd[1539]: Received signal 15; terminating.
Nov 18 11:55:28 cimserver[9716]: Shutdown timeout expired. Forced shutdown initiated.
Nov 18 11:55:41 inetd[1748]: Going down on signal 15
Nov 18 11:59:27 rpcbind: terminate: rpcbind terminating on signal. Restart with "rpcbind -w"
Nov 18 11:59:28 su: + tty?? root-sfmdb
Nov 18 11:59:32 syslogd: going down on signal 15

[Robert-Jan Goossens]

--
1.As its a SAP production box , do we need to shutdown sap and database before doing "mount -a".
--

No

--
2.Looks like we had /dev/root situation happened 2 months before. In between I had to increase one file system which is mounted at /usr/sap. Do you see any error if I have to do "mount -a" after fixing the errors in fstab file?
--

Please post both errors in the /etc/fstab, the one about /dev/root can be easily resolved with the mv and the mount -a command.

And please post the the output of the lvlnboot -v command

--
3. Trying to understand more about /etc/mnttab file. When /etc/mnttab file get updated and used ?
--

The /etc/mnttab is updated after all mount and umount commands.

--
Is it during the reboot and "mount -a"? Is fstab feeds mnttab all the time?
--

Yes

--
If fstab file get corrupted due to typo or so,how /etc/mnttab get updated during reboot?
--

If there is an error in the lvm configuration of the root (/) filesystem, HPUX will use the /dev/root as an emergency device at boot time otherwise you can not boot the server.

--
Questions on " Why it happened"

I did all the reboots and always used shutdown -r.Didn't use init 3 at all.

Do we have any system log other than /var/adm/syslog to check the previous reboot procedures happened in the system?
--

As to the why, someone could made an typo in the fstab or he/she could have booted the system into lvm maintance mode and isuing the sutdown command but the init 3 command.

An other answer to the why could be if someone made a mistake during setup of the mirror boot confuguration. The output of the lvlnboot -v command will answer that.

shutdown log file
/etc/rc.log.old

--
Last time we had a power crash on Nov 18th and I had to do "shutdown -h" to shutdown the box.

Is he below syslog looks normal?
--

Yes looks like a normal shutdown command has been issued.

--
Nov 18 11:50:14 guard_tap[28917]: connect: Network is unreachable
Nov 18 11:50:32 su: + console -root
Nov 18 11:54:29 HP-PRM: [9446]: prmconfig: configuration reset
Nov 18 11:55:11 /usr/sbin/envd[2665]: terminated by signal 15
Nov 18 11:55:12 diagmond[2655]: Exit due to user requested abort
Nov 18 11:55:13 FontServer[2433]: terminating
Nov 18 11:55:13 sshd[1539]: Received signal 15; terminating.
Nov 18 11:55:28 cimserver[9716]: Shutdown timeout expired. Forced shutdown initiated.
Nov 18 11:55:41 inetd[1748]: Going down on signal 15
Nov 18 11:59:27 rpcbind: terminate: rpcbind terminating on signal. Restart with "rpcbind -w"
Nov 18 11:59:28 su: + tty?? root-sfmdb
Nov 18 11:59:32 syslogd: going down on signal 15

[HPquestion]

root>lvlnboot -v
Boot Definitions for Volume Group /dev/vg00:
Physical Volumes belonging in Root Volume Group:
/dev/dsk/c2t1d0s2 (0/1/1/0.1.0) -- Boot Disk
/dev/dsk/c2t0d0s2 (0/1/1/0.0.0) -- Boot Disk
Boot: lvol1 on: /dev/dsk/c2t1d0s2
/dev/dsk/c2t0d0s2
Root: lvol3 on: /dev/dsk/c2t1d0s2
/dev/dsk/c2t0d0s2
Swap: lvol2 on: /dev/dsk/c2t1d0s2
/dev/dsk/c2t0d0s2
Dump: lvol2 on: /dev/dsk/c2t1d0s2, 0

Current path "/dev/dsk/c15t0d1" is an alternate link, skip.
Current path "/dev/dsk/c9t0d1" is an alternate link, skip.
Current path "/dev/dsk/c11t0d1" is an alternate link, skip.
Current path "/dev/dsk/c15t0d2" is an alternate link, skip.
Current path "/dev/dsk/c9t0d2" is an alternate link, skip.
Current path "/dev/dsk/c11t0d2" is an alternate link, skip.
Current path "/dev/dsk/c15t0d3" is an alternate link, skip.
Current path "/dev/dsk/c9t0d3" is an alternate link, skip.
Current path "/dev/dsk/c11t0d3" is an alternate link, skip.

-------------------------
2 Errors on /etc/fstab:
There is /lov0l3 instead of /dev/dsk/lvol3.
And mount point is same for the other 2 entry. Will remove ricy entry and fix /lov0l3 before doing mount -a.


/lvol3 / vxfs delaylog 0 1
ricx:/usr/sap/trans /usr/sap/trans nfs rw,suid 0 0
ricy:/usr/sap/trans /usr/sap/trans nfs rw,suid 0 0

[Robert-Jan Goossens]

Thew output from the lvlnboot command looks fine! The duplicate entry of the nfs /usr/sap/trans mount will only produce a warning "/usr/sap/trans ia allready mounted"

Regards,
Robert-Jan

[HPquestion]

Thanks again! Can we say that our /dev/root situation happened only because of fstab file error and the problem will go away with the above sugested solution without any possible impacts? Please confirm and will do the changes next week asap.

[Laurent Menase]

:: Can we say that our /dev/root situation happened only because of fstab file error and the problem will go away with the above sugested solution without any possible impacts? Please confirm and will do the changes next week asap.


Yes, because when booting the system is only getting the lvlnboot information to mount the root filesystem.
it uses /dev/root which is an abstraction for for the root device configured by lvlnboot.

then when booting, it will remount the fs according to the fstab. If lvol3 entry is corrupted, then /dev/root will stay mounted.