fsck on root FS

[NonGuru]

I have an B2600 w/10.20. The root FS got full and the system started crashing. Cleaned and got it to 65% usage. Now the systems stop autobooting with the lvol3 (root FS) complaining about errors. I've tried cleaning the root FS (even in single user) but it complains about being mounted. I've run dd from rvol3 to check for badblocks and it runs without errors. I'm trying to clean the FS so it boots normaly, any ideas?
Boot capture:
.....cut......
entry 0 - major is 64, minor is 0x2; start = 0, size = 4194304
Unable to add all swap for device: Primary Swap. Increase the tunable parameter maxswapchunks by 512 and re-configure your system.
Starting the STREAMS daemons-phase 1

Checking root file system.
/dev/rroot: UNREF DIR I=67884
OWNER=root MODE=41755
/dev/rroot: SIZE=0 MTIME=Mar 2 10:35 2009 /dev/rroot: UNREF DIR I=112128
OWNER=root MODE=40755
/dev/rroot: SIZE=0 MTIME=Mar 17 05:55 2009 /dev/rroot: UNREF DIR I=112131
OWNER=root MODE=40755
/dev/rroot: SIZE=0 MTIME=Mar 17 05:55 2009 /dev/rroot: UNREF DIR I=118272
OWNER=root MODE=40755
/dev/rroot: SIZE=0 MTIME=Mar 17 05:55 2009 /dev/rroot: UNREF DIR I=118567
OWNER=root MODE=40755
/dev/rroot: SIZE=0 MTIME=Mar 17 05:55 2009 /dev/rroot: UNREF FILE I=15617
OWNER=root MODE=100755
/dev/rroot: SIZE=0 MTIME=Feb 28 23:05 2009 (CLEARED)
/dev/rroot: UNREF FILE I=15618
OWNER=root MODE=100755
/dev/rroot: SIZE=0 MTIME=Mar 9 09:21 2009 (CLEARED)
/dev/rroot: UNREF FILE I=15621
OWNER=root MODE=100755
/dev/rroot: SIZE=0 MTIME=Mar 17 09:00 2009 (CLEARED)
/dev/rroot: UNREF FILE I=15626
OWNER=root MODE=100755
/dev/rroot: SIZE=0 MTIME=Mar 10 22:53 2009 (CLEARED)
/dev/rroot: UNREF FILE I=15628
OWNER=root MODE=100755
/dev/rroot: SIZE=0 MTIME=Mar 10 23:14 2009 (CLEARED)
/dev/rroot: UNREF FILE I=15630
OWNER=root MODE=100755
/dev/rroot: SIZE=0 MTIME=Mar 16 08:43 2009 (CLEARED)
/dev/rroot: UNREF FILE I=15634
OWNER=root MODE=100755
/dev/rroot: SIZE=0 MTIME=Mar 17 08:42 2009 (CLEARED)
/dev/rroot: UNREF FILE I=15636
OWNER=root MODE=100600
/dev/rroot: SIZE=0 MTIME=Mar 12 19:12 2009 (CLEARED)
/dev/rroot: LINK COUNT DIR I=16896
OWNER=root MODE=40755
/dev/rroot: SIZE=1024 MTIME=Mar 17 09:23 2009 COUNT 2 SHOULD BE 3
/dev/rroot: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
Root check done.
Create STCP device files

B2352B/9245XB HP-UX (B.11.00) #1: Wed Nov 5 22:38:19 PST 1997



Memory Information:

physical page size = 4096 bytes, logical page size = 4096 bytes

Physical: 2097152 Kbytes, lockable: 1556812 Kbytes, available: 1792664 Kbytes



/sbin/ioinitrc:
/dev/vg00/lvol1: 36 files, 0 icont, 19533 used, 76168 free (64 frags, 9513 blocks)
insf: Installing special files for pseudo driver framebuf

/sbin/bcheckrc:
Checking for LVM volume groups and Activating (if any exist)
Volume group "/dev/vg00" has been successfully changed.
Checking hfs file systems
/sbin/fsclean: /dev/vg00/lvol3 not ok, run fsck
/sbin/fsclean: /dev/vg00/lvol1 (mounted) ok
/sbin/fsclean: /dev/vg00/lvol4 clean
/sbin/fsclean: /dev/vg00/lvol5 clean
/sbin/fsclean: /dev/vg00/lvol6 clean
FILE SYSTEM(S) NOT PROPERLY SHUTDOWN, BEGINNING FILE SYSTEM REPAIR

/dev/vg00/lvol3: UNREF DIR I=67884
OWNER=root MODE=41755
/dev/vg00/lvol3: SIZE=0 MTIME=Mar 2 10:35 2009 /dev/vg00/lvol3: UNREF DIR I=112128
OWNER=root MODE=40755
/dev/vg00/lvol3: SIZE=0 MTIME=Mar 17 05:55 2009 /dev/vg00/lvol3: UNREF DIR I=112131
OWNER=root MODE=40755
/dev/vg00/lvol3: SIZE=0 MTIME=Mar 17 05:55 2009 /dev/vg00/lvol3: UNREF DIR I=118272
OWNER=root MODE=40755
/dev/vg00/lvol3: SIZE=0 MTIME=Mar 17 05:55 2009 /dev/vg00/lvol3: UNREF DIR I=118567
OWNER=root MODE=40755
/dev/vg00/lvol3: SIZE=0 MTIME=Mar 17 05:55
....cut.....
2009 /dev/vg00/lvol3: UNREF FILE I=15617
/dev/vg00/lvol3: LINK COUNT DIR I=16896
OWNER=root MODE=40755
/dev/vg00/lvol3: SIZE=1024 MTIME=Mar 17 09:23 2009 COUNT 2 SHOULD BE 3
/dev/vg00/lvol3: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
COULD NOT FIX FILE SYSTEM WITH fsck -P, RUN fsck INTERACTIVELY!
LOGGING IN AS root FOR MANUAL fsck, ENTER ^D WHEN FILE SYSTEM FIXED
(in bcheckrc)# (in bcheckrc)#
(in bcheckrc)# fsck -y
fsck: /dev/vg00/lvol3: root file system
fsck: /dev/vg00/rlvol1: mounted file system
** /dev/vg00/rlvol4
** Last Mounted on /home
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
...cut....
** Phase 5 - Check Cyl groups
3581 files, 0 icont, 950391 used, 3986561 free (1473 frags, 498136 blocks)
(in bcheckrc)#

[Steven E. Protter · Accepted solution]

Shalom,

HP-UX systems of this era have a known issue with damaging files on the root fs when they get full. Files can get zeroed out, reduced to zero bytes.

Usually patching could deal with this.

I suggest booting the system to single user mode, and running fsck.

Boot at the console.

Intervene

bo
hpux -is
mount /usr (optional but helpful
fsck -y

SEP

[NonGuru]

Sorry, version is HP-UX 11.00

[Frank de Vries]

If you cannot fsck your root, it may be foobar.

then you need to restore it from an ignite or acronis image backup.

[Patrick Wallek]

You need to boot into LVM maintenance mode to run a full fsck on the '/' file system.

Single user mode will not work since '/' is mounted automatically.

To get to LVM Maintenance mode:

Interrupt the boot sequence at the '10 second' prompt

At the next prompt do: bo pri

When asked "Interact with ISL" say 'yes'

At the ISL prompt type: hpux -lm

Once at the # prompt, you will need to activate vg00 by doing: vgchange -a y vg00

Now you can do an fsck of your '/' volume. Do this by: fsck -y -o full /dev/vg00/rlvol3

When done, reboot the system.

[NonGuru]

OK, I did boot interacting with ISL, but still got errors. This is for a HP-ux v1 (11.00). Got complins on fsck with the -o full option, tried with just the -y (possible swap device), then with the -F vxfs said invalid superblock. Any clues?


ISL Revision A.00.38 OCT 26, 1994

ISL> hpux -lm


Boot
: disk(10/0/15/0.6.0.0.0.0.0;0)/stand/vmunix

7090176 + 872416 + 2981816 start 0x220fe8

gate64: sysvec_vaddr = 0xc0002000 for 1 pages
NOTICE: autofs_link(): File system was registered at index 4.
NOTICE: nfs3_link(): File system was registered at index 5.

System Console is on the Built-In Serial Interface
Logical volume 64, 0x3 configured as ROOT
Logical volume 64, 0x2 configured as SWAP
Logical volume 64, 0x2 configured as DUMP
Swap device table: (start & size given in 512-byte blocks)
entry 0 - major is 31, minor is 0x26003; start = 0, size = 4194304
Unable to add all swap for device: Primary Swap. Increase the tunable parameter maxswapchunks by 512 and re-configure your system.
Starting the STREAMS daemons-phase 1

Checking root file system.
/dev/rroot: UNREF DIR I=67884
OWNER=root MODE=41755
/dev/rroot: SIZE=0 MTIME=Mar 2 10:35 2009 /dev/rroot: UNREF DIR I=112128
OWNER=root MODE=40755
/dev/rroot: SIZE=0 MTIME=Mar 17 05:55 2009 /dev/rroot: UNREF DIR I=112131
OWNER=root MODE=40755
/dev/rroot: SIZE=0 MTIME=Mar 17 05:55 2009 /dev/rroot: UNREF DIR I=118272
OWNER=root MODE=40755
/dev/rroot: SIZE=0 MTIME=Mar 17 05:55 2009 /dev/rroot: UNREF DIR I=118567
OWNER=root MODE=40755
/dev/rroot: SIZE=0 MTIME=Mar 17 05:55 2009 /dev/rroot: UNREF FILE I=15617
OWNER=root MODE=100755
/dev/rroot: SIZE=0 MTIME=Feb 28 23:05 2009 (CLEARED)
/dev/rroot: UNREF FILE I=15618
OWNER=root MODE=100755
/dev/rroot: SIZE=0 MTIME=Mar 9 09:21 2009 (CLEARED)
/dev/rroot: UNREF FILE I=15621
OWNER=root MODE=100755
/dev/rroot: SIZE=0 MTIME=Mar 17 09:00 2009 (CLEARED)
/dev/rroot: UNREF FILE I=15626
OWNER=root MODE=100755
/dev/rroot: SIZE=0 MTIME=Mar 10 22:53 2009 (CLEARED)
/dev/rroot: UNREF FILE I=15628
OWNER=root MODE=100755
/dev/rroot: SIZE=0 MTIME=Mar 10 23:14 2009 (CLEARED)
/dev/rroot: UNREF FILE I=15630
OWNER=root MODE=100755
/dev/rroot: SIZE=0 MTIME=Mar 16 08:43 2009 (CLEARED)
/dev/rroot: UNREF FILE I=15634
OWNER=root MODE=100755
/dev/rroot: SIZE=0 MTIME=Mar 17 08:42 2009 (CLEARED)
/dev/rroot: UNREF FILE I=15636
OWNER=root MODE=100600
/dev/rroot: SIZE=0 MTIME=Mar 12 19:12 2009 (CLEARED)
/dev/rroot: LINK COUNT DIR I=16896
OWNER=root MODE=40755
/dev/rroot: SIZE=1024 MTIME=Mar 17 09:23 2009 COUNT 2 SHOULD BE 3
/dev/rroot: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
Root check done.
Create STCP device files

B2352B/9245XB HP-UX (B.11.00) #1: Wed Nov 5 22:38:19 PST 1997



Memory Information:

physical page size = 4096 bytes, logical page size = 4096 bytes

Physical: 2097152 Kbytes, lockable: 1556808 Kbytes, available: 1792660 Kbytes



/sbin/ioinitrc:
fsck: /dev/vg00/lvol1: possible swap device (cannot determine)
fsck SUSPENDED BY USER.
/dev/vg00/lvol1: No such device or address
Unable to mount /stand - please check entries in /etc/fstab


INIT: Overriding default level with level 's'

INIT: SINGLE USER MODE


RESTRICTED RIGHTS ....
Value of TERM has been set to "vt100".
WARNING: YOU ARE SUPERUSER !!

${HOME:-.}/.profile[56]: /opt/avaya/envdir/rootenv: not found
# vgchange -a y vg00
Activated volume group
Volume group "vg00" has been successfully changed.
# fsck -y -o full /dev/vg00/rlvol3
fsck: illegal option -- o
usage: /sbin/fsck [ -F hfs ] [ -c size ] [-f] [ -V ] [ -p | -P ] [ file system ... ]
or: /sbin/fsck [ -F hfs ] [ -m ] [ -V ] [ -b block # ] [ file system ... ]
or: /sbin/fsck [ -F hfs ] [ -V ] [ -b block # ] [ -c size ] [ -f ] [ -y ] [ -n ] [ -q ] [ file system ... ]
# fsck -y -o full /dev/vg00/rvol3
fsck: illegal option -- o
usage: /sbin/fsck [ -F hfs ] [ -c size ] [-f] [ -V ] [ -p | -P ] [ file system ... ]
or: /sbin/fsck [ -F hfs ] [ -m ] [ -V ] [ -b block # ] [ file system ... ]
or: /sbin/fsck [ -F hfs ] [ -V ] [ -b block # ] [ -c size ] [ -f ] [ -y ] [ -n ] [ -q ] [ file system ... ]
# fsck -y /dev/vg00/rlvol3
fsck: /dev/vg00/rlvol3: possible swap device (cannot determine)
# fsck -y /dev/vg00/rlvol3

fsck: /etc/default/fs is used for determining the file system type
fsck: /dev/vg00/rlvol1: possible swap device (cannot determine)


#
# fsck -y /dev/vg00/rlvol3
fsck: /dev/vg00/rlvol3: possible swap device (cannot determine)
#
# fsck -y -o full /dev/vg00/rlvol3
fsck: illegal option -- o
usage: /sbin/fsck [ -F hfs ] [ -c size ] [-f] [ -V ] [ -p | -P ] [ file system ... ]
or: /sbin/fsck [ -F hfs ] [ -m ] [ -V ] [ -b block # ] [ file system ... ]
or: /sbin/fsck [ -F hfs ] [ -V ] [ -b block # ] [ -c size ] [ -f ] [ -y ] [ -n ] [ -q ] [ file system ... ]
#
# fsck -F vxfs -y /dev/vg00/rlvol3
invalid super-block
vxfs fsck: cannot initialize aggregate
file system check failure, aborting ...
# reboot

[NonGuru]

Additional info:
# more /etc/fstab
/dev/vg00/lvol3 / hfs defaults 0 1
/dev/vg00/lvol1 /stand hfs defaults 0 1
/dev/vg00/lvol4 /home hfs defaults 0 2
/dev/vg00/lvol5 /opt/oracle hfs defaults 0 2
/dev/vg00/lvol6 /opt/avaya hfs defaults 0 2
# fsck -y -F hfs /dev/vg00/rlvol3
fsck: /dev/vg00/rlvol3: possible swap device (cannot determine)
CPU1:Admin>vgdisplay -v
--- Volume groups ---
VG Name /dev/vg00
VG Write Access read/write
VG Status available
Max LV 255
Cur LV 6
Open LV 6
Max PV 16
Cur PV 1
Act PV 1
Max PE per PV 4384
VGDA 2
PE Size (Mbytes) 32
Total PE 4374
Alloc PE 1086
Free PE 3288
Total PVG 0
Total Spare PVs 0
Total Spare PVs in use 0

--- Logical volumes ---
LV Name /dev/vg00/lvol1
LV Status available/syncd
LV Size (Mbytes) 96
Current LE 3
Allocated PE 3
Used PV 1

LV Name /dev/vg00/lvol2
LV Status available/syncd
LV Size (Mbytes) 2048
Current LE 64
Allocated PE 64
Used PV 1

LV Name /dev/vg00/lvol3
LV Status available/syncd
LV Size (Mbytes) 2048
Current LE 64
Allocated PE 64
Used PV 1

LV Name /dev/vg00/lvol4
LV Status available/syncd
LV Size (Mbytes) 32
Current LE 1
Allocated PE 1
Used PV 1

LV Name /dev/vg00/lvol5
LV Status available/syncd
LV Size (Mbytes) 25600
Current LE 800
Allocated PE 800
Used PV 1

LV Name /dev/vg00/lvol6
LV Status available/syncd
LV Size (Mbytes) 4928
Current LE 154
Allocated PE 154
Used PV 1


--- Physical volumes ---
PV Name /dev/dsk/c2t6d0
PV Status available
Total PE 4374
Free PE 3288
Autoswitch On