The "bad" behaviors of the sysinit action scripts execution within the /etc/inittab！

[liuyl_it]

Recently we found that sometimes the sysinit action scripts execution might take a longer time so that the upper dependent scripts started before they ended！

For example, the following ahead ioinitrc script has not finished with the IOtree, and meanwhile the latter vxvm scripts have started.....！

ioin::sysinit:/sbin/ioinitrc >/dev/console 2>&1

...........

vol1::sysinit:/sbin/vxvm-sysboot </dev/console >/dev/console 2>&1 ##vxvm

vol2::sysinit:/sbin/init.d/vxvm-startup start </dev/console >/dev/console 2>&1 ##vxvm

So my question is why this situation would randomly happen with the same H/W & S/W envrionment ？

[Murali-Iyer]

The ioinit command is invoked by the init process when the system is booted, based on the ioin entry in /etc/inittab:

ioin::sysinit:/sbin/ioinitrc > /dev/console 2>&1

where ioinitrc is a script to invoke ioinit with the -i and -r options.

Given the -i option, ioinit checks consistency between the kernel I/O data structures and information read from /etc/ioconfig and /etc/ext_ioconfig. If these are consistent, ioinit invokes insf to install special files for all new devices.

If the kernel is inconsistent with /etc/ioconfig or /etc/ext_ioconfig, ioinit updates /stand/ioconfig from /etc/ioconfig, and /stand/ext_ioconfig from /etc/ext_ioconfig, and, if the -r option is given, reboots the system. 

So, We need to see where the ioinitrc script is stuck using Console Messages etc to take further action 

[liuyl_it]

1) There is no any particular symptom or error/failing during its booting phase in the console log！
    And also there is no such similar information as the following normal message that should be: 
    NOTICE: VxVM vxdmp V-5-0-34 added disk array 09473, datype = TagmaStore-USP

ktracer is off until requested.
spinlock ALAT switching supported on this system
Installing Socket Protocol families AF_INET and AF_INET6
Kernel EVM initialized
sec_init(): kernel RPC authentication/security initialization.
secgss_init():  kernel RPCSEC_GSS security initialization.
rpc_init(): kernel RPC initialization.
rpcmod_install(): kernel RPC STREAMS module "rpcmod" installation. ...(driver_install)
NOTICE: nfs_client_pv3_install(): nfs3 File system was registered at index 10.
NOTICE: nfs_client_pv4_install(): nfs4 File system was registered at index 11.
NOTICE: cachefsc_install: cachefs File system was registered at index 13.
LLT INFO V-14-1-10009 LLT 6.0.500.000 Protocol available
GAB INFO V-15-1-20021 GAB 6.0.500.000 available
VXFEN INFO V-11-1-56 VxFEN 6.0.500.000 loaded. Protocol versions supported: 10,20,30
vxfen: vxfen protocol available
fclp0: Claiming Fibre Channel HBA port at hardware path 0/0/12/0/0/0/0
fclp1: Claiming Fibre Channel HBA port at hardware path 0/0/14/0/0/0/0
fclp2: Claiming Fibre Channel HBA port at hardware path 1/0/12/0/0/0/0
Boot device's HP-UX HW path is: 0/0/0/2/0.0x6.0x0
igelan0: INITIALIZING HP PCI-X 1000Base-T Built-in at hardware path 0/0/0/1/0

    System Console is on the Built-In Serial Interface
iether7: INITIALIZING HP A7011-60001 PCI/PCI-X 1000Base-SX Dual-port Adapter at hardware path 1/0/10/1/0
iether8: INITIALIZING HP A7011-60001 PCI/PCI-X 1000Base-SX Dual-port Adapter at hardware path 1/0/10/1/1
iether1: INITIALIZING HP A7011-60001 PCI/PCI-X 1000Base-SX Dual-port Adapter at hardware path 0/0/2/1/0
iether2: INITIALIZING HP A7011-60001 PCI/PCI-X 1000Base-SX Dual-port Adapter at hardware path 0/0/2/1/1
iether5: INITIALIZING HP A7011-60001 PCI/PCI-X 1000Base-SX Dual-port Adapter at hardware path 1/0/8/1/0
iether6: INITIALIZING HP A7011-60001 PCI/PCI-X 1000Base-SX Dual-port Adapter at hardware path 1/0/8/1/1
iether3: INITIALIZING HP A7011-60001 PCI/PCI-X 1000Base-SX Dual-port Adapter at hardware path 0/0/10/1/0
iether4: INITIALIZING HP A7011-60001 PCI/PCI-X 1000Base-SX Dual-port Adapter at hardware path 0/0/10/1/1
AF_INET socket/streams output daemon running, pid 114
afinet_prelink: module installed
Starting the STREAMS daemons-phase 1 
LVM: Root VG activated
 Swap device table:  (start & size given in 512-byte blocks)
        entry 0 - major is 64, minor is 0xe; start = 0, size = 50331648
Checking root file system.
file system is clean - log replay is not required
Root check done.
Create STCP device files
Starting the STREAMS daemons-phase 2 
     $Revision: vmunix:    B.11.31_LR FLAVOR=perf 
Memory Information:
    physical page size = 4096 bytes, logical page size = 4096 bytes
    Physical: 50195380 Kbytes, lockable: 37642744 Kbytes, available: 42177792 Kbytes

/sbin/ioinitrc:
/dev/vg00/lvol1:file system is clean - log replay is not required
/sbin/krs_sysinit:
       * The module 'rng' has been loaded.
       * The module 'pciinfo' has been loaded.
AMF Driver version 6.0.500.000 loaded
       * The module 'amf' has been loaded.

insf: Installing special files for ipmi instance 0 address 0/250/1

NOTICE: VxVM vxdmp V-5-0-34 [Info] added disk array DISKS, datype = Disk

NOTICE: VxVM vxdmp V-5-0-0 [Info] removed disk array FAKE_ENCLR_SNO, datype = FAKE_ARRAY

/sbin/bcheckrc:
Checking for LVM volume groups and Activating (if any exist)
Volume group "/dev/vg00" is already active on this system.
Resynchronized volume group /dev/vg00
Checking vxfs file systems
UX:vxfs fsck: INFO: V-3-20913: /dev/vg00/lvol3:sanity check: root file system OK (mounted read/write)
UX:vxfs fsck: INFO: V-3-20915: /dev/vg00/lvol4:sanity check: /dev/vg00/lvol4 OK
UX:vxfs fsck: INFO: V-3-20915: /dev/vg00/lvol5:sanity check: /dev/vg00/lvol5 OK
UX:vxfs fsck: INFO: V-3-20915: /dev/vg00/lvol6:sanity check: /dev/vg00/lvol6 OK
UX:vxfs fsck: INFO: V-3-20915: /dev/vg00/lvol7:sanity check: /dev/vg00/lvol7 OK
UX:vxfs fsck: INFO: V-3-20915: /dev/vg00/lvol8:sanity check: /dev/vg00/lvol8 OK
UX:vxfs fsck: INFO: V-3-20915: /dev/vg00/lvinstall:sanity check: /dev/vg00/lvinstall OK
UX:vxfs fsck: INFO: V-3-20915: /dev/vg00/lvopenv:sanity check: /dev/vg00/lvopenv OK
UX:vxfs fsck: INFO: V-3-20915: /dev/vg00/lvapp:sanity check: /dev/vg00/lvapp OK
UX:vxfs fsck: INFO: V-3-20915: /dev/vg00/lvapm:sanity check: /dev/vg00/lvapm OK
UX:vxfs fsck: INFO: V-3-20915: /dev/vg00/lvbackup:sanity check: /dev/vg00/lvbackup OK

Cleaning /etc/ptmp...

/sbin/auto_parms: DHCP access is disabled (see /etc/auto_parms.log)

     
     HP-UX Start-up in progress
     __________________________

     Configure system crash dumps ...................................... OK
     Start Integrity VM Guest Tools .................................... OK
     Removing old vxvm files ........................................... OK
     VxVM INFO V-5-2-3360 VxVM device node check ....................... OK
     VxVM INFO V-5-2-3362 VxVM general startup ......................... OK
     VxVM INFO V-5-2-3366 VxVM reconfiguration recovery ................ OK
     Mount file systems ................................................ OK
     Remounting Root File System ....................................... OK
     Setting hostname .................................................. OK

2) The problem key I think is that the sysinit script should be finished before its next sysinit continues！

sysinit:  Entries of this type are executed before boot init tries to access the console. It is expected that this entry will
be only used to initialize devices on which boot init might attempt to obtain run level information. These entries
are executed and waited for before continuing.

[liuyl_it]

Now this problem has gotten some important and significant progress, that is,
There was no any FC LUNs found after the execution with the /sbin/ioinitrc except for its own local disks！
Notes:  the following console boot log output via adding a custom ioscan inittab entry behind the ioinitrc！

/sbin/ioinitrc:
/dev/vg00/lvol1:file system is clean - log replay is not required
/sbin/krs_sysinit:
* The module 'rng' has been loaded.
* The module 'pciinfo' has been loaded.
AMF Driver version 6.0.500.000 loaded
* The module 'amf' has been loaded.

insf: Installing special files for ipmi instance 0 address 0/250/1

This is the custom ioscan script started at Mon Jun 24 14:50:23 EAT 2019 !
Class I H/W Path Driver S/W State H/W Type Description
===================================================================
disk 1 64000/0xfa00/0x1 esdisk CLAIMED DEVICE Optiarc DVD RW AD-5200A
disk 2 64000/0xfa00/0x2 esdisk CLAIMED DEVICE HP 300 GHUS153030VL3800
disk 0 64000/0xfa00/0x1da esdisk CLAIMED DEVICE HP 300 GST3300655LC
This is the custom ioscan script finished at Mon Jun 24 14:50:24 EAT 2019 !
NOTICE: VxVM vxdmp V-5-0-34 [Info] added disk array DISKS, datype = Disk

NOTICE: VxVM vxdmp V-5-0-0 [Info] removed disk array FAKE_ENCLR_SNO, datype = FAKE_ARRAY