rhel4.3 boots too late(long time)

Maaz — Sun, 19 Jul 2009 13:01:47 GMT

OS: rhel 4 update 3
# uname -a
Linux gbmdbserver 2.6.9-34.ELsmp #1 SMP Fri Feb 24 16:56:28 EST 2006 x86_64 x86_64 x86_64 GNU/Linux

/dev/sda is local disk
/dev/sdb is FC LUN

# fdisk -l /dev/sda

Disk /dev/sda: 72.9 GB, 72999763968 bytes
255 heads, 63 sectors/track, 8875 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sda1 * 1 13 104391 83 Linux
/dev/sda2 14 2363 18876375 82 Linux swap
/dev/sda3 2364 8875 52307640 83 Linux

# fdisk -l /dev/sdb

Disk /dev/sdb: 322.1 GB, 322122547200 bytes
255 heads, 63 sectors/track, 39162 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sdb1 1 39162 314568733+ 5 Extended
/dev/sdb5 1 35855 288005224+ 83 Linux
/dev/sdb6 35856 39162 26563446 83 Linux

# df -H
Filesystem Size Used Avail Use% Mounted on
/dev/sda3 53G 18G 33G 35% /
/dev/sda1 104M 16M 83M 16% /boot
none 13G 0 13G 0% /dev/shm
/dev/sdb6 27G 11G 16G 40% /apps
/dev/sdb5 291G 142G 135G 52% /data

# cat /etc/fstab
LABEL=/ / ext3 defaults 1 1
LABEL=/boot1 /boot ext3 defaults 1 2
LABEL=SWAP-sda2 swap swap defaults 0 0
/dev/sdb6 /apps ext3 defaults 0 0
/dev/sdb5 /data ext3 defaults 0 0

I have to admin/manage this server(previously this server was administered by another person who left the company without notice and proper documentation).

I reboot the server and noticed that the system starts in very long period of time(system startup time is too lengthy).

But all the services are running fine(from the user point of view)

here are the strange outputs
# vgdisplay
Incorrect metadata area header checksum

# lvdisplay
Incorrect metadata area header checksum

# pvdisplay
Incorrect metadata area header checksum
Incorrect metadata area header checksum
Incorrect metadata area header checksum
Incorrect metadata area header checksum
--- NEW Physical volume ---
PV Name /dev/sda2
VG Name
PV Size 67.89 GB
Allocatable NO
PE Size (KByte) 0
Total PE 0
Free PE 0
Allocated PE 0
PV UUID 2vbTLr-pfzc-sywh-nMui-z2Wh-iGaW-tZ9uo5

I checked /var/log/messages and found following

Jul 14 16:10:40 gbmdbserver kernel: scsi2 : qla2xxx
Jul 14 16:10:40 gbmdbserver kernel: qla2400 0000:03:05.1:
Jul 14 16:10:40 gbmdbserver kernel: QLogic Fibre Channel HBA Driver: 8.01.07
Jul 14 16:10:40 gbmdbserver kernel: QLogic QMC2462S - IBM eServer BC 4Gb FC Expansion Card SFF
Jul 14 16:10:40 gbmdbserver kernel: ISP2422: PCI-X Mode 1 (133 MHz) @ 0000:03:05.1 hdma+, host#=2, fw=4.00.26 [IP]
Jul 14 16:10:40 gbmdbserver kernel: Vendor: IBM Model: 1815 FAStT Rev: 0914
Jul 14 16:10:40 gbmdbserver kernel: Type: Direct-Access ANSI SCSI revision: 05
Jul 14 16:10:40 gbmdbserver kernel: qla2400 0000:03:05.1: scsi(2:0:0:0): Enabled tagged queuing, queue depth 32.
Jul 14 16:10:40 gbmdbserver kernel: Vendor: IBM Model: 1815 FAStT Rev: 0914
Jul 14 16:10:40 gbmdbserver kernel: Type: Direct-Access ANSI SCSI revision: 05
Jul 14 16:10:40 gbmdbserver kernel: qla2400 0000:03:05.1: scsi(2:0:1:0): Enabled tagged queuing, queue depth 32.
Jul 14 16:10:40 gbmdbserver kernel: Vendor: IBM Model: 1815 FAStT Rev: 0914
Jul 14 16:10:40 gbmdbserver kernel: Type: Direct-Access ANSI SCSI revision: 05
Jul 14 16:10:40 gbmdbserver kernel: qla2400 0000:03:05.1: scsi(2:0:2:0): Enabled tagged queuing, queue depth 32.
Jul 14 16:10:40 gbmdbserver kernel: 736 [RAIDarray.mpp]Host 2 Target 2 Lun 0 Is a physical device but is an Unconfigured Device.
Jul 14 16:10:41 gbmdbserver kernel: Vendor: IBM Model: Universal Xport Rev: 0914
Jul 14 16:10:41 gbmdbserver kernel: Type: Direct-Access ANSI SCSI revision: 05
Jul 14 16:10:41 gbmdbserver kernel: qla2400 0000:03:05.1: scsi(2:0:2:31): Enabled tagged queuing, queue depth 32.
Jul 14 16:10:41 gbmdbserver kernel: Attached scsi generic sg2 at scsi2, channel 0, id 2, lun 31, type 0
Jul 14 16:10:41 gbmdbserver kernel: scsi3 : mpp virtual bus adaptor :version:09.01.B5.55,timestamp:Tue Mar 6 17:41:57 CST 2007
Jul 14 16:10:41 gbmdbserver kernel: 913 [RAIDarray.mpp]Waiting for 600 seconds to discover LUNS on Current Owning Path
Jul 14 16:10:41 gbmdbserver kernel: 913 [RAIDarray.mpp]Waiting for 570 seconds to discover LUNS on Current Owning Path
Jul 14 16:10:41 gbmdbserver kernel: 913 [RAIDarray.mpp]Waiting for 540 seconds to discover LUNS on Current Owning Path
Jul 14 16:10:41 gbmdbserver kernel: 913 [RAIDarray.mpp]Waiting for 510 seconds to discover LUNS on Current Owning Path
Jul 14 16:10:41 gbmdbserver kernel: 913 [RAIDarray.mpp]Waiting for 480 seconds to discover LUNS on Current Owning Path
Jul 14 16:10:41 gbmdbserver kernel: 913 [RAIDarray.mpp]Waiting for 450 seconds to discover LUNS on Current Owning Path
Jul 14 16:10:41 gbmdbserver kernel: 913 [RAIDarray.mpp]Waiting for 420 seconds to discover LUNS on Current Owning Path
Jul 14 16:10:41 gbmdbserver kernel: 913 [RAIDarray.mpp]Waiting for 390 seconds to discover LUNS on Current Owning Path
Jul 14 16:10:41 gbmdbserver kernel: 913 [RAIDarray.mpp]Waiting for 360 seconds to discover LUNS on Current Owning Path
Jul 14 16:10:41 gbmdbserver kernel: 913 [RAIDarray.mpp]Waiting for 330 seconds to discover LUNS on Current Owning Path
Jul 14 16:10:41 gbmdbserver kernel: 913 [RAIDarray.mpp]Waiting for 300 seconds to discover LUNS on Current Owning Path
Jul 14 16:10:41 gbmdbserver kernel: 913 [RAIDarray.mpp]Waiting for 270 seconds to discover LUNS on Current Owning Path
Jul 14 16:10:41 gbmdbserver kernel: 913 [RAIDarray.mpp]Waiting for 240 seconds to discover LUNS on Current Owning Path
Jul 14 16:10:41 gbmdbserver kernel: 913 [RAIDarray.mpp]Waiting for 210 seconds to discover LUNS on Current Owning Path
Jul 14 16:10:41 gbmdbserver kernel: 913 [RAIDarray.mpp]Waiting for 180 seconds to discover LUNS on Current Owning Path
Jul 14 16:10:41 gbmdbserver kernel: 913 [RAIDarray.mpp]Waiting for 150 seconds to discover LUNS on Current Owning Path
Jul 14 16:10:41 gbmdbserver kernel: 913 [RAIDarray.mpp]Waiting for 120 seconds to discover LUNS on Current Owning Path
Jul 14 16:10:41 gbmdbserver kernel: 913 [RAIDarray.mpp]Waiting for 90 seconds to discover LUNS on Current Owning Path
Jul 14 16:10:41 gbmdbserver kernel: 913 [RAIDarray.mpp]Waiting for 60 seconds to discover LUNS on Current Owning Path
Jul 14 16:10:41 gbmdbserver kernel: 913 [RAIDarray.mpp]Waiting for 30 seconds to discover LUNS on Current Owning Path

# cat /proc/scsi/scsi
Attached devices:
Host: scsi0 Channel: 00 Id: 00 Lun: 00
Vendor: LSILOGIC Model: Logical Volume Rev: 3000
Type: Direct-Access ANSI SCSI revision: 02
Host: scsi1 Channel: 00 Id: 00 Lun: 00
Vendor: IBM Model: 1815 FAStT Rev: 0914
Type: Direct-Access ANSI SCSI revision: 05
Host: scsi1 Channel: 00 Id: 01 Lun: 00
Vendor: IBM Model: 1815 FAStT Rev: 0914
Type: Direct-Access ANSI SCSI revision: 05
Host: scsi1 Channel: 00 Id: 02 Lun: 00
Vendor: IBM Model: 1815 FAStT Rev: 0914
Type: Direct-Access ANSI SCSI revision: 05
Host: scsi1 Channel: 00 Id: 02 Lun: 31
Vendor: IBM Model: Universal Xport Rev: 0914
Type: Direct-Access ANSI SCSI revision: 05
Host: scsi2 Channel: 00 Id: 00 Lun: 00
Vendor: IBM Model: 1815 FAStT Rev: 0914
Type: Direct-Access ANSI SCSI revision: 05
Host: scsi2 Channel: 00 Id: 01 Lun: 00
Vendor: IBM Model: 1815 FAStT Rev: 0914
Type: Direct-Access ANSI SCSI revision: 05
Host: scsi2 Channel: 00 Id: 02 Lun: 00
Vendor: IBM Model: 1815 FAStT Rev: 0914
Type: Direct-Access ANSI SCSI revision: 05
Host: scsi2 Channel: 00 Id: 02 Lun: 31
Vendor: IBM Model: Universal Xport Rev: 0914
Type: Direct-Access ANSI SCSI revision: 05
Host: scsi3 Channel: 00 Id: 00 Lun: 00
Vendor: IBM Model: VirtualDisk Rev: 0914
Type: Direct-Access ANSI SCSI revision: 05

please help me what should I do ? and what/where is the problem ?

Regards
Maaz

Re: rhel4.3 boots too late(long time)

Steven E. Protter — Sun, 19 Jul 2009 21:35:32 GMT

Shalom,

It appears to be slow discovery of storage.

I would take the following steps:

1) Update the OS to a more current version using RHN, on the assumption that the applicaitons support the upgrade.

2) Get updated drivers and management software from IBM

3) Physically inspect the SCSI connection and make sure if they are not self terminating, they are terminated.

SEP

Re: rhel4.3 boots too late(long time)

Maaz — Mon, 20 Jul 2009 08:12:17 GMT

Thanks SEP for quick reply

>3) Physically inspect the SCSI connection and make sure if they are not self
>terminating, they are terminated.
I am sorry I didnt got you here "if they are not self terminating, they are terminated." Please elaborate.

and what does the following mean

# vgdisplay
Incorrect metadata area header checksum

# lvdisplay
Incorrect metadata area header checksum

# pvdisplay
Incorrect metadata area header checksum
Incorrect metadata area header checksum
Incorrect metadata area header checksum
Incorrect metadata area header checksum
--- NEW Physical volume ---
PV Name /dev/sda2
VG Name
PV Size 67.89 GB
Allocatable NO
PE Size (KByte) 0
Total PE 0
Free PE 0
Allocated PE 0
PV UUID 2vbTLr-pfzc-sywh-nMui-z2Wh-iGaW-tZ9uo5

Regards

Re: rhel4.3 boots too late(long time)

Steven E. Protter — Mon, 20 Jul 2009 09:35:36 GMT

Shalom Maaz,

Self terminating scsi connections do not require a terminator device to terminate them. Some of them even have a little light that indicates termination.

An HP made Ultra 3 tape drive is an exampleof a self terminating scsi device.

I think there is a problem either with your disk array or perhaps one of the scsi cables.

SEP

Re: rhel4.3 boots too late(long time)

Maaz — Tue, 21 Jul 2009 03:56:14 GMT

thanks SEP

and what do suggest me in the following matter ?

# fdisk -l /dev/sda

Disk /dev/sda: 72.9 GB, 72999763968 bytes
255 heads, 63 sectors/track, 8875 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sda1 * 1 13 104391 83 Linux
/dev/sda2 14 2363 18876375 82 Linux swap
/dev/sda3 2364 8875 52307640 83 Linux

# vgdisplay
Incorrect metadata area header checksum

# lvdisplay
Incorrect metadata area header checksum

# pvdisplay
Incorrect metadata area header checksum
Incorrect metadata area header checksum
Incorrect metadata area header checksum
Incorrect metadata area header checksum
--- NEW Physical volume ---
PV Name /dev/sda2
VG Name
PV Size 67.89 GB
Allocatable NO
PE Size (KByte) 0
Total PE 0
Free PE 0
Allocated PE 0
PV UUID 2vbTLr-pfzc-sywh-nMui-z2Wh-iGaW-tZ9uo5

Regards
Maaz

Re: rhel4.3 boots too late(long time)

Steven E. Protter — Tue, 21 Jul 2009 11:21:58 GMT

Shalom,

/dev/sda1 * 1 13 104391 83 Linux
/dev/sda2 14 2363 18876375 82 Linux swap
/dev/sda3 2364 8875 52307640 83 Linux

Data on the storage is not LVM.

Therefore LVM commands pvdisplay lvdisplay and vgdisplay are not going to work.

If you know there is not important data on the array, you should pvcreate and build LVM structure on it.

I don't see the disk array at all right now, all I see is the local boot disk, which is also not set up LVM.

If possible, I would start all over with a fresh Linux install.

If not, its a matter of correcting the SCSI connection problem seeing the disk on an fdisk -l display and trying to see what is there.

Your slow boot is due to the inability to connect to the storage array. I would think a good first step would be to install PSP if the server is a HP branded system. Then perhaps the hpscan utilities will help you find the disk.

First step is a visual inspection IMHO. Something is not connected properly.

SEP

topic Re: rhel4.3 boots too late(long time) in Operating System - Linux

rhel4.3 boots too late(long time)

Re: rhel4.3 boots too late(long time)

Re: rhel4.3 boots too late(long time)

Re: rhel4.3 boots too late(long time)

Re: rhel4.3 boots too late(long time)

Re: rhel4.3 boots too late(long time)