Hanged on "disk quota" , while logging..

[Raj D.]

Hi All ,

I am having L2000 hp-ux 11 Server. And doubt some disk problem is there. I am trying to login , but its haning , at "disk quotas" and not giving the prompt.

login: user1
Password:
Please wait...checking for disk quotas
----------------------------------------

Can anyone help me how to bypass it. I dont want to restart now thru GSP, and want to check in syslog.log as what has happened.

Appreciate for quick reply.
Many Thanks,
Raj.
---

[Chris Wilshaw]

can you connect to the box with remsh to just run single commands?

test using

remsh your_server uptime

if this brings back a response, you can then

remsh your_server "chmod a-x /usr/bin/quota"

This will prevent the quota command from running, and with any luck, will mean that you are able to access the server - just beware of running other commands such as bdf which may also hang.

[Raj D.]

Thanks Chris ,

Really appreciate for the quick reply.
I will check that , and update soon.

Thx.
Raj.
----

[Pete Randall]

Any recent changes to .profile, .login, or any other startup script that might cause this to hang? Can you login as another user and test this user's startup scripts?


Pete

[baiju_3]

Hi ,

Please check the home file system for this user is full ?. Did you enable quota on the Home FS ?.



Thanks,
BL.

[S.Rider]

I don't think it's really hanging on disk-quotas. That's just the last message that prints out. It might be another bunch of steps down.

[OldSchool]

FWIW: I've also seen this happen when the machine as nfs-mounted drives and the datalink to the other machine goes down.

[Raj D.]

Hi All ,

I have restarted the server , and its ok now.

[ Model= 9000/800/L2000-44, OS=hp-ux B.11.00]

This is the error in syslog.log

Aug 3 04:37:48 gudvcdw1 EMS [9267]: ------ EMS Event Notification ------ Value: "CRITICAL (5)" for Resource: "/storage/events/disks/default/0_2_0_0.0.0" (Threshold: >= " 3") Execute the following command to obtain event details: /opt/resmon/bin/resdata -R 607322126 -r /storage/events/disks/default/0_2_0_0.0.0 -n 607322155 -a
Aug 3 04:37:53 gudvcdw1 vmunix:
Aug 3 04:37:53 gudvcdw1 vmunix: SCSI: Read error -- dev: b 31 0x084000, errno: 126, resid: 1024,
Aug 3 04:37:53 gudvcdw1 vmunix: blkno: 8, sectno: 16, offset: 8192, bcount: 1024.
Aug 3 04:37:53 gudvcdw1 vmunix: LVM: VG 64 0x020000: Lost quorum.
Aug 3 04:37:53 gudvcdw1 vmunix: This may block configuration changes and I/Os. In order to reestablish quorum at least 3 of the following PVs (represented by current link) must become available:
Aug 3 04:37:53 gudvcdw1 vmunix: <31 0x080000> <31 0x081000> <31 0x082000> <31 0x083000> <31 0x084000> <31 0x085000>
Aug 3 04:37:53 gudvcdw1 vmunix: LVM: VG 64 0x020000: PVLink 31 0x080000 Failed! The PV is not accessible.
Aug 3 04:37:53 gudvcdw1 vmunix: LVM: VG 64 0x020000: PVLink 31 0x081000 Failed! The PV is not accessible.
Aug 3 04:37:53 gudvcdw1 vmunix: LVM: VG 64 0x020000: PVLink 31 0x082000 Failed! The PV is not accessible.
Aug 3 04:37:53 gudvcdw1 vmunix: LVM: VG 64 0x020000: PVLink 31 0x083000 Failed! The PV is not accessible.
Aug 3 04:37:53 gudvcdw1 vmunix: LVM: VG 64 0x020000: PVLink 31 0x084000 Failed! The PV is not accessible.
Aug 3 04:37:53 gudvcdw1 vmunix: LVM: VG 64 0x020000: PVLink 31 0x085000 Failed! The PV is not accessible.

The server is connected to external DS2300 storage , with 6 disk.


ioscan -fnC disk output:
-------------------------
# ioscan -fnC disk
Class I H/W Path Driver S/W State H/W Type Description
======================================================================
disk 0 0/0/1/1.2.0 sdisk CLAIMED DEVICE SEAGATE ST336704LC
/dev/dsk/c1t2d0 /dev/rdsk/c1t2d0
disk 1 0/0/2/0.2.0 sdisk CLAIMED DEVICE SEAGATE ST336704LC
/dev/dsk/c2t2d0 /dev/rdsk/c2t2d0
disk 2 0/0/2/1.2.0 sdisk CLAIMED DEVICE HP DVD-ROM 305
/dev/dsk/c3t2d0 /dev/rdsk/c3t2d0
disk 13 0/2/0/0.0.0 sdisk CLAIMED DEVICE HP 36.4GST336607LC
/dev/dsk/c8t0d0 /dev/rdsk/c8t0d0
disk 14 0/2/0/0.1.0 sdisk CLAIMED DEVICE HP 36.4GST336607LC
/dev/dsk/c8t1d0 /dev/rdsk/c8t1d0
disk 15 0/2/0/0.2.0 sdisk CLAIMED DEVICE HP 36.4GMAP3367NC
/dev/dsk/c8t2d0 /dev/rdsk/c8t2d0
disk 16 0/2/0/0.3.0 sdisk CLAIMED DEVICE HP 36.4GST336706LC
/dev/dsk/c8t3d0 /dev/rdsk/c8t3d0
disk 17 0/2/0/0.4.0 sdisk CLAIMED DEVICE HP 36.4GST336607LC
/dev/dsk/c8t4d0 /dev/rdsk/c8t4d0
disk 18 0/2/0/0.5.0 sdisk CLAIMED DEVICE HP 36.4GST336607LC
/dev/dsk/c8t5d0 /dev/rdsk/c8t5d0
disk 3 0/4/0/0.8.0 sdisk CLAIMED DEVICE SEAGATE ST318404LC
/dev/dsk/c6t8d0 /dev/rdsk/c6t8d0
disk 4 0/4/0/0.9.0 sdisk CLAIMED DEVICE SEAGATE ST318404LC
/dev/dsk/c6t9d0 /dev/rdsk/c6t9d0
disk 5 0/4/0/0.10.0 sdisk CLAIMED DEVICE SEAGATE ST318404LC
/dev/dsk/c6t10d0 /dev/rdsk/c6t10d0
disk 6 0/4/0/0.11.0 sdisk CLAIMED DEVICE SEAGATE ST318404LC
/dev/dsk/c6t11d0 /dev/rdsk/c6t11d0
disk 7 0/4/0/0.12.0 sdisk CLAIMED DEVICE SEAGATE ST318404LC
/dev/dsk/c6t12d0 /dev/rdsk/c6t12d0
disk 8 0/7/0/0.8.0 sdisk CLAIMED DEVICE SEAGATE ST318404LC
/dev/dsk/c7t8d0 /dev/rdsk/c7t8d0
disk 9 0/7/0/0.9.0 sdisk CLAIMED DEVICE SEAGATE ST318404LC
/dev/dsk/c7t9d0 /dev/rdsk/c7t9d0
disk 10 0/7/0/0.10.0 sdisk CLAIMED DEVICE SEAGATE ST318404LC
/dev/dsk/c7t10d0 /dev/rdsk/c7t10d0
disk 11 0/7/0/0.11.0 sdisk CLAIMED DEVICE SEAGATE ST318404LC
/dev/dsk/c7t11d0 /dev/rdsk/c7t11d0
disk 12 0/7/0/0.12.0 sdisk CLAIMED DEVICE SEAGATE ST318404LC
/dev/dsk/c7t12d0 /dev/rdsk/c7t12d0
------------------------------------------

Output of /dev/dsk:
---------------------
# ls -l
total 0
brw-r----- 1 bin sys 31 0x012000 Jul 19 2002 c1t2d0
brw-r----- 1 bin sys 31 0x022000 Jul 19 2002 c2t2d0
brw-r----- 1 bin sys 31 0x032000 Jul 19 2002 c3t2d0
brw-r----- 1 bin sys 31 0x06a000 Jul 19 2002 c6t10d0
brw-r----- 1 bin sys 31 0x06b000 Jul 19 2002 c6t11d0
brw-r----- 1 bin sys 31 0x06c000 Jul 19 2002 c6t12d0
brw-r----- 1 bin sys 31 0x068000 Jul 19 2002 c6t8d0
brw-r----- 1 bin sys 31 0x069000 Jul 19 2002 c6t9d0
brw-r----- 1 bin sys 31 0x07a000 Jul 19 2002 c7t10d0
brw-r----- 1 bin sys 31 0x07b000 Jul 19 2002 c7t11d0
brw-r----- 1 bin sys 31 0x07c000 Jul 19 2002 c7t12d0
brw-r----- 1 bin sys 31 0x078000 Jul 19 2002 c7t8d0
brw-r----- 1 bin sys 31 0x079000 Jul 19 2002 c7t9d0
brw-r----- 1 bin sys 31 0x080000 Oct 12 2004 c8t0d0
brw-r----- 1 bin sys 31 0x081000 Oct 12 2004 c8t1d0
brw-r----- 1 bin sys 31 0x082000 Oct 12 2004 c8t2d0
brw-r----- 1 bin sys 31 0x083000 Oct 12 2004 c8t3d0
brw-r----- 1 bin sys 31 0x084000 Oct 12 2004 c8t4d0
brw-r----- 1 bin sys 31 0x085000 Oct 12 2004 c8t5d0
# pwd
/dev/dsk
--------------------------

/var/opt/resmon/log/event.log output (tail -n 80)

Notification Time: Wed Aug 3 04:38:10 2005

gudvcdw1 sent Event Monitor notification information:

/storage/events/disks/default/0_2_0_0.5.0 is >= 1.
Its current value is CRITICAL(5).



Event data from monitor:

Event Time..........: Wed Aug 3 04:38:10 2005
Severity............: CRITICAL
Monitor.............: disk_em
Event #.............: 3
System..............: gudvcdw1

Summary:
Disk at hardware path 0/2/0/0.5.0 : Drive is not responding.


Description of Error:

As part of the polling functionality, the monitor periodically requests
data from the device. The monitor's request of Test Unit Ready command
failed.

Probable Cause / Recommended Action:

The I/O request that the monitor made to this device failed because the
device timed-out. Check cables, power supply, ensure the drive is powered
ON, and if needed replace the drive.

Additional Event Data:
System IP Address...: 10.23.220.124
Event Id............: 0x42f0827200000002
Monitor Version.....: B.01.00
Event Class.........: I/O
Client Configuration File...........:
/var/stm/config/tools/monitor/default_disk_em.clcfg
Client Configuration File Version...: A.01.00
Qualification criteria met.
Number of events..: 1
Associated OS error log entry id(s):
None
Additional System Data:
System Model Number.............: 9000/800/L2000-44
OS Version......................: B.11.00
STM Version.....................: A.25.00
EMS Version.....................: A.03.20
Latest information on this event:
http://docs.hp.com/hpux/content/hardware/ems/disk_em.htm#3

v-v-v-v-v-v-v-v-v-v-v-v-v D E T A I L S v-v-v-v-v-v-v-v-v-v-v-v-v



Component Data:
Physical Device Path...: 0/2/0/0.5.0
Device Class...........: Disk
Inquiry Vendor ID......: HP 36.4G
Inquiry Product ID.....: ST336607LC
Firmware Version.......: HPC5
Serial Number..........: 3JA7K77200007440DTSV

Product/Device Identification Information:

Logger ID.........: disc30; sdisk
Product Identifier: Disk
Product Qualifier.: HP 36.4GST336607LC
SCSI Target ID....: 0x05
SCSI LUN..........: 0x00

Hardware Status: (not present in log record).

SCSI Sense Data: (not present in log record)


>---------- End Event Monitoring Service Event Notification ----------<
#
-------------------------------------

Any Idea , whats wrong.

Many Thanks,
Raj.
---

[Raj D.]

Also attached the cstm output for disk5 i.e 0/2/0/0.5.0 that says "PV not responding" in syslog :

cstm>infolog
Product Id: ST336607LC Vendor: HP 36.4G
Device Type: SCSI Disk Firmware Rev: HPC5 2/0/0.5.0 File ...
Device Qualifier: HP36.4GST336607LC Logical Unit: 0
Serial Number: 3JA7K77200007440DTSV
Capacity (M Byte): 34732.89
Block Size: 512
Max Block Address: 71132959
Error Logs
Read Errors: 0 Buffer Overruns: N/A
Read Reverse Errors: N/A Buffer Underruns: N/A
Write Errors: 0 Non-Medium Errors: 1561
Verify Errors: 0

------------------------

Just wondering whats wrong. This error comes often in the morning.So far happened (7 to 8 times) And can be found on syslog.log "SCSI Reset lbolt " etc. And I cheked the disks , there is no problem with disk, no stale PE , and at that time if we execute "ioscan -fnC disk" , the 6 disk (at 0/2/0/0 , shows "UNCLAIMED" then "SCAN" then "CLAIMED" and comes ok withing 2-3 minutes. Sometime it never comes, like today and need a reboot.

Any idea.
Many Thanks,

Raj.
------

[Eknath · Accepted solution]

Hi Raj,

It looks like the disk dev/dsk/c8t5d0 is going offline. To check run #diskinfo -v /dev/rdsk/c8t5d0, #pvdisplay -v /dev/dsk/c8t5d0

I think this is alarm that this disk is at end of its life. I would suggest you to take backup and replace the disk to avoid data loss in future...

Cheers!!!
Eknath

[Devender Khatana]

Hi,

The same symptoms would appear if your disk is powered off and then back on. Note that it has reported multiple PVs inaccessible in the syslog.log. Are these different paths to the same disk.

Does this error always come for only one disk ?
If yes replace it immediately.

If it comes for different disk everytime then see your power redundancy to the DS2300.

HTH,
Devender