topic Re: System slow down after lvremove in Operating System - HP-UX

System slow down after lvremove

GraceO — Wed, 21 Nov 2007 16:40:26 GMT

I removed some logical volumes via sam that were no longer needed to free up space. However, before I could even reallocate the space somewhere else, users started to report that the jobs they were submitting was taking a lot longer. One example is a 8 second job taking 14 minutes. The logical volumes I removed have nothing to do with the application they are running and no errors are reported, it's just taking longer. Can anyone help me troubleshoot this?

We are running 11.0 on an rp5470. The application uses an Oracle 9i database. After hours yesterday, we brought down the databases and rebooted the server to see if it would help...it didn't.

Any guidance would be greatly, greatly appreciated!

Re: System slow down after lvremove

Murat SULUHAN — Wed, 21 Nov 2007 16:49:50 GMT

Hi

Did you check Oracle's cpu usage and load average with top. You can also check Oracle's alert.log file to catch Oracle related problems.

Did you check /var/adm/syslog/syslog.log, /var/opt/resmon/log/event.log files and ioscan -fn output.

If you find strange issues in above items you can submit here

Best Regards
Murat

Re: System slow down after lvremove

GraceO — Wed, 21 Nov 2007 17:04:59 GMT

Our Oracle DBA says it's not an Oracle problem.

syslog.log and ioscan look fine to me.

The event.log, however, shows me a software configuration error, see excerpt below:

Software configuration error

Description of Error:

The device is in a condition where it requires action on the part of the
device driver or a human operator.

Probable Cause / Recommended Action:

The device has been reset by a Bus Device Reset message, a hard reset
condition, or a power-on reset.

If this is the case, no action is necessary.

Alternatively, a removable medium has been loaded or replaced.

If this is the case, no action is necessary.

Alternatively, the mode parameters, microcode, or inquiry data for the
device have been changed.

If this is the case, no action is necessary.

Alternatively, the installed version of the device driver does not match
that of the installed version of HP-UX. Install the correct version of the
driver.

This error appeared after I rebooted the server yesterday. The disk listed is one that has swap space on it. I used stm to get information on the disk, and no errors were reported so I thought it must be fine.

Re: System slow down after lvremove

Murat SULUHAN — Wed, 21 Nov 2007 17:18:47 GMT

Hi GraceO

Don't you have any extra information about which device generated errors in the event log.

Which disks had removed logical volumes. SAN or local ?

I think you should extra info about LV removing
operation

Can you verify your system with stm

Best Regards
Murat

Re: System slow down after lvremove

GraceO — Wed, 21 Nov 2007 17:48:52 GMT

Here is the EMS notification:

>------------ Event Monitoring Service Event Notification ------------<

Notification Time: Tue Nov 20 19:24:03 2007

xxxxxx sent Event Monitor notification information:

/storage/events/disks/default/0_9_0_0.11.0 is >= 3.
Its current value is MAJORWARNING(3).

Event data from monitor:

Event Time..........: Tue Nov 20 19:24:03 2007
Severity............: MAJORWARNING
Monitor.............: disk_em
Event #.............: 100091
System..............: xxxxxx

Summary:
Disk at hardware path 0/9/0/0.11.0 : Software configuration error

Description of Error:

The device is in a condition where it requires action on the part of the
device driver or a human operator.

Probable Cause / Recommended Action:

The device has been reset by a Bus Device Reset message, a hard reset
condition, or a power-on reset.

If this is the case, no action is necessary.

Alternatively, a removable medium has been loaded or replaced.

If this is the case, no action is necessary.

Alternatively, the mode parameters, microcode, or inquiry data for the
device have been changed.

If this is the case, no action is necessary.

Alternatively, the installed version of the device driver does not match
that of the installed version of HP-UX. Install the correct version of t
he
driver.

Additional Event Data:
System IP Address...: xxx.xxx.xxx.xxx
Event Id............: 0x4743c0f300000000
Monitor Version.....: B.01.00
Event Class.........: I/O
Client Configuration File...........:
/var/stm/config/tools/monitor/default_disk_em.clcfg
Client Configuration File Version...: A.01.00
Qualification criteria met.
Number of events..: 1
Associated OS error log entry id(s):
None
Additional System Data:
System Model Number.............: 9000/800/L3000-5x
OS Version......................: B.11.11
STM Version.....................: A.28.00
EMS Version.....................: A.03.20
Latest information on this event:
http://docs.hp.com/hpux/content/hardware/ems/scsi.htm#100091

v-v-v-v-v-v-v-v-v-v-v-v-v D E T A I L S v-v-v-v-v-v-v-v-v-v-v-v-v

Component Data:
Physical Device Path...: 0/9/0/0.11.0
Device Class...........: Disk
Inquiry Vendor ID......: HP 36.4G
Inquiry Product ID.....: MAM3367MC
Firmware Version.......: HP00
Serial Number..........: KP006472

Product/Device Identification Information:

Logger ID.........: disc30; sdisk
Product Identifier: Disk
Product Qualifier.: HP 36.4GMAM3367MC
SCSI Target ID....: 0x0B
SCSI LUN..........: 0x00

SCSI Command Data Block:

Command Data Block Contents:
0x0000: 00 00 00 00 00 00

Command Data Block Fields (6-byte fmt):
Command Operation Code...(0x00)..: TEST UNIT READY
Logical Unit Number..............: 0

Hardware Status: (not present in log record).

SCSI Sense Data:

Undecoded Sense Data:
0x0000: 70 00 06 00 00 00 00 28 00 00 00 00 5D 43 00 00
0x0010: 00 00 0B 00 00 68 00 01 00 00 00 00 00 00 00 00
0x0020: 00 00 00 23 1C 00 00 00 00 00 00 00 00 00 00 00

SCSI Sense Data Fields:
Error Code : 0x70
Segment Number : 0x00
Bit Fields:
Filemark : 0
End-of-Medium : 0
Incorrect Length Indicator : 0
Sense Key : 0x06
Information Field Valid : FALSE
Information Field : 0x00000000
Additional Sense Length : 40
Command Specific : 0x00000000
Additional Sense Code : 0x5D
Additional Sense Qualifier : 0x43
Field Replaceable Unit : 0x00
Sense Key Specific Data Valid : FALSE
Sense Key Specific Data : 0x00 0x00 0x00

Sense Key 0x06, UNIT ATTENTION, indicates that the target has been
reset by a BUS DEVICE RESET message, a hard reset condition, or by
a
power-on reset. If not a reset, then one of the following may have
occurred.
1. A removable medium may have been changed.
2. The mode parameters in effect for this initiator have been
changed by another initiator.
3. The version or level of microcode has been changed.
4. Tagged commands queued for this initiator were cleared by
another initiator.
5. INQUIRY data has been changed.
6. The mode parameters in effect for this initiator have been
restored from non-volatile memory.
7. A change in the condition of a synchronized spindle.
8. Any other event that requires the attention of the initiator.

>---------- End Event Monitoring Service Event Notification ----------<

The is a disk in one of two disk enclosures, so I guess that would be local as opposed to SAN.

The disks from which I removed the logical volumes are physically different disks. The process was as follows:
1. Oracle DBA removed files from file system
2. I use sam to remove the file system (List...File systems, select file system, then Action...Remove)
3. I use sam to remove logical volume (List...Logical volumes, select logical volume, then Action...Remove)

Using stm, I get a 'verify successful' on the disk with the EMS error, but it won't run for 'system'.

Re: System slow down after lvremove

whiteknight — Wed, 21 Nov 2007 22:37:05 GMT

Grace,

things to check
1)/var/adm/syslog/syslog.log ?
2) swapinfo -tam
3) dmesg got any error ?

WK

Re: System slow down after lvremove

GraceO — Fri, 23 Nov 2007 14:54:51 GMT

syslog.log shows nothing unusual except for the EMS warning (in previous post). swapinfo looks like this:
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 4096 64 4032 2% 0 - 1 /dev/vg00/lvol2
dev 6144 66 6078 1% 0 - 1 /dev/disk0900110/SWAP2
reserve - 3316 -3316
memory 6312 4262 2050 68%
total 16552 7708 8844 47% - 0 -

No messages are in dmesg.

I'm wondering if something happened to swap?

Re: System slow down after lvremove

Murat SULUHAN — Fri, 23 Nov 2007 16:26:46 GMT

Hi GraceO

Did you run diskinfo -v on that disk?

Could you run cstm's verify or exercise tool on that disk and scsi device that connect to same disk?

Best Regards
Murat

Re: System slow down after lvremove

GraceO — Fri, 23 Nov 2007 17:00:17 GMT

The diskinfo is as follows:

SCSI describe of /dev/rdsk/c6t11d0:
vendor: HP 36.4G
product id: MAM3367MC
type: direct access
size: 35566480 Kbytes
bytes per sector: 512
rev level: HP00
blocks per disk: 71132960
ISO version: 0
ECMA version: 0
ANSI version: 2
removable media: no
response format: 2
(Additional inquiry bytes: (32)50 (33)30 (34)30 (35)36 (36)34 (37)37 (38)32 (39)20 (40)20 (41)20
(42)20 (43)46 (44)55 (45)4a (46)49 (47)54 (48)53 (49)55 (50)20 (51)0 (52)0 (53)0 (54)40 (55)b (56)18
(57)1 (58)3c (59)1 (60)9b (61)0 (62)0 (63)0 (64)0 (65)0 (66)0 (67)0 (68)0 (69)0 (70)0 (71)0 (72)0 (73
)0 (74)0 (75)0 (76)0 (77)0 (78)0 (79)0 (80)0 (81)0 (82)0 (83)0 (84)0 (85)0 (86)0 (87)0 (88)0 (89)0 (90
)0 )

The verify/exercise was successful on the disk and interface. I'm not sure what the 'Additional inquire bytes' in the output above are...good news or bad.

Re: System slow down after lvremove

Murat SULUHAN — Fri, 23 Nov 2007 17:13:32 GMT

Hi Grace

event.log error's explanation like that
Event 100091

* Code:0xXX Qualifier:0xXX Key:0x06 Status:0xXX
* Device: TAPE
* Severity: INFORMATION

* Code:0xXX Qualifier:0xXX Key:0x06 Status:0xXX
* Device: ALL
* Severity: MAJOR_WARNING

* Event Summary: Software configuration error
* Problem Description: The device is in a condition where it requires action on the part of the device driver or a human operator.
* Probable Cause / Recommended Action: The device has been reset by a Bus Device Reset message, a hard reset condition, or a power-on reset. If this is the case, no action is necessary.
Alternatively, a removable medium has been loaded or replaced. If this is the case, no action is necessary.
Alternatively, the mode parameters, microcode, or inquiry data for the device have been changed. If this is the case, no action is necessary.
Alternatively, the installed version of the device driver does not match that of the installed version of HP-UX. Install the correct version of the driver.

Did you check Online Diagnostics daemon's CPU usage?

Your STM release is very old, why don't you upgrade from following url
http://h20392.www2.hp.com/portal/swdepot/displayProductInfo.do?productNumber=B6191AAE

Best Regards
Murat

Re: System slow down after lvremove

GraceO — Fri, 23 Nov 2007 17:38:57 GMT

Murat,

Thanks for your continued support! As far as the EMS reasons/probable causes, none seem to fit my situation. This is an 'older' server that we are slowly retiring, so not much is being done on it as far as upgrades/patches are concerned. We're just try to keep it going for a little longer.

I'm not familiar with the Online Diagnostics daemon. (I'm new to HP-UX.) Can you point me to some documentation on how to use/run it?

Thanks again,
Grace

Re: System slow down after lvremove

GraceO — Fri, 23 Nov 2007 17:54:36 GMT

Is the Online Diagnostics another name for EMS?