LVM 'device busy' error all the time

[The Gunners]

Hi Guys / Gals ,
Firstly , Im fairly new to HP/UX from a build perspective.
Im hoping you can help me. I have v strange issue. Was setting up a new RP4410 with HP/UX 11i. All was fine until I patched it for Oracle Installs. Now I cant build Volume Groups or anything in LVM. It just keeps saying stuff like 'device busy' and hangs for ages. I've even wiped the system at this stage to start from scratch and still cannot get this to work. Any ideas on such an issue. Its holding me up on a big project , Im really stuck and would appreciate any ideas or suggesstions. I reckon I may have 'overwritten' something when I was patching , could that be possible ?

[Sivakumar MJ._1]

Hi..

Pls. go through our forum posting related to your issue...

http://forums11.itrc.hp.com/service/forums/questionanswer.do?threadId=1152812

[Duncan Edmonstone]

Ok, so what are the disks you are trying to run this command on? Local or SAN? ANd what type? HP EVA, EMC DMX? Something else?

Can you actually show the set of commands you are running and the output...

HTH

Duncan

[Roopesh Francis_1]

Davey,

can you please post the output of the command which you excuted.
Thanks

[The Gunners]

Hi Everyone , thanks for replying to me already on this. Well basically I've tried using the sam menu / creating volume groups and thats where its stuck , see below disks on the system. If I try create a volume group on either of the 'LOGICAL' external disks below it says device busy on both , I've tried everything , vgscans , etc , even thought it might be a hardware problem and tried diff disks etc and no joy. Im mainly an Oracle DBA , if there is anything else I could try to check that may help you help me , let me know what to run and I paste output etc






Group Mbytes Description â
â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â
â â 0/0/3/0.0.0.0 1 Unused -- 0 DVD-ROM Drive ( ^ â
â â 0/1/1/0.0.0 1 LVM -- 0 HP 73.4GST37345 â
â â 0/1/1/0.1.0 1 LVM vg00 70007 HP 73.4GST37345 â
â â 0/6/1/0/4/0.0.0 1 Unused -- 138920 HP LOGICAL â
â â 0/6/1/0/4/0.0.1 1 Unused -- 138920 HP LOGICAL â

[Duncan Edmonstone]

OK a usefull starting point would be to post the output of the command:

ioscan -kfn

HTH

Duncan

[The Gunners]

Hiya doin Duncan , ok here we go

dubhptst: # ioscan -kfn
Class I H/W Path Driver S/W State H/W Type Description
=============================================================================
root 0 root CLAIMED BUS_NEXUS
ioa 0 0 sba CLAIMED BUS_NEXUS System Bus Adap
ter (880)
ba 0 0/0 lba CLAIMED BUS_NEXUS Local PCI-X Bus
Adapter (783)
tty 0 0/0/1/0 asio0 CLAIMED INTERFACE PCI SimpleComm
(103c1290)
/dev/diag/mux0 /dev/mux0 /dev/tty0p0
tty 1 0/0/1/1 asio0 CLAIMED INTERFACE PCI Serial (103
c1048)
/dev/GSPdiag1 /dev/mux1 /dev/tty1p2
/dev/diag/mux1 /dev/tty1p0 /dev/tty1p4
usb 0 0/0/2/0 hcd CLAIMED INTERFACE NEC USB Interfa
ce
usbhub 0 0/0/2/0.1 hub CLAIMED DEVICE USB Root Hub
usb 1 0/0/2/1 hcd CLAIMED INTERFACE NEC USB Interfa
ce
usbhub 1 0/0/2/1.1 hub CLAIMED DEVICE USB Root Hub
/dev/hub
usb 2 0/0/2/2 ehci CLAIMED INTERFACE NEC USB Interfa
ce
sideba 0 0/0/3/0 side_ba CLAIMED INTERFACE CMD IDE control
ler
ext_bus 0 0/0/3/0.0 side CLAIMED INTERFACE IDE Primary Cha
nnel
target 0 0/0/3/0.0.0 tgt CLAIMED DEVICE
disk 0 0/0/3/0.0.0.0 sdisk CLAIMED DEVICE TEAC DV-28E-
N
/dev/dsk/c0t0d0 /dev/rdsk/c0t0d0
target 1 0/0/3/0.0.7 tgt CLAIMED DEVICE
ctl 0 0/0/3/0.0.7.0 sctl CLAIMED DEVICE Initiator
/dev/rscsi/c0t7d0
ext_bus 1 0/0/3/0.1 side CLAIMED INTERFACE IDE Secondary C
hannel
target 2 0/0/3/0.1.7 tgt CLAIMED DEVICE
ctl 1 0/0/3/0.1.7.0 sctl CLAIMED DEVICE Initiator
/dev/rscsi/c1t7d0
ba 1 0/1 lba CLAIMED BUS_NEXUS Local PCI-X Bus
Adapter (783)
ext_bus 2 0/1/1/0 mpt CLAIMED INTERFACE SCSI Ultra320 A
6961-60011
/dev/mpt2
target 3 0/1/1/0.0 tgt CLAIMED DEVICE
disk 1 0/1/1/0.0.0 sdisk CLAIMED DEVICE HP 73.4GST37345
4LC
/dev/dsk/c2t0d0 /dev/rdsk/c2t0d0
target 4 0/1/1/0.1 tgt CLAIMED DEVICE
disk 2 0/1/1/0.1.0 sdisk CLAIMED DEVICE HP 73.4GST37345
4LC
/dev/dsk/c2t1d0 /dev/rdsk/c2t1d0
target 5 0/1/1/0.7 tgt CLAIMED DEVICE
ctl 2 0/1/1/0.7.0 sctl CLAIMED DEVICE Initiator
/dev/rscsi/c2t7d0
ext_bus 3 0/1/1/1 mpt CLAIMED INTERFACE SCSI Ultra320 A
6961-60011
/dev/mpt3
target 6 0/1/1/1.7 tgt CLAIMED DEVICE
ctl 3 0/1/1/1.7.0 sctl CLAIMED DEVICE Initiator
/dev/rscsi/c3t7d0
lan 0 0/1/2/0 iether CLAIMED INTERFACE HP AB352-60003
PCI/PCI-X 1000Base-T Dual-port Core
lan 1 0/1/2/1 iether CLAIMED INTERFACE HP AB352-60003
PCI/PCI-X 1000Base-T Dual-port Core
ba 2 0/2 lba CLAIMED BUS_NEXUS Local PCI-X Bus
Adapter (783)
ba 3 0/4 lba CLAIMED BUS_NEXUS Local PCI-X Bus
Adapter (783)
%U username given at login time
%M maximum allowed number of users in this class
%N current number of users in this class

The message will only be displayed once to avoid annoying the
user. Remember that when messages are triggered by an anonymous
ba 4 0/5 lba CLAIMED BUS_NEXUS Local PCI-X Bus
Adapter (783)
ba 5 0/6 lba CLAIMED BUS_NEXUS Local PCI-X Bus
Adapter (783)
ba 6 0/6/1/0 PCItoPCI CLAIMED BUS_NEXUS PCItoPCI Bridge
ext_bus 4 0/6/1/0/4/0 ciss CLAIMED INTERFACE PCI-X SmartArra
y 6402 RAID Controller
/dev/ciss4
target 7 0/6/1/0/4/0.0 tgt CLAIMED DEVICE
disk 3 0/6/1/0/4/0.0.0 sdisk CLAIMED DEVICE HP LOGIC
AL VOLUME
/dev/dsk/c4t0d0 /dev/rdsk/c4t0d0
disk 4 0/6/1/0/4/0.0.1 sdisk CLAIMED DEVICE HP LOGIC
AL VOLUME
/dev/dsk/c4t0d1 /dev/rdsk/c4t0d1
target 8 0/6/1/0/4/0.7 tgt CLAIMED DEVICE
ctl 4 0/6/1/0/4/0.7.0 sctl CLAIMED DEVICE Initiator
/dev/rscsi/c4t7d0
memory 0 8 memory CLAIMED MEMORY Memory
ipmi 0 16 ipmi CLAIMED INTERFACE IPMI Controller
/dev/ipmi
processor 0 128 processor CLAIMED PROCESSOR Processor
processor 1 129 processor CLAIMED PROCESSOR Processor

[Mel Burslan]

Other useful commands to see output from, are:

strings /etc/lvmtab
vgdisplay
swlist -l fileset -a state

they have rather long outputs, so if you can capture your screen to log file and attach that, it would be the easiest.

[The Gunners]

Okay , thats no problem , Wow yes theres a good bit of info. Find attached doc with output

[Duncan Edmonstone]

So the disk devices you are trying to use seem to be RAID volumes configured through a SmartARRAY 6402 card. To see the current status of those 2 RAID volumes, what output do you get from:

saconfig /dev/ciss4

Let's also see if the disks are responding at all...


diskinfo /dev/rdsk/c4t0d0
diskinfo /dev/rdsk/c4t0d1

and check there are no LVM headers already on these disks:

pvdisplay /dev/dsk/c4t0d0
pvdisplay /dev/dsk/c4t0d1

HTH

Duncan

[The Gunners]

Hi again Duncan , some interesting output from your last requests there . Some errors for sure. See attached

[Duncan Edmonstone]

well something is wrong with those 2 drives:

Let's see the output of:

sautil /dev/ciss4

HTH

Duncan

[TTr]

It appears that the special file /dev/rdsk/c4t0d0 may have been overwritten with a regular file. Post back the output of
ll /dev/rdsk/c4t0d0
ll /dev/dsk/c4t0d0

[The Gunners]

Hi Again Duncan , ok ,here we go , see attached output as per requested command. Its all very strange , As I say , it was all working fine until I patched for Oracle , afterwhich when it didnt work after 3 or 4 days of struggling I basically wiped everything bar the O/S , but it looks like something is hanging onto those disks somewhere along the line

[The Gunners]

dubhptst: #
dubhptst: # ll /dev/rdsk/c4t0d0
crw-r----- 1 bin sys 188 0x040000 Jul 21 16:40 /dev/rdsk/c4t0d0
dubhptst: # ll /dev/dsk/c4t0d0
brw-r----- 1 bin sys 31 0x040000 Jan 12 2009 /dev/dsk/c4t0d0
dubhptst: #

[TTr]

The disk files are fine. Lookin at all your attachements they all have extraneous text in them which is probably caused by an incorrect terminal setting. Nothing to do with your disk problem.

Have you installed any patches on this server?

The logical drives are marked failed when all 3 physical drives are OK. The only reason I can think of is if you pulled out the disks (and they are marked as having been pulled) and while you were doing that the controller was rebuilding the disks. You probably have to recreate them. Maybe Duncan will see something else. (Duncan, sorry for jumping in)

[The Gunners]

HI TTR , Thanks for that indeed , when you say rebuild them , what do you mean exactly ? , (btw , yes it is a possibility that I pulled them during a rebuild all the various hair pulling exercises trying to get this solved myself)

[Duncan Edmonstone]

I'm no expert on these RAID cards, but what you said in your last post concerns me - are you saying you might have pulled out these disks from their enclosure? And possibly not put them back in the same location??

Anyway at this stage, I think you might as well try resetting the RAID card and rescanning, to see if that rests the logical volumes...

sautil /dev/ciss4 reset_ctlr
sautil /dev/ciss4 scan
saconfig /dev/ciss4


HTH

Duncan

[The Gunners]

Hi Duncan , yes I will try that , just to confirm that the issue was already there before I started pulling disks out etc , it was kinda last resort type stuff when I thought that mabey the issue had been a hardware related one.

[TTr]

Perhaps "rebuilding" is not the right word to use when talking about disk arrays. I was refering to having to delete them and recreate them. If you have data on them it will be lost so you may want to play around a little bit more before resorting to deleting the logical disks. I don't know what status your firmware is but since this is an old server you might want to check if any firmware upgrades are needed on the server and the raid controller.

[The Gunners]

No , all the data that was on them was test data at the time of the issue , as I say I was in middle of a fresh build at the time. Everything that was on those disks can go (I imagine it is gone at this stage now anyway with all the various things I have tried) So my main aim is to literally just get back to scratch again nd be in a position to use the disks and start again with the whole build

[TTr]

At this point you may want to do just that, start over. If you have problems again, use the SA utilities to check the status of the devices. Actually you should check the status of the logical disks in between each step as you start configuring them to narrow down at which step in the process they start to go bad. Also I have a concern about sam. It may not treating those logical volumes correctly when it initializes them. I don't know why I somehow do not trust sam.

[The Gunners]

Hi Guys,
Here is the output from the sautil commands (Duncan thanks) . Also this is the exact steps which happen when Im trying to create a new volume group on those external disks :
using SAM
Checking for Volume Group on disk at 0/6/1/0/4/0.0.0

Checking for an unmounted file system

Creating the Volume Group

Pvcreate : writing lvm record: device busy

Was having a think about all this at the weekend. Im thinking (altho not 100% sure) one of the 'internal' disks was also used in the initial Volume groups , so a combo of the external and one of the internals ? (obviously not the vg00 Operating System one) and if this mabey has a hold over the whole thing now ?

[Duncan Edmonstone · Accepted solution]

OK, so reading some of the docs on this it seems that when drives are removed and then re-inserted, this is treated as a "medium exchange" and the logical drive is marked as failed until you tell it otherwise... if this is the case here (and I'm only surmising/guessing) then maybe running the following commands will sort this out:

sautil /dev/ciss4 accept_media_xchg 0

sautil /dev/ciss4 accept_media_xchg 1

saconfig /dev/ciss4

you will have to confirm the action for each command.

Lets see if that puts the LOGICAL drives back into an OK state...

HTH

Duncan

[The Gunners]

Duncan , just ran that and it looks like it has done the trick Wow! I created two volume groups no problem now see below

File List View Options Actions Help â
â Press CTRL-K for keyboard help. â
â â
â â
â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â
â Volume Groups 0 of 3 selectedâ
â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â
â Mbytes Physical Logical â
â Name Type Available Volumes Volumes â
â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â â
â â vg00 LVM 50096 of 69984 1 8 ^ â
â â vg01 LVM 138896 of 138896 1 0 â
â â vg02 LVM 138896 of 138896 1 0 â
â â â

Its my first time using this forum and its been a brilliant help to me (as a novice on hp/ux) I'd like to say thank you guys for the great advice. Can I fill out anything for you in the form of a customer feedback or something / or is it a points award on this forum) Thanks a mill