Cannot create VG using drives in JBOB DS2400

[Steve Longenecker]

I am having problems creating a volume group using drives in a DS2400.
I can see the drives with ioscan and diskinfo. I can pvcreate as well. But issuing vgcreate results in:

# vgcreate /dev/vg02/ /dev/dsk/c6t3d0
vgcreate: Physical Volume "/dev/dsk/c6t3d0" contains no LVM information

The problem exists with all disks in the DS2400.

Tried using SAM to create a VG, but received the same error message.

Correct 11i drivers and patches are installed. Any ideas?

[S.K. Chan]

Try the "-f" (force) option with pvcreate. Caution though, be very sure c6t3d0 is not used in any existing VG or it's LVM header will be overwritten.
# pvcreate -f /dev/rdsk/c6t3d0
==> Use the raw device.
# mkdir /dev/vg02
# mknod /dev/vg02/group c 64 0x020000
==> Ignore the "mkdir" and "mknod" if you have done this earlier.
# vgcreate /dev/vg02 /dev/dsk/c6t3d0

[harry d brown jr]

First you have do the device: pvcreate /dev/dsk/c6t3d0, then you have to create the directory /dev/vg02. then you have to create the group file: mknod /dev/vg02/group c 64 0x020000. Then vgcreate /dev/vg02 /dev/dsk/c6t3d0.

live free or die
harry

[Steve Longenecker]

Been there... done that. All manual steps followed "by the book".

[Jeff Schussele]

Hi Steve'

I'd suspect a minor device number problem.

When you do the vgcreate manually you MUST have created the group file prior.
It's possible that in the past someone may have created a VG using a minor number out-of-sequence, which will screw up SAM.
Other thing to check is the maxvg kernel parameter. By default the kernel only supports 10 VGs. If you already have ten VGs - you'll have to increase maxvg.

Also you do not need the trailing / in the /dev/vg02 parameter in the vgcreate command. But I doubt that's the problem here.
Also did you use -f on the pvcreate command?

HTH,
Jeff

[harry d brown jr]

What does

ls -l /dev/vg*/group /dev | grep vg

return??

And

pvdisplay /dev/dsk/c6t3d0

returns?


live free or die
harry

[harry d brown jr]

and ls -l /dev/dsk/c*t*d* ???


live free or die
harry

[Steve Longenecker]

This is a new server with only 2 volume groups, vg00 and vg01, with vg01 created using SAM as a test with internal drives.

vg02 is the first volume group created with the drives in the DS2400. Tried SAM first, but it failed. Then tried to create vg02 manually, using the correct commands, but received the error message during the vgcreate.

Kernal parameter is not the problem.
Minor device number is not the problem.
The trailing "/" was a typo creating the forum message, and was not used with the actual vgcreate.
Used "-f" with pvcreate.
Used the "raw" device files.

# ls -l /dev/vg*/group /dev | grep vg
crw-r----- 1 root sys 64 0x000000 Sep 28 14:19 /dev/vg00/group
cr--r--r-- 1 root sys 64 0x010000 Sep 29 13:00 /dev/vg01/group
crw-rw-rw- 1 root sys 64 0x020000 Sep 29 14:59 /dev/vg02/group
drwxr-xr-x 2 root root 1024 Sep 28 14:19 vg00
dr-xr-xr-x 2 root sys 96 Sep 29 14:23 vg01
drwxrwxrwx 2 root sys 96 Sep 29 14:59 vg02

NOTE:
vg00 created at install
vg01 created using SAM
vg02 created manually, but failed.

# pvdisplay /dev/dsk/c6t3d0
pvdisplay: Couldn't find the volume group to which
physical volume "/dev/dsk/c6t3d0" belongs.
pvdisplay: Cannot display physical volume "/dev/dsk/c6t3d0".

This physical volume, along with the other three, are all new and have never been used.

# ls -l /dev/dsk/c*t*d*
brw-r----- 1 bin sys 31 0x010000 Sep 29 12:30 /dev/dsk/c1t0d0
brw-r----- 1 bin sys 31 0x012000 Sep 28 14:19 /dev/dsk/c1t2d0
brw-r----- 1 bin sys 31 0x020000 Sep 29 12:30 /dev/dsk/c2t0d0
brw-r----- 1 bin sys 31 0x022000 Sep 28 14:19 /dev/dsk/c2t2d0
brw-r----- 1 bin sys 31 0x032000 Sep 28 14:19 /dev/dsk/c3t2d0
brw-r----- 1 bin sys 31 0x050000 Sep 28 16:24 /dev/dsk/c5t0d0
brw-r----- 1 bin sys 31 0x051000 Sep 28 16:24 /dev/dsk/c5t1d0
brw-r----- 1 bin sys 31 0x052000 Sep 28 16:24 /dev/dsk/c5t2d0
brw-r----- 1 bin sys 31 0x053000 Sep 28 16:24 /dev/dsk/c5t3d0
brw-r----- 1 bin sys 31 0x060000 Sep 28 16:24 /dev/dsk/c6t0d0
brw-r----- 1 bin sys 31 0x061000 Sep 28 16:24 /dev/dsk/c6t1d0
brw-r----- 1 bin sys 31 0x062000 Sep 28 16:24 /dev/dsk/c6t2d0
brw-r----- 1 bin sys 31 0x063000 Sep 28 16:24 /dev/dsk/c6t3d0
#

The last eight entries are the four FC drives as seen through two FC HBAs.

[Jeff Schussele]

Hi (again) Steve,

Well you're system has certainly covered the gamut with permissions on the vg devices.

I'd now suspect the SCSI subsystem itself....cables, termination ,IDs, etc.

I'm not sure if you can, But I'd try to dd some info INTO that that device. You say you can diskinfo it, but that's a minute read. I'd be curious if it can handle a stream of data.

Lastly, you obviously have dual paths to the array - Do you have different SCSI IDs on the controllers?

Rgds,
Jeff

[Jeff Schussele]

Oops...meant group files....

Jeff

[Steve Longenecker]

Don't forget that this is a DS2400 that is attached to the server via dual fibre channel host bus adapters, therefore no SCSI termination is involved. I am new to fibre channel storage, and documentation in extremely thin, I.E. the DS2400 came with a Virtual Array manual that does not address using the DS2400 in a standalone configuration.

[Jeff Schussele]

Hi Steve,

Sorry, am totally unfamiliar with the DS2400.
But if it's fibre channel array & you're using a Tachyon card then you should be able to run the fcmsutil command.

Try this command:

fcmsutil /dev/td1
(Your dev file may vary & you'll have two of them)

This is going to give basic info such as WWN (World Wide Number - like a MAC address for the FC card), fabric topology, link speed, driver state, active logins, etc.

You may see the problem right away here.

If nothing obvious here - try:

fcmsutil /dev/td1 stat

This will give detailed fibre channel statistics.

Sure sounds like an FC communication problem.

Is there a Brocade switch in the path or is this direct connect? Your link type may be mismatched.

Rgds,
Jeff

[Steve Longenecker]

Jeff,

You may be on to something... fcmsutil stat shows on both Tachyon cards:

Loss of Sync 5
Bad Rx Char 255
CE_FCP_UNFREEZE recvd in CS_FCP_FROZEN 5485
Error Idle Interrupts 5486
ERQ/FCP Assists Resumed 5486
SF failed due to ASN, LKF or FTO 5485

[Jeff Schussele]

Steve,

OK, first thing I'd now suspect seeing those stats is bad/dirty fibre. Definitely clean both ends & preferrably try new fibre cables.

If after cleaning, all is well, it could have just needed to relogin to the array. I've seen that before....

Do you have a Storage team? If not who installed the system including cabling? I'd have them back & have them try new fibres. Also have them verify the setup on the DS2400 for the link type i.e. is the DS2400 expecting switched fabric or loop topology.

Rgds,
Jeff

[Steve Longenecker]

Jeff,

Hmmm.... Would you believe that it shipped that way from the factory. Sounds like it's time to invite HP for a visit if my spare cables don't work. What do you mean "relogin to the array"? Thanks.

[Jeff Schussele]

Steve,

Any time a fibre channel device powers up or resets it has to login in to the destination.

I've had Brocade to XP array connections that have "frozen" & simply unplugging the Brocade-to-XP fibre reset the connection.
Our storage guy told me that unplugging/replugging it causes the Brocade to relogin to the XP.
Same's true for a Tachyon-to-Brocade & in your case the Tachyon to DS2400, I'd believe that the same holds true.

I think he calls it logging in, but it's probably just a communication negotiation between devices or an initialization if you will. I really don't fully grasp the complexities of fibre comm - I do know that it's capable of handling many different protocols including SCSI.

But the basics apply to it as it would to a SCSI device running on copper - rules are to be followed & initialization must take place for the communication to be successful. There must be arbitration so no one device can hog all the bandwidth. There's a lot of error detection & recovery constantly going on & if somebody doesn't play by the rules, they are essentially cut off or ignored as they could bring the whole bus down.

You definitely need somebody to check your setup - something's not right.

Rgds,
Jeff

[Ted Ellis_2]

we have the 2405 disk systems here and when we received them the disks would not let us vgcreate either.... when we ran pvcreate -f, the command took a long time, but would report s successful completion. Then vgcreate would fail. Our solution was to load this utility called CommandView SDM... it was shipped with the unit. The disks required "formatting", if you can believe that. Normal swinstall will load the software, no reboot. Their is a pdf file with all the goods... if this applies to you.

Once the CommandView has been installed (after you have the disk systems on the server), you can run "cvui", which is just a little interface. If this is you, you may also need to format the disks. After the formatting, the pvcreate and vgcreates all worked normally.

Ted

[Steve Longenecker]

Ted,

I like your answer better than HP's. HP remarketing told our reseller that the DS2400 will work standalone. HP support tells me that the DS2400 will only work as an expansion shelf for a VA7400/7410, and is not supported as a standalone product.

I had the same experience with pvcreate -f. The command would take a long time, but would completed without error.

While I am waiting for my reseller to fight the good battle with HP remarketing, I'll have to find a copy of CommandView and give it a try. I am not having any success finding a downloadable copy at the HP S/W depot, so would you mind doing an swlist for me and update this thread with the product number for CommandView?

Thanks,

Stev

[Ted Ellis_2]

interesting indeed.. the DS2405 is JBOD supported...here is the swlist

CMDVIEWSDM A.1.05.00 hp StorageWorks CommandView SDM

and I can almost guaruntee this is your problem too. You can see the disks with ioscan just fine, but vgcreate fails. When the product installs it will build some sort of small database that lists all of the current disks and fibre adapters on the unit. You may need to update your path so you can run "cvui"... this will launch a cheesy little interface... you can then select each of the disks on your array and issue a format request. They can be done in parallel.... 10 - 73 GB drives took 1-2 hours. Then pvcreate and vgcreate work just fine...

feel free to email me direct at tellis@triu.com for more info

Ted

[Ted Ellis_2]

here is what I found with HP:

http://www.hp.com/products1/storage/products/disk_arrays/modular/commandview/overview/index.html

Ted

[Ted Ellis_2 · Accepted solution]

http://www.hp.com/cposupport/information_storage/support_doc/lpg64105.html

oops.. looks like a license will cost some $... my 2405 systems came with this in the box... and it was not on my order anywhere

Ted

[Steve Longenecker]

Ted,

Installing CommandView SDM and formatting the drives did the trick.

Thanks,

Steve

[Tommy_8]

Hi Steve,

I have also been experiencing the same problems so I thought I'd try the Command View Util.

I have installed Command View, and when I run "CVUI" the simple interface comes up and say's "getting for devices", then I'm left with a list of options:

1 Refresh
a App menu
x Exit

It appears that the utility cannot see my disk array attached?? Because from what I can gather the util scans for arrays connected and then brings up a sub menu for that array.

However when I do an ioscan -fnC disk all the disks in the array are "CLAIMED" and they all have the special files with them (eg: /dev/dsk/c7t0d0 and /dev/rdsk/c7t0d0).

Any advice or help would be greatly appreciated.

Regards.

[Steve Longenecker]

Tony,

Here's what I followed to format the DS2405 drives:

1. Insert the drive in the DS2405 chassis

2. "discover" the drive
# ioscan -fnC disk

3. Create device files
# insf -e

4. Discover the DS2405 if not already "visible"
# armdiscover

5. To display DS2405 information
# JBODdsp -I

6. Format the drive - command line
# JBODfmt -f /dev/dsk/c8t4d0
-or- if already formatted
# JBODfmt -foverride /dev/dsk/c8t4d0

7. Format the drive using cvui
select drive number
select "Disk"
select "Format"

You probably need to "Discover" the disk array before running cvui. If that does not work, you may have run into a problem I had recently when I upgraded Command View SDM from v1.06 to v1.08 (v1.07 applies as well...), which required a firmware upgrade for the drives and Link Control Cards (LCCs) BEFORE the CVSDM upgrade. What version of CVSDM did you install? What firmware version are you running?

You also may want to consult the CVSDM JBOD manual titled "HP StorageWorks Command View SDM for Disk Systems" for current info. The download page is at: http://h20000.www2.hp.com/bizsupport/TechSupport/DocumentIndex.jsp?contentType=SupportManual&locale=en_US&docIndexId=179911&taskId=101&prodTypeId=12169&prodSeriesId=64112

Good luck,

Steve

[Tommy_8]

# armdiscover

and then

# JBODfmt -f .......

did the trick.

Thanks Steve.