Phantom drives

[TheJuiceman]

We are getting the following errors during boot-up:

lvlnboot: warning: couldn't query physical volume.
The specified path does not correspond to physical volume
attached to this volume group.

We have some 164 "phantom" device files that our machine sees.

To give you all some background, our HP system is plugged into a EMC Symmetrix disk array. EMC makes some software called TimeFinder that allows for the creation of what they call "BCV"'s that can be used on the array itself for mirroring without using LVM or using system resources on the HP box. This, in turn, created a multitude of "place keeper" devices on the array that were used for this software. From the HP side, these appear as usable drives, only they are too small in size to partition. We eliminated the software, but now we are getting the errors I mentioned earlier. EMC has been at a loss to correct the problem.
These error messages only appear at boot. They have not recorded to the syslog on the HP.
Has anyone ever encountered something like this before? Thanks.

[S.K. Chan]

Can't really comment much about the EMC configuration since I do not have much experience with it yet but what you can do to get rid of these errors is modify /sbin/ioinitrc which get called during bootup to scan for boot VG. Look for the line that has ..
/sbin/lvlnboot -c

and change that to
/sbin/lvlnboot -c /dev/vg00

Assuming your root VG is called "vg00". That way it won't scan the rest of the VG. The /stand/rootconf will get overwritten everytime /sbin/lvlnboot -c /dev/vg00 is run.

[Michael Tully]

The normal way that these drives are set up are as follows.

EMC R1 drive (real LUN)
EMC R2 drive (copy LUN if used)
EMC BCV drive (copy of R2 LUN if used or copy of R1)
The BCV's are not normally defined on your server unless they have been deliberately done using a number of commands provided by EMC. 'symrdf' 'symmir' and 'symbcv'

Do these phantom drives appear in your ioscan output at all?
If you've removed the EMC software the drives may be corrupted anyway, so perhaps that can be removed by using 'rmsf'

[TheJuiceman]

Thank you for the input. Running an ioscan does display these devices. Also, removing the devices is only temporary because as soon as a reboot is executed, the devices return (also an ioscan still sees the devices without drivers).

As for the BCV's, they were deliberately configured on the system for use with the TimeFinder software. The software has since been removed.

Thanks again for your help. Any other thoughts?

[Brian M Rawlings]

Bobby: rmsf has lots of options. By default, it doesn't remove the device files that are stored in the running kernel (in memory). See the man page for rmsf, there are options to remove both the device file in /dev, and the entry in the running kernel. That may help with troubleshooting, at least.

As to the phantom drives that are presumably being presented by the EMC array, there is a control file for the array, once accessible by EMC techs only, called the 'bin file'. This file defines all operational parameters of the array, such as which hypers (volumes within the array) are presented down which host directors, to be seen by which hosts, etc.

If you have EMC's ECC/Open Edition (EMC Control Center) software, you can manipulate many elements in the 'bin file', such as which hypers are presented on which ports, etc. This includes the BCVs (timefinder volumes). If you don't have this very nice Control Center suite, EMC will have to make a 'bin file' change, if you wanted to redefine which hypers (or BCVs) are seen on which ports, and by which machines.

Just removing the Timefinder SW does not change how BCVs are presented, which channels/ports they are seen on, etc. I suspect that you are still seeing the BCVs from the Timefinder implementation.

BCVs are a wierd animal, and change state as they are synched up or split off from the standard hypers/volumes. In some states, they can be seen & mounted by servers to which they are presented. In other states, they either cannot be seen at all, or, if seen, they cannot be mounted or otherwise manipulated. As I said, strange... but they have to work this way do do their job.

I'm concerned that EMC is baffled by this, it seems unlikely that this is all a mystery to them, since all installations with Timefinder, or just with BCVs, work this way. Maybe I'm off-base, and this isn't the issue at all... but it sure sounds like what I would expect to see, if BCVs were once in use by a system, and you stopped using the SW that controlled them. If left to their own devices, they could bedevil you pretty good. Nice pun, if unintended...

Hope this helps some, strange problem (or you wouldn't have asked for help, I guess).

Best Regards, --bmr

[TheJuiceman]

Brian,

Thank you for the in-depth response. EMC confirmed today that you are on the right track. Even though we had a bin file change after the TimeFinder software was uninstalled, apparently there is some corruption somewhere inside the EMC. The bin file shows all of the drives available to each path on each server, however the drives that the HP is seeing do not exist as far as the bin file is concerned. Of course there must be something there for the HP to keep scanning and displaying these drives as part of its ioscan. EMC confirmed that all the proper steps were taken to eliminate the TimeFinder software and that this is an unusual case.
We also discovered that another one of our HP servers that is plugged into the same EMC array is doing the same thing. The catch here is that this server NEVER had TimeFinder on it!!! No BCV partitions were ever created for that server.
Any additional thoughts are always welcome. Thanks for the help.

[Kavan Bhatt]

Hi,


I would like to know where is the bin file stored on symmetrix and what is its use.

2.On which type of device can backup and clone operations occur simultaneously?

Thanks a lot.

Kavan.