vgimport to wrong disk

[TWBSupport]

This has been h*ll week for me. I've never had so many issues with my HP servers, but I thought I was fixing an issue on my backup server & in turn made a stupid move that affected my production server. We have 2 2470's that acess a disk array. Our backup server lost vg01, drives checked out fine, so upon importing the vg on the backup server, I accidently selected a disk that our production server is using (which is also in productions vg01)... Everything worked fine, so I didn't notice until 1/2 hour later when people were trying to print out of our database & it was giving goofy errors.. Upon troubleshooting a rampid remshd error in the syslogs, on the production server, I noticed what I did.. After the shock of me doing this wore off, I immediately deactivated the vg on the backup back & exported it off the disk. UGh! Database, per the DBA, is working fine, except for printing certain forms. Good Gracious what have I done & what are the ramifications for what I done on the production box...

[Bill Hassell · Accepted solution]

My first suggestion is to isolate the two machines on the disk array. Normally, a modern disk array with multiple ports has a smart controller that grants access to seleted diskspace by only one system (to avoid this problem). If the controller does not allow this, I would get another set of disks (JBOD or another array) and move the test system off production disk hardware.

Now vgimport is fairly benign. It's not like vgcfgrestore where the LVM structures (VGRA/VGDA) write onto over the disk. If the LVM maps are identical on both servers (do a diff between production and test for /etc/lvmconf/vg01.mapfile). If they are different, that is where the problem may have started. Even if they are the same, if both servers are looking at the same disk areas while you're running your database, some codes and data may have been changed without the knowledge of the production machine.

I assume you have used vgexport to remove the vg01 connection (it simply deactivates vg01, then removes all the device files) and shutdown the test machine for now. The DBA will need to do some sanity checking on all the spaces defined on vg01 for the production server.

[TWBSupport]

so what you are saying is that I haven't killed everything? I can crawl out from under my rock? The dba called me a couple hours ago & states that the directories that our oracle scripts looked prior to me doing this are no longer working & state that oracle doesn't have permissions on those directories anymore (log files) specifically /home which resides on VG00 on the local hard drives of the rp2470. I checked for a vgmap on the production box & it doesn't exist, is there somewhere else I can see what the vg structure looked like prior to this change? Could it just be as simple as me re-running vgscan after I rename lvmtab? The box is down right now for a scheduled power maintenance at work, it will be back up in a couple hours & then I have about 4 hours to get this figured out prior to production starting back up.

[Bill Hassell]

It really depends on what was changed. I know that it is quite difficult to remember all the steps. Start with .sh_history for root on the test box. You mentioned remsh so it sounds like you were on another box sending the commands, so check .sh_history there too. That will give you some insight as to the commands that were run on the test box.

Now if the directory permissions indeed changed on a local disk (not NFS, or otherwise shared) than something else is going on. What did the ownership change to? root? Is /home on production shared in any way?

Now the vgmaps are automatically created in /etc/lvmconf whenever you make any change to a VG. If nothing is there, then you'll have to go to a backup to retrieve the maps. But if /etc/lvmconf isn't there, something else is going on.

vgscan simply looks at all the disks and builds the volume groups in lvmtab. The devicefile names and the order in which they map to the VG are in those mapfiles. But, there is hope. Use the -p (preview) option for vgexport to build a copy of map:

vgexport -m vg01.backupmap -p -v -s vg01

This looks at the disks and reads the VG map information. The local file vg01.backupmap should have your current map for this VG.

[TWBSupport]

thanks Bill. Hopefully I will be able to get back into work in the next hour or so & try this out.. HP's telling that more than likely the data is corrupt on the disk & will have to be restored from backup... I'm REALLY hoping not.

[Devender Khatana]

Hi,

It is a very silly situation where your one server is accessing a disk & other server also using same simultaneously. It seems you just only imported the vg onto the test box. Was something like mounting & updation was done on the file systems lying on the disks belnging to production box.

If only vgimport was attempted or even after mount nothing was updated on test box in the VG the data should be intact but again only if this disk contains static data in production such as forms. If this is dynamic data such as datafiles & the data was updated on the test box after vgimport then data integrity would have been lost.

So just find out which file system is there on this disk in production and what does that file system /mount point contain. You should be a lucky one if these were forms.

Moreover this also alrms you for improper system planning previously & you should try to rectify this whenever possible.

HTH,
Devender

[TWBSupport]

well, I'm in the office now. /home drive perms were changed for some odd reason.. I fixed those.

Davender - The only thing I did was a vgimport & vgchange on the test box which the disks I STUPIDLY selected were already being used on the prod machine. Right now, I'm not concerned with the test box, I just want to make sure I fix the prod box.

Actually after the reboot of the prod & test system everything is looking OK, but I still need to test the report printing once the dba is done with a cold backup of the db...

[Devender Khatana]

Hi,

It seems you have escaped from a big problem here, if the problem seems to only for printing. Printing normally uses static forms & should not have any problems like permissions & ownerships. Even if it is effected there should not be any problems in restoring the backup of that file system itself & not the whole data base.

Keep posting the status.

HTH,
Devender

[TWBSupport]

Oh yes, I escaped a bullet here... It looks as though, which I have no explaination of but when the data started synchronizing from the test box, since the 'critical' data files were being used, the rights changed on those files, which in turn caused the permission issues with the logs & printing.. Once I corrected the permissions, everything started working properly again.... Any explaination on that would be greatly appreciated...