topic Re: URGENT!!!! system down, please help! in Operating System - HP-UX

URGENT!!!! system down, please help!

Alex Lavrov — Thu, 02 Oct 2003 20:10:15 GMT

hi,
i got a superdome with 4 hard partitions.
just rebooted all of them, and two of the partitions booted very weird:
it seems that somehow the instance numbers of LAN cards and ext_buses, so now it does not see disks the same way (c14tXdX is now C1tXdX etc...). same with lans (lanX became lanY)

we installed a quad network card on both machines lately but we have rebooted since and it was just fine..... now, for some reason, the instance numbers in ioscan, that were unsorted, became serial (1,2,3,4...) thus ext_buses with high number like 14 no longer exist.

please help!

Re: URGENT!!!! system down, please help!

Michael Steele_2 — Thu, 02 Oct 2003 20:18:38 GMT

Boot off of old kernel, etc.

Re: URGENT!!!! system down, please help!

Steven E. Protter — Thu, 02 Oct 2003 22:06:11 GMT

You have a major problem with the I/O subsystem. One of the core I/O subsystems has probably failed. Now you are not getting good data from I/O scan.

Since you can afford a superdome, you probably have a hardware contract. its time to call them.

Use cstm, mstm or xstm to test the I/O on the system and try and get some more information.

SEP

Re: URGENT!!!! system down, please help!

Jakes Louw — Fri, 03 Oct 2003 01:19:00 GMT

I've had a similar case where the installation of an extra card caused major havoc with our hardware, due to the card and instance numbers all changing.
I assume the old paths (/dev/dsk/c14tXdX) are also missing (special files gone)?
Have you tried an "insf -e" ?

Re: URGENT!!!! system down, please help!

Alex Lavrov — Fri, 03 Oct 2003 02:44:27 GMT

well, as it might surprise you, ITRC guys are more competent than regular support sometimes.

I got an old /etc/ioconfig file from backup and booted with it. Everything is ok. BUT, whenever I reboot again, the file is somehow corrupted again and the machine loses all the instance numbers.

anyone? why does it happen?

Re: URGENT!!!! system down, please help!

Duncan Edmonstone — Fri, 03 Oct 2003 03:23:29 GMT

When a system boots, /sbin/ioinitrc compares the current IO configuration with what the kernel thinks is the current IO configuration (it does an ioinit -i -r). My guess is that your ioconfig in /stand doesn't match the one in /etc. Check them using cksum. Here's an example from one of my lab systems:

cksum /stand/ioconfig /etc/ioconfig
3190746677 2400 /stand/ioconfig
3190746677 2400 /etc/ioconfig

HTH

Duncan

Re: URGENT!!!! system down, please help!

Alex Lavrov — Fri, 03 Oct 2003 03:40:31 GMT

Here is the situation:
0. System boots
1. All card instances are mixed. nothing works.
2. I restored /etc/ioconfig and copied it to /stand/ioconfig.
3. I reboot
4. System boots ok, finds a quad card during ioinitrc and configures it ok.
5. Everything works
6. I reboot
7. Something messes up the ioconfig and returns me to step 1.

What the hell is it???

Re: URGENT!!!! system down, please help!

Jakes Louw — Fri, 03 Oct 2003 03:48:37 GMT

As I mentioned in my previous posting, it would appear that the quad card is forcing the card and instance IDs to shift, thereby forcing a reconfig of ALL the device file paths.
I guarantee that if you remove the cards, your problem will go away, but that's hardly a solution, right?
So what you are going to have to do is build vgexport map files (vgexport -m mapfile -f outfile....).
Then when you have a screwed-up config, import the VGs using the map files....

Re: URGENT!!!! system down, please help!

Nico Schockaert — Fri, 03 Oct 2003 04:01:48 GMT

Hello,
I didn't tried, but maybe this can help.
Reliable, requires two reboots and works without additional tools.

1. Extract a configuration template from the current ioscan output.
Execute the following command:

# ioscan -f | grep -e INTERFACE -e DEVICE | \
grep -v target | \
awk '{print $3, $1, $2}' > /infile

Make sure to store infile to the root file system!

2. Edit /infile and change the ext_bus and lan instances as desired.
No class is allowed to get more than one line for the same instance!

3. Move away the current ioconfig files and Shutdown/Reboot:

# mv /stand/ioconfig /stand/ioconfig.sav
# mv /etc/ioconfig /etc/ioconfig.sav
# shutdown -ry 0

4. Due to the missing ioconfig files the system will come to an
ioinitrc prompt. Now recreate new ioconfig files from scratch.
This prevents you from running into possible assignment conflicts.

(in ioinitrc)# /sbin/ioinit -c

5. Apply the ioconfig change with your prepared infile:

(in ioinitrc)# /sbin/ioinit -f /infile -r

The system will reboot again now if the change was successful.
Warnings like 'Input is identical to kernel' can be ignored.

6. Once the system reboots, verify that all the instance numbers
were changed as expected. It may be necessary to re-import volume
groups to ensure that /etc/lvmtab contains the correct
entries. The lan configuration may need to be changed also.

Hope this helps,
Nico

Re: URGENT!!!! system down, please help!

Michael Steele_2 — Fri, 03 Oct 2003 07:56:02 GMT

The instance number is determined by its slot location. So if you remove the quad card and reload the ioconfig you'll be back to where you were. Once there determine where to place the quad card, i.e., after everything else.

Note Twin Turbo slots and weather or not the quad card requires one.

Re: URGENT!!!! system down, please help!

Colin Topliss — Fri, 03 Oct 2003 08:33:18 GMT

Just re-instance the cards then.

Do a man on ioinit. Basically you can force cards to an instance number you define. No need to remove ioconfig, vgexport disks, or remove/move hardware etc etc.

See the thread:
http://forums1.itrc.hp.com/service/forums/bizsupport/parseCurl.do?CURL=%2Fcm%2FQuestionAnswer%2F1%2C%2C0x9549b82b2d63d71190080090279cd0f9%2C00.html&forumId=2&admit=716493758+1065187789323+28353475

-f infile Use the file infile to reassign instance numbers to
devices within a specified class. infile may have
multiple entries, each to appear on a separate line,
each field in the entry separated by 1 or more blanks.
Entries should conform to the following format:

h/w_path class_name instance_#

ioinit preprocesses the contents of infile, looking for
invalid entries, and prints out explanatory messages.
An entry is considered to be invalid if the specified
hardware path or class name does not already exist in
the system, or if the specified instance number already
exists for the given class.

Regards

Col