BladeSystem - General
cancel
Showing results for 
Search instead for 
Did you mean: 

Advice on setting up new blade systems

Mike O.
Regular Advisor

Advice on setting up new blade systems

We recently purchased two C7000 blade systems to use with VMWare to migrate approximately 130 older Proliant systems. We're working with a consultant on the virtualization migration, but we need to do the basic hardware setup ourselves.

Along with the hardware, we also purchased some HP training for the Blade system, the Insight software, and the Virtual Connect interconnects. However, due to a whole series of issues, the training purchase was delayed, and the result is that we have the hardware, and need to get it going, but won't be attending any of the training for at least a month.

We've been assembling the hardware and getting it ready, and I'm looking for some suggestions on the next steps.

Here's the status:

We have two C7000 chassis. Each chassis has four Virtual Connect Ethernet interconnects, two VC Fiber interconnects, and two onboard admin cards. We have 5 BL465G5 blades for each chassis, each with two AMD quad core CPU, 32G of RAM, and an Ethernet and Fiber Channel mezzanine card.

All of the chassis hardware has been assembled, but I haven't powered it up yet. None of the Ethernet or Fiber Channel virtual connect modules have been connected to our production network or SAN infrastructure.

We're also setting up a dedicated DL360 to run the HP Insight software. The network ports from the onboard admin cards (two from each chassis) and one of the NIC's on the DL360 are all connected to each other on a private network.

The DL360 has Windows 2003 installed, but I haven't installed any of the HP management software yet.

I'm not sure what I should do at this point. I want to get the hardware all powered up and configured, and update any firmware (if necessary).

What I was thinking I would do next is power up the blade chassis (with no blades) and use the built in LCD to set the IP of the onboard admin cards. Then install the software on the DL360 and let it find the OA cards. Then use the installed software to configure the chassis and interconnect modules. After the chassis is configured, install the blades and install the O/S on them.

Does this sound like the correct sequence?
On the HP software, what modules should I install? It looks like the disk contains a lot more things than what we'll need.

I'm working through the HP documentation, and searching the net, but any suggestions would be appreciated.

Mike O.
17 REPLIES
Adrian Clint
Honored Contributor

Re: Advice on setting up new blade systems

Mike,

Setup the OA IP on the front.
Connect up the OA's to the network.
Login to the OA's and run thru the first time wizard and make sure you setup the EBIPA for all the blades and interconnects.
Download the Blade Firmware 1.41 Update CD. (On the drivers page)
Connect to each iLO on the blades and using virtual media mount the CD or an ISO and let each blade boot up and then let it update the firmware.
After they are all done.
Download the 2.32 OA firmware and upgrade that on the OAs.
Download the Virtual Connect firmware 1.34 and the firmware update utility. Update the firmware using the CLI.
Connect up any stacking cables between the left hand VCs and between the right hand VCs.
For the DL360.
Download the Microsoft WAIK DVD
Install the ICE DVD and let it run the ICE advisor program. Get all the pre-reqs it wants done.
And that lot should take you at least a day. So post where you are at when finished.
I am doing exactly the same today.
The Brit
Honored Contributor

Re: Advice on setting up new blade systems

Mike,
first thing to think about is where to put your Mezz cards and VC Modules. (As you will learn, this is critical)

You should put your 4-Port NIC card on MEZZ 2, since this is the only slot that can map all 4 NIC Ports. In this case the ports map to IC modules 5, 6, 7 and 8, so this is where you must put your 4 ENet VC modules.

Put your dual port FC HBA on Mezz 1, this maps to IC 3 and IC 4, so that is where your FC VC modules should go.

Now remember that ALL of your servers must have the Mezz cards configured the same since the connections from the servers to the IC modules is Hardwired and cannot be modified.

Hope this helps

Dave
Mike O.
Regular Advisor

Re: Advice on setting up new blade systems


We not using 4 port mezzanine cards, we're using a 2 port Ethernet card. The first two NIC's on the motherboard will be interconnect bay 1 and 2, the 2 port NIC on Mezzanine card 1 will be interconnect 3 and 4, and the two port Fiber Channel (on mezz 2) will go to interconnect 5 & 6 (the VC FC modules). Interconnect 7 and 8 are empty.

The Brit
Honored Contributor

Re: Advice on setting up new blade systems

Sorry about that Mike,
I was the one who got confused. I was thinking that the 4 eNet modules were additional to IC bays 1 & 2.

The reason I suggested Mezz2 for your NIC, (2-port or 4-port) is that you mentioned that you are going to use this for VMWare.

I'm not a VMWare guy, however I work closely with a guy who is, and his constant comment is that there aren't enough NIC's for what he needs.

The point to consider is that Mezz1, by definition, will never map more than 2 ports, whereas Mezz 2 has the capability of mapping 4 ports. The current family of FC cards do not include any 4-port cards that I am aware of. Therefore putting the FC card on Mezz1 loses you nothing, whereas having the Ethernet card on Mezz2 allows you the ability to upgrade to a 4-port card in the future, without having to disturb your FC setup, all you will have to do is add 2 extra ENet modules in 7 and 8.

(this discussion only applies to halfheight blades, however)

Anyway, just a thought

Dave.
Adrian Clint
Honored Contributor

Re: Advice on setting up new blade systems

Mike,

still .... you want to connect the SAN modules to interconnect bays 3 & 4 (HBA in lowest mezzanine) and the 2nd pair of network interconnects in interconnects 5 & 6.
This will allow you to expand to 6 NICs on a half height server without moving the SAN interconnects.

Best practice... best for 99% of installations.
Mike O.
Regular Advisor

Re: Advice on setting up new blade systems

Thanks, that's a good idea. I didn't realize that you could access interconnect 7&8 with any half height server, I thought you needed full height blades to access those ports.

When I first saw your comment, I didn't understand it, but now I realize that if I put a quad port mezzanine card in connetor two, I can access interconnect port 7 & 8.

Steven Clementi
Honored Contributor

Re: Advice on setting up new blade systems

Mike:

Are you planning on being able to vMotion any of your VM's?

If so, you may want to look at getting the 4port mezz card now or at least plan for it in the future.

Your vMotion network should be separate from your console and public data networks. You would be able to do that now, but end up with no redundancy for both networks.

Now, you can use your console/management network for vMotion as well, but Best Practice calls for them being separate.

Just an idea to think about before you move into your project full force.


Steven
Steven Clementi
HP Master ASE, Storage and Clustering
MCSE (NT 4.0, W2K, W2K3)
VCP (ESX2, Vi3, vSphere4, vSphere5)
RHCE
NPP3 (Nutanix Platform Professional)
Mike O.
Regular Advisor

Re: Advice on setting up new blade systems

Yes, we're going to be using VMotion, unfortunately we're pretty much locked on the hardware at this time (very little additional funding).

We're working with a consultant on the VMWare installation and configuration; they're the ones that recommended the current NIC config (two on board and two port mezz card). From what I understand, they're going to be using 2 NIC for production, and 2 for VMWare. I'm not sure if the idea is one each for VMotion and one for VM management, or set the two NIC's up as a redundant pair and run both VM functions through them.

Initially we only have 10 blades (5 in each chassis), and we're looking at about 130 virtual machines total, so it probably won't be a major performance hit at this time. By the time we need to expand with additional blades we can look at upgrading the existing two port cards (I did move the interconnects so we can take advantage of the 4 port modules).
Mike O.
Regular Advisor

Re: Advice on setting up new blade systems

Now I'm having a problem (or just missing something) on the connection for the OA.

Our network group set up a private subnet on the switches in the 10.5.100.x subnet.

I set up the IP for each OA module using static IP in the 10.5.100.x subnet. I then set the 2nd NIC on the DL360 to the same 10.5.100.x subnet. I was able to ping each OA module and get <4ms response.

I then opened up IE and connected to one of the OA modules. I got the "certificate error" message, then told it to connect anyway. At that point, IE waited a while then timed out. Doing a PING on the OA address either timed out or returned response times in the thousands of ms.

I dug into the documentation some more and found the section on "temporarily" connecting a PC to the chassis uplink port on the 169.254.x.x. subnet. I went ahead and did a direct connection from the chassis to the 2nd NIC on the DL360, set it to the 169.254.1.254 address, and was able to access the system using IE to go to the OA service address the OA credentials.

What am I missing on the connections? If I want to use the DL360 server to manage the chassis and the devices in it, do I connect to the port(s) on the OA, or through the uplink port?

I really wish our purchasing group had got the training set up first...

Mike O.
Steven Clementi
Honored Contributor

Re: Advice on setting up new blade systems

I would definitely make the changes suggested... fibre in mezz 1 and nic in mezz 2 for easier restructuring later on.


It will make life simpler later on.

The only comment I have is that I would probably pair the 2 nics and use them for management/vmotion rather then splte them. Redundancy is more important in my eyes and running both on the same connection is acceptable in small environments.

The only thing I would do is use 1 onboard nic and 1 mezz nic as the first pair, then use the other onboard and other mezz nic as the other pair... so that you are covered incase the mezz card dies for some reason. I haven't seen one die yet.


Steven
Steven Clementi
HP Master ASE, Storage and Clustering
MCSE (NT 4.0, W2K, W2K3)
VCP (ESX2, Vi3, vSphere4, vSphere5)
RHCE
NPP3 (Nutanix Platform Professional)
Mike O.
Regular Advisor

Re: Advice on setting up new blade systems

I've already moved the interconnects and will put the mezzanine cards as suggested.

Thanks again.
Steven Clementi
Honored Contributor

Re: Advice on setting up new blade systems

Mike:

You should not need to use the uplink ports for anything. They are there mostly for trubleshooting problems.

double check your IP settings. Directly attach the server to the OA if you have to using a cross over cable to make sure they can talk to each other properly.

Generally speaking, the networking is usually at fault 75% of the time. (or more). They swear up and down that everything is right on their side.... until they actually check it.


Steven
Steven Clementi
HP Master ASE, Storage and Clustering
MCSE (NT 4.0, W2K, W2K3)
VCP (ESX2, Vi3, vSphere4, vSphere5)
RHCE
NPP3 (Nutanix Platform Professional)
Mike O.
Regular Advisor

Re: Advice on setting up new blade systems

So the port directly on the OA can handle the HTTP page properly, correct? It just seemed odd that the connectivity seemed to be OK according to PING until I tried to send the web info through it.

I was able to get to the login screen once, but while I was typing in the credentials, the connection started timing out.

I;ve been able to connect in fine and run the web page and wizards using the uplink connection.

One more related question: Normally our default gateway is the ".1" entry on each subnet. However, since this subnet they set up isn't routed out anywhere, there is no gateway IP defined. Will this cause any problems for any of the devices if I leave the default gateway blank, or set it to 10.5.100.1?
Mike O.
Regular Advisor

Re: Advice on setting up new blade systems

It looks like I've got it working now..
I checked with our network group, they said that everything was OK at their end.

However, they may be right (at least this time). Using the uplink 169.254.x.x port, I ran the first time setup on the one chassis. I configured the OA modules using the IP's that I had been setting through the front panel. When the wizard completed, I disconnected the uplink port and connected through the regular IP on the OA. It's working fine now. I'm not sure what was different by setting it up that way, but at this point as long as it's working...


Mike O.
Regular Advisor

Re: Advice on setting up new blade systems

Now that the OA connectivity is working, I've been doing the firmware updates. The OA was already at 2.32, so I didn't have to do any on them. I used the Virtual Connect Support Utility to push out firmware 2.01 to the VC ENet modules (1.30 to the VCFC). The update ran and updated the ENet ones, but "failed" on the Fiber Channel ones. After some more research, I reset each VCFC module (nothing it using them yet), then re-ran the utility and they updated to 1.30 OK.

I'm now working on the blades. I have a question about placement of the blades themselves. We're starting out with 5 blades per chassis. Would I do better to put all 5 adjacent to each other, or space the blades out in the chassis? I was thinking that by spacing out the blades, it will spread the heat load out on the fans so that they'll all run at a fairly even speed, instead of some at higher and the others low.


damiri
Regular Advisor

Re: Advice on setting up new blade systems

Mike,

as per HP documentation, you should be able to cool down c-7000 fully packet with the set of fans supplied with it. From this point, it really does not matter where do you put your blades if you are not talking about adding some storage or pci blades later on ...
Mike O.
Regular Advisor

Re: Advice on setting up new blade systems

Thanks. We're not planning on anything except server blades. I just thought it might be better to spread out the load instead of having a few of the fans blasting out at high speed and the others pretty much on idle.