CPU Numbering on rx8640

[Jim Hintze]

we are tring to optimize the CPU assignments for the I/O bound and CPU bound processes on a rx8640( clue config below ). It would be handy to understand how the hardware presents the CPU's to VMS. LID field may have a cell byte and CU ID encoded but that is mere speculation.

2 cells 4 cpus each, 2 IO cages with 1 FC in each and 2 or 3 NIC's in each.

Thanks for the reply's
Jim




System Type HP rx8640 (1.60GHz/12.0MB) Primary CPU ID 0.
Cycle Time 0.63 nsec (1598 MHz) Pagesize 8192 Byte

System Management BIOS Information:
BIOS Version 9.048 Manufacturer hp
Release Date 6/3/2008 Product Name server rx8640
SMBIOS Table 00000000.3FFEC020 Serial Number USE481341R
UUID 40C9252A.428E5691.11DD18CC.4EBFF09E

CPU ID 0 CPU State rc,pa,pp,cv,pv,pmv,pl
CPU Type Dual-Core Itanium (Intel Itanium 9100 Rev A1)
Halt PC 00000000.00000000 Halt PS 00000000.00000000
Halt code Bootstrap or Powerfail Halt Req. Default, No Action
Slot VA FFFFFFFF.DDCF3000 CPUDB VA FFFFFFFF.90428000
Package 0 Core 0
Thread id 0 Cothread id None
FW Usage 00000000.00000000 CPU die 0
ACPI CPU id 00000000.00000000 Serial Num 0001688443fcbe36
LID 00000000.00000000 CFG flags 00000000.00000631 Hardware Initialized Primary Present Reassignable

CPU ID 1 CPU State rc,pa,pp,cv,pv,pmv,pl
CPU Type Dual-Core Itanium (Intel Itanium 9100 Rev A1)
Halt PC 00000000.00000000 Halt PS 00000000.00000000
Halt code Bootstrap or Powerfail Halt Req. Default, No Action
Slot VA FFFFFFFF.DDCF4000 CPUDB VA FFFFFFFF.90698600
Package 0 Core 1
Thread id 0 Cothread id None
FW Usage 00000000.00000100 CPU die 0
ACPI CPU id 00000000.00000001 Serial Num
LID 00000000.01000000 CFG flags 00000000.00000611 Hardware Initialized Present Reassignable

CPU ID 2 CPU State rc,pa,pp,cv,pv,pmv,pl
CPU Type Dual-Core Itanium (Intel Itanium 9100 Rev A1)
Halt PC 00000000.00000000 Halt PS 00000000.00000000
Halt code Bootstrap or Powerfail Halt Req. Default, No Action
Slot VA FFFFFFFF.DDCF5000 CPUDB VA FFFFFFFF.9069A500
Package 1 Core 0
Thread id 0 Cothread id None
FW Usage 00000000.00010000 CPU die 0
ACPI CPU id 00000000.00000002 Serial Num
LID 00000000.04000000 CFG flags 00000000.00000611 Hardware Initialized Present Reassignable

CPU ID 3 CPU State rc,pa,pp,cv,pv,pmv,pl
CPU Type Dual-Core Itanium (Intel Itanium 9100 Rev A1)
Halt PC 00000000.00000000 Halt PS 00000000.00000000
Halt code Bootstrap or Powerfail Halt Req. Default, No Action
Slot VA FFFFFFFF.DDCF6000 CPUDB VA FFFFFFFF.9069C480
Package 1 Core 1
Thread id 0 Cothread id None
FW Usage 00000000.00010100 CPU die 0
ACPI CPU id 00000000.00000003 Serial Num
LID 00000000.05000000 CFG flags 00000000.00000611 Hardware Initialized Present Reassignable

CPU ID 4 CPU State rc,pa,pp,cv,pv,pmv,pl
CPU Type Dual-Core Itanium (Intel Itanium 9100 Rev A1)
Halt PC 00000000.00000000 Halt PS 00000000.00000000
Halt code Bootstrap or Powerfail Halt Req. Default, No Action
Slot VA FFFFFFFF.DDCF7000 CPUDB VA FFFFFFFF.9069E380
Package 0 Core 0
Thread id 0 Cothread id None
FW Usage 00000000.01000000 CPU die 1
ACPI CPU id 00000000.00000004 Serial Num
LID 00000000.08000000 CFG flags 00000000.00000611 Hardware Initialized Present Reassignable

CPU ID 5 CPU State rc,pa,pp,cv,pv,pmv,pl
CPU Type Dual-Core Itanium (Intel Itanium 9100 Rev A1)
Halt PC 00000000.00000000 Halt PS 00000000.00000000
Halt code Bootstrap or Powerfail Halt Req. Default, No Action
Slot VA FFFFFFFF.DDCF8000 CPUDB VA FFFFFFFF.906A8E00
Package 0 Core 1
Thread id 0 Cothread id None
FW Usage 00000000.01000100 CPU die 1
ACPI CPU id 00000000.00000005 Serial Num
LID 00000000.09000000 CFG flags 00000000.00000611 Hardware Initialized Present Reassignable

CPU ID 6 CPU State rc,pa,pp,cv,pv,pmv,pl
CPU Type Dual-Core Itanium (Intel Itanium 9100 Rev A1)
Halt PC 00000000.00000000 Halt PS 00000000.00000000
Halt code Bootstrap or Powerfail Halt Req. Default, No Action
Slot VA FFFFFFFF.DDCF9000 CPUDB VA FFFFFFFF.906AAC00
Package 1 Core 0
Thread id 0 Cothread id None
FW Usage 00000000.01010000 CPU die 1
ACPI CPU id 00000000.00000006 Serial Num
LID 00000000.0C000000 CFG flags 00000000.00000611 Hardware Initialized Present Reassignable

CPU ID 7 CPU State rc,pa,pp,cv,pv,pmv,pl
CPU Type Dual-Core Itanium (Intel Itanium 9100 Rev A1)
Halt PC 00000000.00000000 Halt PS 00000000.00000000
Halt code Bootstrap or Powerfail Halt Req. Default, No Action
Slot VA FFFFFFFF.DDCFA000 CPUDB VA FFFFFFFF.906ACA00
Package 1 Core 1
Thread id 0 Cothread id None
FW Usage 00000000.01010100 CPU die 1
ACPI CPU id 00000000.00000007 Serial Num
LID 00000000.0D000000 CFG flags 00000000.00000611 Hardware Initialized Present Reassignable

CPU ID 8 CPU State rc,pa,pp,cv,pv,pmv,pl
CPU Type Dual-Core Itanium (Intel Itanium 9100 Rev A1)
Halt PC 00000000.00000000 Halt PS 00000000.00000000
Halt code Bootstrap or Powerfail Halt Req. Default, No Action
Slot VA FFFFFFFF.DDCFB000 CPUDB VA FFFFFFFF.906AE800
Package 0 Core 0
Thread id 0 Cothread id None
FW Usage 00000001.00000000 CPU die 256
ACPI CPU id 00000000.00000008 Serial Num
LID 00000000.00010000 CFG flags 00000000.00000611 Hardware Initialized Present Reassignable

CPU ID 9 CPU State rc,pa,pp,cv,pv,pmv,pl
CPU Type Dual-Core Itanium (Intel Itanium 9100 Rev A1)
Halt PC 00000000.00000000 Halt PS 00000000.00000000
Halt code Bootstrap or Powerfail Halt Req. Default, No Action
Slot VA FFFFFFFF.DDCFC000 CPUDB VA FFFFFFFF.906B0600
Package 0 Core 1
Thread id 0 Cothread id None
FW Usage 00000001.00000100 CPU die 256
ACPI CPU id 00000000.00000009 Serial Num
LID 00000000.01010000 CFG flags 00000000.00000611 Hardware Initialized Present Reassignable

CPU ID 10 CPU State rc,pa,pp,cv,pv,pmv,pl
CPU Type Dual-Core Itanium (Intel Itanium 9100 Rev A1)
Halt PC 00000000.00000000 Halt PS 00000000.00000000
Halt code Bootstrap or Powerfail Halt Req. Default, No Action
Slot VA FFFFFFFF.DDCFD000 CPUDB VA FFFFFFFF.906B2400
Package 1 Core 0
Thread id 0 Cothread id None
FW Usage 00000001.00010000 CPU die 256
ACPI CPU id 00000000.0000000A Serial Num
LID 00000000.04010000 CFG flags 00000000.00000611 Hardware Initialized Present Reassignable

CPU ID 11 CPU State rc,pa,pp,cv,pv,pmv,pl
CPU Type Dual-Core Itanium (Intel Itanium 9100 Rev A1)
Halt PC 00000000.00000000 Halt PS 00000000.00000000
Halt code Bootstrap or Powerfail Halt Req. Default, No Action
Slot VA FFFFFFFF.DDCFE000 CPUDB VA FFFFFFFF.906B4200
Package 1 Core 1
Thread id 0 Cothread id None
FW Usage 00000001.00010100 CPU die 256
ACPI CPU id 00000000.0000000B Serial Num
LID 00000000.05010000 CFG flags 00000000.00000611 Hardware Initialized Present Reassignable

CPU ID 12 CPU State rc,pa,pp,cv,pv,pmv,pl
CPU Type Dual-Core Itanium (Intel Itanium 9100 Rev A1)
Halt PC 00000000.00000000 Halt PS 00000000.00000000
Halt code Bootstrap or Powerfail Halt Req. Default, No Action
Slot VA FFFFFFFF.DDCFF000 CPUDB VA FFFFFFFF.906B6000
Package 0 Core 0
Thread id 0 Cothread id None
FW Usage 00000001.01000000 CPU die 257
ACPI CPU id 00000000.0000000C Serial Num
LID 00000000.08010000 CFG flags 00000000.00000611 Hardware Initialized Present Reassignable

CPU ID 13 CPU State rc,pa,pp,cv,pv,pmv,pl
CPU Type Dual-Core Itanium (Intel Itanium 9100 Rev A1)
Halt PC 00000000.00000000 Halt PS 00000000.00000000
Halt code Bootstrap or Powerfail Halt Req. Default, No Action
Slot VA FFFFFFFF.DDD00000 CPUDB VA FFFFFFFF.906B7E00
Package 0 Core 1
Thread id 0 Cothread id None
FW Usage 00000001.01000100 CPU die 257
ACPI CPU id 00000000.0000000D Serial Num
LID 00000000.09010000 CFG flags 00000000.00000611 Hardware Initialized Present Reassignable

CPU ID 14 CPU State rc,pa,pp,cv,pv,pmv,pl
CPU Type Dual-Core Itanium (Intel Itanium 9100 Rev A1)
Halt PC 00000000.00000000 Halt PS 00000000.00000000
Halt code Bootstrap or Powerfail Halt Req. Default, No Action
Slot VA FFFFFFFF.DDD01000 CPUDB VA FFFFFFFF.906B9C00
Package 1 Core 0
Thread id 0 Cothread id None
FW Usage 00000001.01010000 CPU die 257
ACPI CPU id 00000000.0000000E Serial Num
LID 00000000.0C010000 CFG flags 00000000.00000611 Hardware Initialized Present Reassignable

CPU ID 15 CPU State rc,pa,pp,cv,pv,pmv,pl
CPU Type Dual-Core Itanium (Intel Itanium 9100 Rev A1)
Halt PC 00000000.00000000 Halt PS 00000000.00000000
Halt code Bootstrap or Powerfail Halt Req. Default, No Action
Slot VA FFFFFFFF.DDD02000 CPUDB VA FFFFFFFF.906BBA00
Package 1 Core 1
Thread id 0 Cothread id None
FW Usage 00000001.01010100 CPU die 257
ACPI CPU id 00000000.0000000F Serial Num
LID 00000000.0D010000 CFG flags 00000000.00000611 Hardware Initialized Present Reassignable

TR Adapter ADP Hose Bus BusArrayEntry Node GSIN iVec SCB Port Slot Device Name / HW-Id
-- ----------- ----------------- ---- -------------------------- ---- ------------------- ---- ---- ---------------------------
1 ACPI_IA64_P FFFFFFFF.905C4100 0 00 IA64_BUS
2 ACPI_IA64_S FFFFFFFF.905C6A00 0 00 IA64_BUS
3 XBUS FFFFFFFF.905C9CC0 0 00 XBUS
FFFFFFFF.905CA098 0 0011 00DE 15E0 CMA: 0 IPMI port
FFFFFFFF.905CA1A8 1 0017 00DF 15F0 SRA: 1 Console Serial Line Driver
FFFFFFFF.905CA2B8 2 007C 00DD 15D0 SEA: 2 ACPI System Event
FFFFFFFF.905CA3C8 3 007C 00DD 15D0 SEB: 3 ACPI System Event
FFFFFFFF.905CA4D8 4 0010 00DC 15C0 SEC: 4 ACPI System Event
4 ACPI_IA64_S FFFFFFFF.905D0400 0 00 IA64_BUS
5 ACPI_IA64_I FFFFFFFF.905D3580 0 00 IA64_BUS
6 PCI FFFFFFFF.905E8780 0 00 PCI
FFFFFFFF.905E8C68 8 001D 00DB 15B0 EWA: 1 BCM5703 LOM (Gigabit Ethernet)
FFFFFFFF.905E8D78 10 FFFF FFFF 0 2 MFPCI
FFFFFFFF.905E8E88 18 FFFF FFFF 0 3 MFPCI
7 PCI FFFFFFFF.905EC280 0 00 PCI
FFFFFFFF.905EC658 10 0019 00DA 15A0 PKA: 2 LSI Logic 1030 U320
FFFFFFFF.905EC768 11 001A 00D9 1590 PKB: 2 LSI Logic 1030 U320
8 PCI FFFFFFFF.905ED480 0 00 PCI
FFFFFFFF.905ED858 18 001B 00D8 1580 PKC: 3 LSI Logic 1030 U320
FFFFFFFF.905ED968 19 001C 00D7 1570 PKD: 3 LSI Logic 1030 U320
9 PCI FFFFFFFF.905F2BC0 0 1C PCI
10 PCI FFFFFFFF.905F7E80 0 38 PCI
11 PCI FFFFFFFF.905FD2C0 0 54 PCI
FFFFFFFF.905FD7A8 5408 FFFF FFFF 0 1 MFPCI
12 PCI FFFFFFFF.90600D40 0 54 PCI
FFFFFFFF.90601118 5408 003A 00D6 1560 FGA: 1 Qlogic ISP2422 Fibre
FFFFFFFF.90601228 5409 003B 00D5 1550 FGB: 1 Qlogic ISP2422 Fibre
13 PCI FFFFFFFF.90603A00 0 70 PCI
FFFFFFFF.90603EE8 7008 FFFF FFFF 0 1 MFPCI
14 PCI FFFFFFFF.90607380 0 70 PCI
FFFFFFFF.90607758 7008 0045 00D4 1540 EIA: 1 A7012A (dual Gigabit Ethernet)
FFFFFFFF.90607868 7009 0046 00D3 1530 EIB: 1 A7012A (dual Gigabit Ethernet)
15 ACPI_IA64_I FFFFFFFF.906095C0 0 00 IA64_BUS
16 PCI FFFFFFFF.9061C780 0 8C PCI
17 PCI FFFFFFFF.90621AC0 0 A9 PCI
18 PCI FFFFFFFF.90626F00 0 C6 PCI
19 PCI FFFFFFFF.9062C340 0 E3 PCI
20 ACPI_IA64_S FFFFFFFF.90630580 0 00 IA64_BUS
21 ACPI_IA64_S FFFFFFFF.90636C40 0 00 IA64_BUS
22 ACPI_IA64_I FFFFFFFF.906392C0 0 00 IA64_BUS
23 PCI FFFFFFFF.9064C840 1 00 PCI
FFFFFFFF.9064CD28 8 0089 00D2 1520 EWB: 1 BCM5703 LOM (Gigabit Ethernet)
FFFFFFFF.9064CE38 10 FFFF FFFF 0 2 MFPCI
FFFFFFFF.9064CF48 18 FFFF FFFF 0 3 MFPCI
24 PCI FFFFFFFF.90650340 1 00 PCI
FFFFFFFF.90650718 10 0085 00D1 1510 PKE: 2 LSI Logic 1030 U320
FFFFFFFF.90650828 11 0086 00D0 1500 PKF: 2 LSI Logic 1030 U320
25 PCI FFFFFFFF.90651540 1 00 PCI
FFFFFFFF.90651918 18 0087 00CF 14F0 PKG: 3 LSI Logic 1030 U320
FFFFFFFF.90651A28 19 0088 00CE 14E0 PKH: 3 LSI Logic 1030 U320
26 PCI FFFFFFFF.90654800 1 1C PCI
27 PCI FFFFFFFF.90659AC0 1 38 PCI

TR Adapter ADP Hose Bus BusArrayEntry Node GSIN iVec SCB Port Slot Device Name / HW-Id
-- ----------- ----------------- ---- -------------------------- ---- ------------------- ---- ---- ---------------------------
28 PCI FFFFFFFF.9065EF00 1 54 PCI
FFFFFFFF.9065F3E8 5408 FFFF FFFF 0 1 MFPCI
29 PCI FFFFFFFF.90662980 1 54 PCI
FFFFFFFF.90662D58 5408 00A6 00CD 14D0 FGC: 1 Qlogic ISP2422 Fibre
FFFFFFFF.90662E68 5409 00A7 00CC 14C0 FGD: 1 Qlogic ISP2422 Fibre
30 PCI FFFFFFFF.90665640 1 70 PCI
FFFFFFFF.90665B28 7008 FFFF FFFF 0 1 MFPCI
31 PCI FFFFFFFF.90668FC0 1 70 PCI
FFFFFFFF.90669398 7008 00B1 00CB 14B0 EIC: 1 A7012A (dual Gigabit Ethernet)
FFFFFFFF.906694A8 7009 00B2 00CA 14A0 EID: 1 A7012A (dual Gigabit Ethernet)
32 ACPI_IA64_I FFFFFFFF.9066B200 0 00 IA64_BUS
33 PCI FFFFFFFF.9067E3C0 1 8C PCI
34 PCI FFFFFFFF.90683700 1 A9 PCI
35 PCI FFFFFFFF.90688B40 1 C6 PCI
36 PCI FFFFFFFF.9068DF80 1 E3 PCI
FFFFFFFF.9068E468 E308 FFFF FFFF 0 1 MFPCI
37 PCI FFFFFFFF.90691980 1 E3 PCI
FFFFFFFF.90691D58 E308 00DD 00C9 1490 EIE: 1 A7012A (dual Gigabit Ethernet)
FFFFFFFF.90691E68 E309 00DE 00C8 1480 EIF: 1 A7012A (dual Gigabit Ethernet)

[John Gillings]

Jim,

"Optimize" for what? What are you trying to improve and what will you be sacrificing for that improvement?

Do you have evidence that there are performance bottlenecks?

In general people are very bad at predicting how resources should be distributed. Any "tuning" typically restricts the operating system (like forcing particular processes to execute on specific CPUs), which is more likely to have a negative impact on performance than improve anything.

You should have hard data showing bottle necks before trying to fix anything. With the right data, details like how CPUs are physically numbered will be irrelevant, because the data will show you what needs tweaking.

[Guy Peleg]

As John mentioned, more details are required
to understand exactly what are you trying
to optimize.

The $ SHOW FASTPATH command will show
how devices are spread across CPUs.

The SET DEVICE/PREFERRED_CPU can be used
to tying a device to a CPU.

Guy Peleg
Maklee Engineering
www.maklee.com

[Jim Hintze]

We have a dedicated application implemented in 20 detached process sharing data in a global section. Some processes are I/O bound with tcpip traffic others are I/O bound with disk I/O and some are CPU bound.
What we have found is that NUMA machines exhibit a latency that is reflected in the performance of our application. This is evidenced by sample runs on 6600's vs. 8640's. We also noticed an improvement when we assigned a cpu bound process to a given CPU(core) and prevented any other process from being scheduled in that core thus perserving the address translation buffer content.

[Hoff]

Various of the Integrity cellular boxes can be run as coherent NUMA (eg: ccNUMA) with cell-local memory or as globally-interleaved memory. Or a combination.

Most (all?) OpenVMS I64 boxes run interleaved (eg: ccUMA). IIRC, ccNUMA wasn't supported with OpenVMS I64 when last I looked.

I'd tend to presume OpenVMS follows the ACPI assignments; the manuals point to the socket order, though I don't see anything that confirms that. CPUs 0 and 1 in socket 0 in cell 0. But given the box is likely configured ccUMA, you can target your CPUs where you want (as was discussed earlier) and then (as usual when tuning) benchmark and compare the change against your baseline.

And as others have stated, this looks to be premature optimization; most any box combined with a complex application almost never works exactly the way you think it does; this is why tools such as DECset PCA and the T4 tool and the alignment fault monitoring and such are valuable.

[Guy Peleg]

Jim,

An rx8640 would typically be slower comparing
to an rx6600. What you have witnessed is
normal and expected. Memory latency on
an rx660 is much lower comparing to an
rx8640. Actions you have taken so far,
affinitzing processes to CPU and devices
are exactly the way to deal with these
limitations. Remote memory access (out
side of the cell) would always be slower
comparing to local cell access.

Taking all this into consideration, what
exactly is it that you are asking?

Guy Peleg
Maklee Engineering
www.maklee.com

[Hoff]

Ok; your reply got in before my previous post.

And in addition to the other bits I mentioned previously, also look for the memory lock granularity with that shared memory; for lock collisions.

If you've not already started the thought processes involved here, look for ways to shard this application, too. Whether that's grouping or fragmenting the data or grouping or fragmenting the locks, or otherwise.

The speeds and feeds on the rx3600 and rx6600 series boxes are usually faster than the older cellular boxes. (Typical rx3600 memory latency should be lower than that of the rx8640 series, for instance.) And I'd presume that any future Integrity boxes will do better here, too. If you're running out of headroom here and will be looking for an upgrade, you might want to see what HP is willing to discuss about its product line plans.

And gain a baseline and start tweaking and tuning, if you've not already done so.