Memory Information

[Pattabhi]

Hi All,

I have a feeling that there is a memory failure on my HP Box. But i am a bit confused

Glance says

Total VM : 3.01gb Sys Mem : 720.0mb User Mem: 1.07gb Phys Mem: 3.99gb
Active VM: 2.95gb Buf Cache: 1.92gb Free Mem: 308.3mb

Syslog Says
# grep -i physical /var/adm/syslog/syslog.log
May 30 10:27:08 benz vmunix: physical page size = 4096 bytes, logical page size = 4096 bytes
May 30 10:27:08 benz vmunix: Physical: 4182800 Kbytes, lockable: 2967904 Kbytes, available: 3456868 Kbytes

and

# echo "selclass qualifier memory;info;wait;infolog"|cstm | grep "Total Configured Memory"

Total Configured Memory: 4096 MB

echo "selclass qualifier memory;info;wait;infolog" | cstm > /tmp/meminfo says


DIMM Location Size(MB) DIMM Location Size(MB)
-------------------- -------- -------------------- --------
DIMM 0A 1024 DIMM 0B 1024
DIMM 1A 1024 DIMM 1B 1024
DIMM 2A ---- DIMM 2B ----
DIMM 3A ---- DIMM 3B ----
DIMM 4A ---- DIMM 4B ----
DIMM 5A ---- DIMM 5B ----




But Sam Says

Physical Memory: 4088.4 MB
Real Memory:
Active: 1134600.1 KB
Total: 1161328.1 KB
Virtual Memory:
Active: 3261604.5 KB
Total: 3331708.0 KB
Free Memory Pages: 55619 at 4 KB/page
Swap Space:
Avail: 4096 MB
Used: 1608 MB


What do i infer out of these outputs??? But my machine's performance has come down drastically...

Regards,
Pattabhi Raman

[A. Clay Stephenson]

Glance says

Total VM : 3.01gb Sys Mem : 720.0mb User Mem: 1.07gb Phys Mem: 3.99gb
Active VM: 2.95gb Buf Cache: 1.92gb Free Mem: 308.3mb

You have 4GiB of physical memory but there are really not enough data here to identify your performance problem.

One of the things that jumps out at me is the BuF Cache: 1.92gb. This implies that your dbc_max_pct is very likely at the default 50% value --- which is almost always bad. Because it is at the default, there is a strong possability that all the kernel tunables are also at the default value and thus the box could do with some tuning. I would first reduce dbc_max_pct to no more than about 25% and dbc_min_pct to about 5%.

You really need to pay attention to your pageout rate. You can use Glance or simply monitor the po column of vmstat. Ideally, you want that value to be zero; certainly no more than about 15 or so for any length of time. If you are swapping (more accurately paging out) then that is the first thing to address because swapping tends to overwhelm everything else.

[Pattabhi]

thanks for the input.
The reason i feel it might be a memory issue is that the performance issue has shown up only in the last 2 days... and also last week my machine went down abruptly due to airco issue as the temp in the DC went up. So this thought.

Other wise my machine used to run like a gem.

What do you infer from teh 3rd and sam's output?

Regards,
Pattabhi Raman

[Pattabhi]

Alos please note that i havent made any changes to the system for a long time now.

[A. Clay Stephenson]

Since you haven't bothered to post any of the page-out metrics, it's not possible to make any informed responses. I suppose that you will need to go with your feelings that this is a memory problem. I am not going to use "The Force" for this diagnosis. You might have a single runaway process that is your fundamental problem; it might indeed be memory pressure but the best way to determine that is by examining the pageout metrics. Of course, you are free to do as you choose.

[Marcel Burggraeve]

Not a performance expert so I can't really comment on the problem your facing.

However, I can make some comments about the remark that nothing has changed.
Basically, a system on which nothing changes has been shutdown and isn't running.
Any system which is up and running and especially in production will be subject to changes almost constantly.
It might not be the administrator loading software and/or patches but there's other things to think about.
- More data is being stored
- Databases grow
- Users might be added
- Increase of memory use due to new users
- logfiles are increasing in size
- Etc etc
Because of that you might be reaching a certain treshold causing the system to have a performance drop 'all of a sudden'
For what it's worth, it might even be a bug which kicks in at a certain treshold and it might even have been fixed already in recent patches.

I would advise to post some metrics like A. Clay Stephenson wrote as well.
There's loads of people here willing and able to help but to do that more information from your side is needed as well.

[Pattabhi]

Here is the glance output that i have managed to capture every 5 seconds..

Page Faults 1 747 0.2 3.3 120.0
Page In 0 161 0.0 0.7 12.6
Page Out 0 0 0.0 0.0 0.0
KB Paged In 0kb 2.0mb 0.0 9.3 294.6
KB Paged Out 0kb 0kb 0.0 0.0 0.0
Page 1 of 1
Page Faults 0 747 0.0 3.2 120.0
Page In 0 161 0.0 0.6 12.6
Page Out 0 0 0.0 0.0 0.0
KB Paged In 0kb 2.0mb 0.0 9.0 294.6
KB Paged Out 0kb 0kb 0.0 0.0 0.0
Page 1 of 1
Page Faults 0 747 0.0 3.1 120.0
Page In 0 161 0.0 0.6 12.6
Page Out 0 0 0.0 0.0 0.0
KB Paged In 0kb 2.0mb 0.0 8.8 294.6
KB Paged Out 0kb 0kb 0.0 0.0 0.0
Page 1 of 1
Page Faults 0 747 0.0 3.1 120.0
Page In 0 161 0.0 0.6 12.6
Page Out 0 0 0.0 0.0 0.0
KB Paged In 0kb 2.0mb 0.0 8.6 294.6
KB Paged Out 0kb 0kb 0.0 0.0 0.0
Page 1 of 1
Page Faults 0 747 0.0 3.0 120.0
Page In 0 161 0.0 0.6 12.6
Page Out 0 0 0.0 0.0 0.0
KB Paged In 0kb 2.0mb 0.0 8.4 294.6
KB Paged Out 0kb 0kb 0.0 0.0 0.0
Page 1 of 1
Page Faults 0 747 0.0 2.9 120.0
Page In 0 161 0.0 0.6 12.6
Page Out 0 0 0.0 0.0 0.0
KB Paged In 0kb 2.0mb 0.0 8.3 294.6
KB Paged Out 0kb 0kb 0.0 0.0 0.0
Page 1 of 1
Page Faults 82 829 15.7 3.2 120.0
Page In 27 188 5.1 0.7 12.6
Page Out 0 0 0.0 0.0 0.0
KB Paged In 0kb 2.0mb 0.0 8.1 294.6
KB Paged Out 0kb 0kb 0.0 0.0 0.0
Page 1 of 1
Page Faults 0 829 0.0 3.1 120.0
Page In 0 188 0.0 0.7 12.6
Page Out 0 0 0.0 0.0 0.0
KB Paged In 0kb 2.0mb 0.0 7.9 294.6
KB Paged Out 0kb 0kb 0.0 0.0 0.0
Page 1 of 1
Page Faults 0 829 0.0 3.1 120.0
Page In 0 188 0.0 0.7 12.6
Page Out 0 0 0.0 0.0 0.0
KB Paged In 0kb 2.0mb 0.0 7.8 294.6
KB Paged Out 0kb 0kb 0.0 0.0 0.0
Page 1 of 1
Page Faults 5 834 0.9 3.0 120.0
Page In 0 188 0.0 0.6 12.6
Page Out 0 0 0.0 0.0 0.0
KB Paged In 0kb 2.0mb 0.0 7.6 294.6
KB Paged Out 0kb 0kb 0.0 0.0 0.0
Page 1 of 1
Page Faults 0 834 0.0 3.0 120.0
Page In 0 188 0.0 0.6 12.6
Page Out 0 0 0.0 0.0 0.0
KB Paged In 0kb 2.0mb 0.0 7.5 294.6
KB Paged Out 0kb 0kb 0.0 0.0 0.0
Page 1 of 1
Page Faults 38 872 7.3 3.1 120.0
Page In 0 188 0.0 0.6 12.6
Page Out 0 0 0.0 0.0 0.0
KB Paged In 0kb 2.0mb 0.0 7.4 294.6
KB Paged Out 0kb 0kb 0.0 0.0 0.0
Page 1 of 1
Page Faults 0 872 0.0 3.0 120.0
Page In 0 188 0.0 0.6 12.6
Page Out 0 0 0.0 0.0 0.0
KB Paged In 0kb 2.0mb 0.0 7.2 294.6
KB Paged Out 0kb 0kb 0.0 0.0 0.0
Page 1 of 1
Page Faults 0 872 0.0 2.9 120.0
Page In 0 188 0.0 0.6 12.6
Page Out 0 0 0.0 0.0 0.0
KB Paged In 0kb 2.0mb 0.0 7.1 294.6
KB Paged Out 0kb 0kb 0.0 0.0 0.0
Page 1 of 1
Page Faults 0 872 0.0 2.9 120.0
Page In 0 188 0.0 0.6 12.6
Page Out 0 0 0.0 0.0 0.0
KB Paged In 0kb 2.0mb 0.0 7.0 294.6
KB Paged Out 0kb 0kb 0.0 0.0 0.0
Page 1 of 1
Page Faults 0 872 0.0 2.8 120.0
Page In 0 188 0.0 0.6 12.6
Page Out 0 0 0.0 0.0 0.0
KB Paged In 0kb 2.0mb 0.0 6.9 294.6
KB Paged Out 0kb 0kb 0.0 0.0 0.0
Page 1 of 1
Page Faults 0 872 0.0 2.8 120.0
Page In 0 188 0.0 0.6 12.6
Page Out 0 0 0.0 0.0 0.0
KB Paged In 0kb 2.0mb 0.0 6.7 294.6
KB Paged Out 0kb 0kb 0.0 0.0 0.0
Page 1 of 1
Page Faults 0 872 0.0 2.8 120.0
Page In 0 188 0.0 0.6 12.6
Page Out 0 0 0.0 0.0 0.0
KB Paged In 0kb 2.0mb 0.0 6.7 294.6
KB Paged Out 0kb 0kb 0.0 0.0 0.0
Page 1 of 1
Page Faults 0 872 0.0 2.7 120.0
Page In 0 188 0.0 0.5 12.6
Page Out 0 0 0.0 0.0 0.0
KB Paged In 0kb 2.0mb 0.0 6.5 294.6
KB Paged Out 0kb 0kb 0.0 0.0 0.0
Page 1 of 1
Page Faults 0 872 0.0 2.7 120.0
Page In 0 188 0.0 0.5 12.6
Page Out 0 0 0.0 0.0 0.0
KB Paged In 0kb 2.0mb 0.0 6.4 294.6
KB Paged Out 0kb 0kb 0.0 0.0 0.0
Page 1 of 1
Page Faults 0 872 0.0 2.6 120.0
Page In 0 188 0.0 0.5 12.6
Page Out 0 0 0.0 0.0 0.0
KB Paged In 0kb 2.0mb 0.0 6.3 294.6
KB Paged Out 0kb 0kb 0.0 0.0 0.0
Page 1 of 1
Page Faults 0 872 0.0 2.6 120.0
Page In 0 188 0.0 0.5 12.6
Page Out 0 0 0.0 0.0 0.0
KB Paged In 0kb 2.0mb 0.0 6.2 294.6
KB Paged Out 0kb 0kb 0.0 0.0 0.0
Page 1 of 1
Page Faults 0 872 0.0 2.5 120.0
Page In 0 188 0.0 0.5 12.6
Page Out 0 0 0.0 0.0 0.0
KB Paged In 0kb 2.0mb 0.0 6.1 294.6
KB Paged Out 0kb 0kb 0.0 0.0 0.0
Page 1 of 1
Page Faults 0 872 0.0 2.5 120.0
Page In 0 188 0.0 0.5 12.6
Page Out 0 0 0.0 0.0 0.0
KB Paged In 0kb 2.0mb 0.0 6.1 294.6
KB Paged Out 0kb 0kb 0.0 0.0 0.0
Page 1 of 1
Page Faults 0 872 0.0 2.5 120.0
Page In 0 188 0.0 0.5 12.6
Page Out 0 0 0.0 0.0 0.0
KB Paged In 0kb 2.0mb 0.0 6.0 294.6
KB Paged Out 0kb 0kb 0.0 0.0 0.0

I did follow Clay's advice and did tune the recommeded kernel parameters and i also increased the swap by 4Gb. there are no DB's running on the box.

Hope this helps. Is there a way i can run any command and check if any memory module has gone corrupted.

[Andrew Merritt_2]

Hi Pattabhi,
To check for memory errors, run the Memory Info tool in STM, and look at the output. That should show if there are any memory modules failing. Are there any entries in the PDT? You should also get EMS events logged in /var/opt/resmon/log/event.log, and a warning in syslog, if memory is being deallocated.

From what you've presented so far, I don't see any signs of memory failure.

Andrew

[Pattabhi]

Andrew...

The requested output is as follows

-- Current Device Status Log for IPF_MEMORY on path memory --

The following information was archived during the last hardware scan:

Product : IPF_MEMORY
Qualifier : 1004
Hardware Path : memory
Device Type : Memory
Device Qualifier : Memory

Most Recent Tool Run : Information
Tool Exit Status : Success
Current Status : Success

Recommended Action : None

Installed tools:
Diagnostic : None
Verifier : None
Exerciser : memory
Information : memory_ia64
Expert Tool : None
Firmware Update : None

=================================================================================================

-- Information Tool Log for IPF_MEMORY on path memory --

Log creation time: Thu May 31 14:03:24 2007

Hardware path: memory


Basic Memory Description

Module Type: MEMORY
Page Size: 4096 Bytes
Total Physical Memory: N/A
Total Configured Memory: 4096 MB
Total Deconfigured Memory: N/A

Memory Board Inventory

DIMM Location Size(MB) DIMM Location Size(MB)
-------------------- -------- -------------------- --------
DIMM 0A 1024 DIMM 0B 1024
DIMM 1A 1024 DIMM 1B 1024
DIMM 2A ---- DIMM 2B ----
DIMM 3A ---- DIMM 3B ----
DIMM 4A ---- DIMM 4B ----
DIMM 5A ---- DIMM 5B ----

===========================================================================

Memory Error Log Summary

The memory error log is empty.

Page Deallocation Table (PDT)

The Page Deallocation Table is empty.

PDT Entries Used: 0

[Andrew Merritt_2]

That's pretty self-explanatory, but I'd say there's no sign of any hardware memory problems there. I suggest you follow up on Clay's suggestions, to try to characterize the nature of the performance problem and work from there.

It's possible the overheating may have damaged something, I suppose, but it doesn't appear to have caused memory failures.

Andrew

[Pattabhi]

Thanks Andrew, I shall follow Clays advice and see if it helps.