Dear Almond
When considering the performance of any system it is important to determine a
baseline of what is acceptable. How does the system perform when there is no
load from applications or users? What are the systems resources in terms of
memory , both physical and virtual? How many processors does the system have
and what is the speed and RISC level? What is the layout of the data ? What
are the key kernel parameters set to and how are those resources being
utilized? What are the utilities to measure these ?
Memory Resources
HP-UX utilizes both physical memory , RAM and virtual memory, referred to as
swap. There are three resources that can be used to determine the amount
of RAM: syslog.log,dmesg, and adb (absolute de-bugger) .The information dmesg
reports comes from /var/adm/syslog/syslog.log ,while using dmesg is
convienient, if the system has logged too many errors recently, the memory
information may not be available.
Insufficient memory resources are a major cause of performance problems and
should be the first area to check .
The memory information from dmesg is at the bottom of the output .
example:
Memory Information :
physical page size =4096 bytes, logical page size= 4096 bytes
Physical: 524288 Kbytes, locakble: 380880 Kbytes , available: 439312
Using adb reads the memory from a more reliable source, the kernel.
To determine the physical memory (RAM) using adb:
for HP-UX 10.X
example:
echo physmem/D | adb -k /stand/vmunix /dev/kmem
physmem:
physmem: 24576
for HP-UX 11.X systems running on 32 bit architecture:
example:
echo phys_mem_pages/D | adb /stand/vmunix /dev/kmem
phys_mem_pages:
phys_mem_pages: 24576
for HP-UX 11.X systems running on 64 bit architecture:
example:
echo phys_mem_pages/D | adb64 -k /stand/vmunix /dev/mem
phys_mem_pages:
phys_mem_pages: 262144
The results of these commands are in 4 Kb memory pages, to determine the size
in bytes multiply by 4096 .
To fully utilize all of the RAM on a system there must be a sufficient amount
of virtual memory to accomodate all processes . The HP recommendation is at
that virtual memory be at least eaqual to physical memory plus application
size. This is outlined in the System Adminstration Tasks Manual .
To determine virtual memory configuration run the following command :
#swapinfo –tam
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 1024 0 1024 0 1 /dev/vg00/lvol1
reserve 184 -184
memory 372 96 276 26
total 1396 280 1116 20
The key areas to monitor are reserve , memory and total . For a process
to spawn it needs a sufficient amount of virtual memory to be placed in
reserve. There should be a sufficient amount of free device swap to open any
processes that may be spawned during the course of operations. By subtracting
thereserve from the device total you can determine this value.
If there is an insufficient amount available ( typically from device swap)
you will receive an error : cannot fork : not enough virtual memory. If
this error is received , you will need to allocate more device swap. This
should be configured on a disk with no other swap partitions, and ideally of
the same size and priority of existing swap logical volumes to enable
interleaving.
Refer to the Application Note KBAN00000218
Configuring Device Swap for details on the procedure.
The memory line is enabled with the kernel parameter swapmem_on set to 1 . This
allows a percentage of RAM to be allocated for pseudo-swap. This is the default
and should be used unless the amount of lockable memory exceeds 25% of RAM.
You can determine the amount of lockable memory by running the command:
example:
echo total_lockable_mem/D | adb -k /stand/vmunix /dev/mem
total_lockable_mem:
total_lockable_mem: 185280
This will return the amount in Kbytes of lockable memory in use.
If pseudo-swap is disabled by setting swapmem_on to 0 , there will typically be
a need to increase the amount of device swap in the system to accommodate
paging and reserve area.
If the total under PCT USED is 90 or greater it is recommended to increase the
amount of device swap.
After physical and virtual memory is determined , we need to determine how much
buffer cache has been configured and how much is being used. By default the
system will use dynamic buffer cache . The kernel will show buf pages and nbuf
set to 0 in SAM. The parameters that govern the size of the dynamic buffer
cache are dbc_min_pct and dbc_max_pct , these define the minimum
and maximum percentage of RAM allocated. The default values are 5% mimimum and
50% maximum .
On systems with small amounts of RAM these values may be useful for
dedicated applications. Since the introduction of HP-UX 11.0 the amount of RAM
a system can have has increased from 3.75Gb to our newest systems with up to
256Gb. Keeping the default values for systems with a large amount of RAM can
have a negative impact on performance, due to the time the lower level routines
that check on free memory in the cache take.
To monitor the use of the buffer cache run the following command :
sar –b 5 30
You will see output similar to :
bread/s lread/s %rcache bwrit/s lwrit/s %wcache pread/s pwrit/s
0 95 100 1 2 54 0 0
Ideally we want to see a %wcache of 95 or greater. If the system consistenly
shows %wcache less than 75 it would be advisable to lower the value of
dbc_max_pct. In 32 bit architecture, the buffer cache resides in quadrant 3,
limiting the maximum size to 1 Gb. Typically values less than 300Mb are
preferable.
Memory for applications
For applications to have a sufficient amount of space for text, data and stack
in memory the kernel has to be tuned. The total size for text, data and stack
for 32 bit systems using EXEC_MAGIC is in quadrant 1 and 2, and is 2Gb less the
size of the Uarea . These are represented by the kernel parameters maxtsize,
maxdsiz and maxssiz . Under SHMEM_MAGIC the total size is limited to 1Gb in
quadrant 1.
If these parameters are undersized the system will error. Insufficient maxdsiz
will return “out of memory “ and insufficient maxsiz will
return “stack growth failure”.
The last configurable area of memory to check is shared memory . Any
application running within the 32 bit domain will have a limit of 1.75Gb total
for shared memory for EXEC_MAGIC and 2.75Gb using SHMEM_MAGIC. Individual
processes cannot cross quadrant boundaries , so the largest shmmax can be for
32 bit is 1Gb.
It is important to determine if the application is running 32 bit or 64 bit
when troubleshooting 64 bit systems.
This can be done with the file command :
example :
file /stand/vmunix
/stand/vmunix: ELF-64 executable object file - PA-RISC 2.0 (LP64)
PA-RISC versions under 2.0 are 32 bit
For an overview on shared memory for 32 bit systems refer to the Application
Note RCMEMKBAN00000027 Understanding Shared Memory on PA RISC Systems .
The kernel parameter shmmax determines the size of the shared memory
region. SAM will not allow this to be configured greater than 1 quadrant , or
1Gb even on 64 bit systems. If a larger shmmax value is needed for 64 bit
systems it has to be done using a manual kernel build.
In a 64 bit system 32 bit applications will only address the 32 bit shared
memory region,64 bit applications will only address the 64 bit regions.
Example of a creating a kernel with shmmax at 2Gb:
cd /stand/build
/usr/lbin/sysadm/system_prep -v -s system
kmtune -s shmmax= 2147483648 -S /stand/build/system
/usr/sbin/mk_kernel -s ./system
mv /stand/system /stand/system.prev
mv /stand/build/system /stand/system
kmupdate
shutdown -ry 0
To determine shared memory allocation, use ipcs this utility is used to
report status of interprocess communication facilities. Run the following
command:
ipcs –mob
You will see an output similar to this :
ipcs -mob
IPC status from /dev/kmem as of Tue Apr 17 09:29:33 2001
T ID KEY MODE OWNER GROUP NATTCH SEGSZ
Shared Memory:
m 0 0x411c0359 --rw-rw-rw- root root 0 348
m 1 0x4e0c0002 --rw-rw-rw- root root 1 61760
m 2 0x412006c9 --rw-rw-rw- root root 1 8192
m 3 0x301c3445 --rw-rw-rw- root root 3 1048576
m 4004 0x0c6629c9 --rw-r----- root root 2 7235252
m 5 0x06347849 --rw-rw-rw- root root 1 77384
m 206 0x4918190d --rw-r--rw- root root 0 22908
m 6607 0x431c52bc --rw-rw-rw- daemon daemon 1 5767168
The two fields of the most interest are NATTCH and SEGSZ.
NATTCH -The number of processes attached to the associated shared
memory segment. Look for those that are 0, they indicate processes who have not
released their shared memory segment.
If there are multiple segments showing with an NATTACH of zero , especially if
they are owned by a database, this can be an indication that the segments are
not being efficiently released . This is due to the program not calling
detachreg . These segments can be removed using ipcrm -m shmid.
Note : Even though there is no process attached to the segment , the data
structure is still intact. The shared memory segment and data structure
associated with it are destroyed by executing this command.
SEGSZ The size of the associated shared memory segment in bytes. The
total of SEGSZ for a 32 bit system using EXEC_MAGIC cannot exeed 1879048192
bytes or 1.75Gb, or 2952790016 bytes or 2.75Gb for SHMEM_MAGIC.
