If you have a few processes that try to use more than the available physical memory on the box, there is no substitue to physical memory.
These are some arguments to support this:
If you are not performing any disk I/O the operating system can shrink the buffer cache (down from anywhere as high as dbc_max_pct to as low as dbc_min_pct--you should not use nbuf or bufpages to fix the size, just set dbc_min_pct = dbc_max_pct), thus allowing the high memory utilisation processes to get more of their pages into memory (RSS--where the processes actual size is VSS. If RSS=VSS things are really hunky dorey! ; ).
(You are probably aware that:) The swap system needs to be able to allocate as much space on swap as all the processes' VSS (the swap space can include pseudo-swap, but if the O/S ever comes to need pseudo-swap--which it generally doesn't, the processes will be killed by the kernel!! It follows on (round about) that the amount of physical memory on the machine is NOT AN INDICATOR of how much swap you need. You MUST have as much swap (possibly including pseudo, so turn on pseudo swap in the kernel) as all the processes' VSS size (use: size /path/to/binary/binary) you want to load into memory. This is why SAP are sticking to their guns.
Your three(?) SAP processes will try to get as much of their VSS space into memory as they try to access ~quickly~. If they access data close together the vhand daemon can page out the pages not in use. If the SAP processes try to address pages all over their VSS far more pages will be brought back into main memory, thus greater memory pressure.
The problem is compounded when the SAP processes also try to do disk I/O since the filesystem requires buffer cache (which has been shrunk to accomodate the extra pages the processes are addressing). Obviously, the less buffer cache there is, the longer it will take to do disk I/O, and this memory is not available for the SAP processes.
Solution: "use less memory hogging processes, or buy more memory". This is the official line (don't quote me directly) taught on the HP-UX Tuning and Performance course H4262S. Therefore, it is important (and besides there is no benefit) to use device swap (I am also operating with FS swap, and am learning from my mistake), since the operating system DOES NOT REQUIRE BUFFER CACHE to write to the swap disk. You should make sure that the device swap is preferred over any filesystem swap (swap priority).
The SAP processes now have as much main memory and/or buffer cache as they can possibly get their hands on, because it is not being chewed up by swap I/O. If your SAP processes still thrash memory/swap, you REALLY WILL need to get more physical memory. This should be the foundation of your argument for investment in your development machine.
Aside: you can also check the I/Os per second and/or MB/s on your swap device. If you have more than 50 I/Os per second you could be in trouble, equally, if you have more than 5MB/s (on a standard SCSI device) to the swap disk you could be in trouble. You can have up to 10 swap devices (kernel parameter), they should ideally be the same size disks, and they should all be at the same priority. The operating system takes case of load balancing between the ('non-fs') swap disks.
I hope this has not been overkill, and you find this helpful, though it does not offer much solace.
"Don't shoot the messenger"--Anonymous.
