memory mapped files... virtual memory... physical memory

[Anthony deRito]

Hello, this is killing me... we are having problems with our application. I have a 10.20 T500 server running an OLTP application that does fast and short reads from memory mapped files. The code uses mmap(). The code also uses madvise() to indicate to the kernel that any objects mapped in this way will be accessed in a sequential manner. We do not use shared memory and all memory regeons are marked private. (No lectures please). During production hours, we never get to the point of swapping out. We do swap in pages of memory at a normal rate... not a problem. We process about 40,000 transactions per day. It is only during 1-2 minutes at 1:30AM that we have transactions the bomb off.

During this 1-2 minutes we are also performing filesystem I/O on the same device as where the master copy of the memory mapped files reside. (The I/O is actually what is making a new updated copy of files to be memory mapped at a later time.) I can see with sar -d that during this I/O (and exectly when we get failed transactions) the disk looks like this:

device %busy avque r+w/s blks/s avwait avserv

c6t5d2 44 14 25 1501 249 211
c6t5d2 100 32 58 3669 522 229
c6t5d2 99 68 54 3243 1026 175
c6t5d2 100 193 48 3175 3259 182
c6t5d2 65 131 33 1946 4141 151
c6t5d2 28 8 10 665 152 194
c6t5d2 62 1473 75 2005 2083 202
c6t5d2 99 2841 138 3250 9216 176
c6t5d2 99 1929 133 3416 16620 204
c6t5d2 99 650 126 3457 21455 230
c6t5d2 95 104 64 2588 3110 222

These are 10 second samples and totals just under 2 minutes... the same time we have transaction errors. There is nothing else going on with that path and I feel that the disk is just having a hard time keeping up.

My question is... how does this relate to how our applications reads these memory mapped files which are on the same LUN as where the I/O is taking place. If the memory mapped files are in memory, then why should the I/O be a concern. The output of vmstat shows no paging out and a large free list. There is no swapping out but we are paging in from disk.

Just for a racap in memory mapped files, from the HP memory management white paper:
"Memory-mapped files allow applications to map file data into a process's virtual address space. Once mapped, the process can directly manipulate the file data as a portion of memory. I/O to the files can be performed through direct loads and stores of memory instead of using read(2) and write(2), avoiding copying of data between user and system buffers.Memory-mapped files provide identical representation of in-memory and on-disk data."

Thanks,

Tony

[Stefan Farrelly · Accepted solution]

Hi Tony,

This is an interesting problem. I would pose a couple of things.
1. What priority for disk I/O do memory-mapped files have ? I would suspect lower than other processes carrying out direct I/O. This may be a possible cause of your transaction errors.
2. Your disk from your sar output is very very busy. Are you experiencing errors in dmesg (SCSI) on that path due to I/O queue overload ? If you check logtool from STM are there any errors at all for that disk ? (which may again be caused by overload.). If this disk is indeed overloaded I would expect to see some errors.
3. What type/model is that disk ? From this we can work out how fast it is and possibly use a faster disk instead.

[CHRIS_ANORUO]

You have answered part of you question bearing in mind that: "sar -d reports activity for block devices". Seeing that the disk is very busy due to avwait being higher than avserv. Kernel and the hardware coordinate a mapping of these virtual addresses, which must be mapped to physical addresses for compiled programs to execute. These must be available to the CPU by residing in physical memory. When this is not availble, you get the I/O errors.
Increase you maxdsiz to 1GB, nbuf and bufpages to zero and swapmem_on=1.

[Anthony deRito]

Hello Stefan,

>>1. What priority for disk I/O do memory-mapped files have ? I would suspect lower than other processes carrying out direct I/O. This may be a possible cause of your transaction errors.

I am not sure about the priority between these two types of tranasctions but that is an interesting thought.

>> 2. Your disk from your sar output is very very busy. Are you experiencing errors in dmesg (SCSI) on that path due to I/O queue overload ? If you check logtool from STM are there any errors at all for that disk ? (which may again be caused by overload.). If this disk is indeed overloaded I would expect to see some errors.

I am seeing error from dmesg. Thanks for that suggestion. I have attached the output below. Unfortunately the errors are not showing up in syslog after I applied the Sept path bundle so I do not know the timestamps of these entries.
The hardware is a Model 20 disk array and I use alternate links. The switch you see is a result of switching between the primary and the alternate device file for that LUN. I am not sure why it is switching... maybe because of excessive write errors. THIS IS NOT THE SAME ARRRAY WHERE THE DISK CONTENTION IS OCCURING but the two Model 20s are daisy chained and use the same path just different target IDs. Interesting..... do you think that these problems are related?

I am pushing a little over 1MB/sec on the LUN having the contention.

Thanks for your help.

Tony