file-cache-create: Error mmap()ing file /index.html (Not enough space)

[mahsuma]

Friends,

I got a chance to go through the Netscape web server error logs. I found the below following message in the nes error logs-

[07/Apr/2010:15:36:41] failure (14607): file-cache-create: Error mmap()ing file /opt/nes/docs/index.html (Not enough space)
[07/Apr/2010:15:37:24] failure (14607): file-cache-create: Error mmap()ing file /opt/nes/docs/index.html (Not enough space)

I also searched for the information for the above error in ITRC and found that the
above error for mmap() point to insufficient data space (the 'maxdsiz' kernel parameters) and seems to appear in many platforms of HP-UX machines.

This could be the result of the image cache filling up on the server and we can adjust the size of the cache in obj.conf:
Eg-
Init fn=cache-init cache-size=512 mmap-max=10000
Where, cache-size is the number of entries, and should be greater than the number of URLs on your server; the default is 512.
mmap-max is the number of 1024 bytes chunks to set aside; the default is 10000 (or 10MB).

I have increased the mmap-max space (30 MB) and cache-size (1024 entries) in the obj.conf as estimated.

cp-co1# more obj.conf | grep 'mmap'
Init fn="cache-init" cache-size="1024" mmap-max="30000"
cp-co1#

The filesystem under Netscape (/opt/nes) looks good before and after the config changes.

Still I'm getting the same mmap() errors.,

Please feel free to suggest me ..

Thanks in advance :)

[James R. Ferguson]

Hi:

Details like the processor architecture you are running (PA-RISC or Itanium) and the bitness (32 or 64-bit) of the application (I suspect 32-bit) would be helpful.

Regards!

...JRF...

[mahsuma]

The details are,

HP-UX cp-co1 B.10.20 E 9000/891 349339221 8-user license

[James R. Ferguson]

Hi (again):

An old 10.20 release...I should have known.

It is possible that you need to increase the kernel's 'shmhmax' as this tunable constrains 'mmap()'.

You should probably also *decrease* the image cache size. After all, it may be better to have sluggish performance than none at all.

Regards!

...JRF...

[Dennis Handly]

Do you have enough swap when it fails? swapinfo -tam

[Dennis Handly]

>JRF: It is possible that you need to increase the kernel's 'shmhmax' as this tunable constrains mmap(2).

(I assume that is a typo for shmmax(5).)

shmmax(5) doesn't seem to indicate this. For MAP_PRIVATE it would be limited to maxdsiz but it seems Netscape is mapping in files, so you may be running out of shared memory virtual addresses.

[mahsuma]

cp-co1# swapinfo -tam
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 512 0 512 0% 0 - 1 /dev/vg00/lvol2
dev 528 0 528 0% 0 - 1 /dev/vg00/swap
reserve - 427 -427
memory 727 217 510 30%
total 1767 644 1123 36% - 0 -
cp-co1#

[mahsuma]

well, the index.html is pointing to web url using Java Script.

window.location="https://cp-co1:4444"

which is creating above errors.

[Dennis Handly]

>total 1767 644 1123 36%

You only have 1.1 Gb now, is it busier when it fails?

[mahsuma]

Not sure, the more increase in number of users accessing/requesting the application is creating the above mmap()ing errors (during online hours). Once the user provide the credentials, the user will get authenticated and redirect to call the program (/default.sph/atrprod.exe). You can view this in the below netscape webserver access log:

10.1.214.60 - - [06/Apr/2010:07:12:58 -0600] "GET / HTTP/1.1" 200 111

10.30.52.58 - - [06/Apr/2010:07:13:55 -0600] "GET /default.sph/atr_30.exe?FNC=GoToTestWebClient__Aindex_html HTTP/1.1" 200 -

10.26.86.42 - - [06/Apr/2010:07:16:50 -0600] "GET /default.sph/atr_30.exe?FNC=GoToTestWebClient__Aindex_html HTTP/1.1" 200 -

10.5.33.33 - - [06/Apr/2010:07:18:29 -0600] "GET /default.sph/atr_30.exe?FNC=GoToTestWebClient__Aindex_html HTTP/1.1" 200 -

151.119.20.113 - - [06/Apr/2010:07:18:59 -0600] "GET / HTTP/1.0" 200 314

10.2.5.34 - - [06/Apr/2010:07:19:59 -0600] "GET /default.sph/atr_30.exe?FNC=GoToTestWebClient__Aindex_html HTTP/1.1" 200 -

10.3.120.97 - - [06/Apr/2010:07:19:50 -0600] "GET / HTTP/1.1" 200 111

10.2.4.112 - - [06/Apr/2010:07:20:52 -0600] "GET /default.sph/atr_30.exe?FNC=GoToTestWebClient__Aindex_html HTTP/1.1" 200 -

10.2.43.201 - - [06/Apr/2010:07:21:53 -0600] "GET /default.sph/atr_30.exe?FNC=GoToTestWebClient__Aindex_html HTTP/1.1" 200 -

10.29.248.27 - - [06/Apr/2010:07:24:12 -0600] "GET /default.sph/atr_30.exe?FNC=GoToTestWebClient__Aindex_html HTTP/1.1" 200 -


I have looked further into the status code and bytes transferred fields in the access log which can provide more information on the errors occurred.

If we look into the bytes transferred field in the above access log, it seems like the failures are happening with the requests which exceeds 111 bytes.



08:27:37 -0600] "GET / HTTP/1.1" 200 111

07:18:59 -0600] "GET / HTTP/1.0" 200 314



Not sure whether this is the reason which consume whole cache memory and load ATR-WEB slowly.



On other hand, the filesystem under Netscape (/opt/nes) and swapspace (1.1 GB free now) also looks good.



/opt/nes (/dev/vg00/lv_netscapesrvr) : 498727 total allocated Kb

199324 free allocated Kb

299403 used allocated Kb

60 % allocation used



cp-co1# swapinfo -t

Kb Kb Kb PCT START/ Kb

TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME

dev 524288 45316 478972 9% 0 - 1 /dev/vg00/lvol2

dev 540672 43536 497136 8% 0 - 1 /dev/vg00/swap

reserve - 362012 -362012

memory 744544 222356 522188 30%

total 1809504 673220 1136284 37% - 0 -

cp-co1#



Still I'm seeing the same mmap() errors in the Netscape webserver logs.

Could you please suggest if any kernel parameters (memory) needs to be tuned, so that I can get hold of our OS supoort team and get them updated.

Please feel free to suggest your ideas to investigate further on above errors happening.

Thanks all for your cooperation :)

[Dennis Handly]

>I'm seeing the same mmap() errors in the Netscape webserver logs.
>Could you please suggest if any kernel parameters (memory) needs to be tuned

I think you have used up all of your shared memory and have too many/large mapped files or shared memory segments, 2Gb.
What does "ipcs -ma" show?

[mahsuma]

cp-co1# ipcs -ma
IPC status from /dev/kmem as of Mon Apr 19 09:37:48 2010
T ID KEY MODE OWNER GROUP CREATOR CGROUP NATTCH SEGSZ CPID LPID ATIME DTIME CTIME
Shared Memory:
m 0 0x41180617 --rw-rw-rw- root root root root 0 348 1502 1502 1:07:46 1:07:46 1:07:40
m 1 0x4e040002 --rw-rw-rw- root root root root 1 31008 1502 1502 1:07:44 1:07:46 1:07:40
m 2 0x411c588f --rw-rw-rw- root root root root 1 8192 1502 1514 1:07:44 1:07:40 1:07:40
m 6147 0x0c6629c9 --rw-r----- root root root root 2 10988048 2092 2045 1:10:28 no-entry 1:10:25
m 4 0x06347849 --rw-rw-rw- root root root root 1 77384 2092 2102 1:10:31 1:10:26 1:10:26
m 2053 0xffffffff --rw-r--rw- root root root root 0 22908 2045 2045 1:10:29 1:10:29 1:10:29
m 20486 0x52574801 --rw-rw---- informix informix informix informix 16 235315200 2185 10846 9:30:17 9:30:18 1:10:53
m 51207 0x40601080 --rw-rw-rw- mqm mqm mqm mqm 5 1036 2336 13082 6:37:03 6:37:13 1:11:47
m 2056 0x00602401 --rw-rw---- mqm mqm mqm mqm 4 962172 2341 13082 6:37:03 6:37:13 1:11:49
m 9 0x00602402 --rw-rw---- mqm mqm mqm mqm 4 1089248 2341 13082 6:37:03 6:37:13 1:11:49
m 10 0x00602403 --rw-rw---- mqm mqm mqm mqm 3 152864 2341 13082 6:37:03 6:37:13 1:11:49
m 1035 0x00603c02 --rw-rw-rw- mqm mqm mqm mqm 3 1089252 2341 13082 6:37:03 6:37:13 1:11:49
m 12 0x00603c01 --rw-rw-rw- mqm mqm mqm mqm 3 962176 2341 13082 6:37:03 6:37:13 1:11:49
m 13 0x00603c03 --rw-rw-rw- mqm mqm mqm mqm 3 1001540 2341 13082 6:37:03 6:37:13 1:11:49
m 14 0x00603c04 --rw-rw-rw- mqm mqm mqm mqm 1 65540 2341 13076 6:37:03 6:37:03 1:11:49
m 15 0x00603c05 --rw-rw-rw- mqm mqm mqm mqm 2 5744 2341 13082 6:37:03 6:37:13 1:11:49
m 16 0x00602404 --rw-rw---- mqm mqm mqm mqm 3 132608 2341 13082 6:37:03 6:37:03 1:11:50
m 17 0x00602405 --rw-rw---- mqm mqm mqm mqm 3 67072 2341 13082 6:37:03 6:37:03 1:11:50
m 18 0x00602406 --rw-rw---- mqm mqm mqm mqm 3 132608 2341 13082 6:37:03 6:37:03 1:11:50
m 19 0x00602407 --rw-rw---- mqm mqm mqm mqm 3 2098580 2341 13082 6:37:03 6:37:03 1:11:50
m 20 0x00602408 --rw-rw---- mqm mqm mqm mqm 3 132608 2341 13082 6:37:03 6:37:03 1:11:50
m 21 0x00602409 --rw-rw---- mqm mqm mqm mqm 4 263680 2341 13082 6:37:03 6:37:03 1:11:50
m 3094 0x00603c06 --rw-rw-rw- mqm mqm mqm mqm 1 207104 2336 13076 6:37:03 6:37:03 1:11:52
m 2071 0x00600f03 --rw-rw-rw- mqm mqm mqm mqm 1 962172 2353 2353 1:11:55 no-entry 1:11:55
m 1048 0x00600f04 --rw-rw-rw- mqm mqm mqm mqm 1 1089248 2353 2353 1:11:55 no-entry 1:11:55
m 38937 0x00000000 --rw------- root root root root 2 1232 2556 2443 20:01:16 no-entry 20:01:16
cp-co1#

[mahsuma]

cp-co1# ipcs -ma | awk '{print$10}'
09:48:57
SEGSZ

348
31008
8192
10988048
77384
22908
235315200
1036
962172
1089248
152864
1089252
962176
1001540
65540
5744
132608
67072
132608
2098580
132608
263680
207104
962172
1089248
1232
cp-co1#

[Dennis Handly]

Your total is 256 Mb: 256,859,572

I don't know how many mapped files you have.
I don't know if tusc would work on 10.20 or if you could use pstat_getprocvm(2) to total your shared virtual address space usage.

[mahsuma]

are there any other possible causes which will create above errors?

[Dennis Handly]

>are there any other possible causes which will create above errors?

1) Out of address space.
2) Fragmented address space
3) Out of swap for anonymous maps