Shared memeory.

[Juan Gonzalez_2]

My backup software is complaining of running out of shared memory at certain times of the night. I have the times that my backup software complains and I am out of shared memory so process die. I know that the kernel parameters for shared memory are shmmni which is 512, shmseg which is 120, shmmax which is 1073741824, and shmem which is 1. My server is a N4000 with 4 CPU's and 8GB of memory. Which of these parameters memtioned above can I increase in value and not have a major impact on the operating system? The only thing we use this server for is to do backups.

[Mark Grant]

U guess you could increase shmmax fairly safely but how are you managing to eat up 8GB or RAM on a backup server?

[Rita C Workman]

Possibly you could give a little more details on your kernel parms...before some of us jump in with suggestions on making changes to them.

Also, what are you running?? Oracle? Informix? SAP??? Give us a little more info please...


Rgrds,
Rita

[Juan Gonzalez_2]

Mark we backup 150 Windows servers and 16 unix servers to this server and another N4000 server. We backup about 10 tetrabytes of data every day.

[Steven E. Protter · Accepted solution]

Lots of modern software has a huge hunger for shared memory.

ipcs

That displays shared memory segment use.

Note that shmmax can not be more than 25% of total memory, which is defined as swap plus memory. Any higher setting will be ignored.

Here is a good tuning doc. I'm also attaching a good script to collect data on your system and look for bottlenecks.

http://www1.itrc.hp.com/service/cki/search.do?category=c0&docType=Security&docType=Patch&docType=EngineerNotes&docType=BugReports&docType=Hardware&docType=ReferenceMaterials&docType=ThirdParty&searchString=UPERFKBAN00000726&search.y=8&search.x=28&mode=id&admit=-682735245+1066151323011+28353475&searchCrit=allwords

SEP

[A. Clay Stephenson]

You could increase shmmax and that should be the only one that really needs increasing BUT I suspect what is really wrong is that your backup software is 32-bit. You can get beyond the 1GB limitation by using the chatr command to enable SHMEM_MAGIC.

[Juan Gonzalez_2]

Rita,

Not running oracle on this server only backing up oracle. The only thing we are running on this server is veritas netbackup software version 4.5. I attached my current kernel parameters see attachment.

[Alzhy]

SHMMAX is one kernel paramter that you can "safely' max out IMHO. If it is a 32bit backup apps that you're using that uses intimate shared memory a lot then set it to 4 GB. Also increase your SHMSEG.
.
On a Solaris NetBackup server (8 LTO Ultriums, dual GigE - trunkked) that I built that serves out to about 200 servers (with a concurrency of save streams at about 100), I set up my SHM environment all at the above values...

[Juan Gonzalez_2]

I have 36 LTO HP Ultrim drives being run off of these two servers and I run eight data streams two each drive. The only time I have memory problems is on the weekend when we run backups of oracle databases that are only backed up on the weekend. I have seen all tape drives become active during peak backup times.

[Alzhy]

Whoa! THIRTY-SIX (36) LTO's to an N4000? Can the system bus support such a large number of HBA's serving your LTO's? How many LTO's do you have stringed up per HBA? Do you ever actually fully utilize each LTO's bandwidth while streaming to all drives?
.
If you're saying 8 streams per drive, that means 8x36=288 concurrent save streams. Consult the backup software's kernel tuning requirements if ever such a large number of streams/drives are at all supported.

[Juan Gonzalez_2]

Nelson I have the LTO tape drives in a tape san with brocades 2900's. I know what the recommendations are to use the LTO tape drive to its fullest potential but that is what management wanted here.

[Alzhy]

Juan,

Okay.. so the N4000 is it the lone media server or you do have several media servers and running SSO (shared storage option)? If it is the lone media server, do you mean to say you have one FC connection to this N4000 hope it's a 2Gbps one) and all 36 LTOs are presented to the N-Class using thos lone FC connection? Wow.. I'd say 4 LTO's is more than enough to hang off one 1 GBps FC link and keep the drives busy...
.
Anyways, I hope increasing your SHMMAX to 4 GB and increasing your shmseg to more than the number of concurrent savestreams would fix your problems... But it looks like your LTO's are not really getting their fill during backups if at all all 36 LTO's are served by this lone N-Class during backups...

[Juan Gonzalez_2]

Nelson I got two N4000 servers with the same amount of memory and the same amount of CPU's. There are also six HBA's that are 2GHz in each of the N4000 servers.