System Administration
cancel
Showing results for 
Search instead for 
Did you mean: 

RED Global swap space is nearly full

SOLVED
Go to solution
0rb1t0
Advisor

RED Global swap space is nearly full

Hello to all!

I read all internet sites but not undestand whe botllneck on server HPUX/

Glance say to me 14:15:15 RED Global swap space is nearly full

 

And i have:

 

Disk Util F FV V |100% 81% 100%
Mem Util S SU U |100% 100% 100%
Swap Util U UR R | 96% 96% 96%

 

Please help me

17 REPLIES
0rb1t0
Advisor

Re: RED Global swap space is nearly full

#> vmstat 1 5
procs memory page faults cpu
r b w avm free re at pi po fr de sr in sy cs us sy id
1 2 0 196842 7986 13 0 26 13 7 0 338 5416 233880 678 3 1 96
1 2 0 196842 7982 186 0 184 14 31 0 2017 6905 13859 1326 1 5 94
1 2 0 196842 8106 197 0 194 11 24 0 2035 6848 16564 1299 3 8 89
1 2 0 196842 8130 198 0 186 8 19 0 1998 7065 16232 1436 1 8 91
1 2 0 196842 8167 189 0 177 6 15 0 1926 6919 16247 1356 3 2 95

Patrick Wallek
Honored Contributor

Re: RED Global swap space is nearly full

What does the command:

 

# swapinfo -tam

 

show?

 

My guess is that you are, or have been, swapping to such an extent that your swap space is nearly used up.

0rb1t0
Advisor

Re: RED Global swap space is nearly full

#> swapinfo -tam
                 Mb          Mb           Mb   PCT      START/          Mb
TYPE          AVAIL        USED         FREE  USED       LIMIT     RESERVE  PRI  NAME
dev           32768       15005        17763   46%           0           -    1  /dev/vg00/lvol2
reserve           -       10414       -10414
memory       130496      124372         6124   95%
total        163264      149791        13473   92%       -           0    -

I dont know what happen, not new applications be added  to him

Patrick Wallek
Honored Contributor

Re: RED Global swap space is nearly full

You've got something that is using lots of memory and causing the system to page out.

 

Perhaps something opened some new share memory segments?  Check the output of 'ipcs -moab' to see shared memory.

 

To see what processes are using lots of memory you can do:

 

# UNIX95= ps -eo user,pid,ppid,vsz,sz,args | sort -nk 4

 

The 4th column will show the virtual memory size of the process and the list will be sorted in ascending order.

0rb1t0
Advisor

Re: RED Global swap space is nearly full

In this server a lot of process opened by oracle.

 

#> ipcs -moab
IPC status from /dev/kmem as of Fri Jun 19 23:05:31 2015
T         ID     KEY        MODE        OWNER     GROUP   CREATOR    CGROUP NATTCH      SEGSZ  CPID  LPID   ATIME    DTIME    CTIME 
Shared Memory:
m          0 0x410c185d --rw-rw-rw-      root      root      root      root      0        348  1341  1341  8:37:58  8:37:58  8:37:53
m          1 0x4e0c0002 --rw-rw-rw-      root      root      root      root      2      61760  1341 19728 10:50:19 11:07:39  8:37:53
m          2 0x410efccd --rw-rw-rw-      root      root      root      root      2       8192  1341 11059  1:33:09  1:33:09  8:37:53
m          3 0x410c40cb --rw-r--r--      root      root      root      root      3    2850096  2388  4202 23:05:30 23:05:30  8:38:46
m          4 0x490ded32 --rw-r--r--      root      root      root      root      1      22908  2388  3496 23:05:30 23:05:30  8:38:46
m          5 0x500c40ca --rw-r--r--      root   hpvmsys      root   hpvmsys      1   17115608  2394  3496 23:05:30 23:05:30  8:38:47
m     622598 0x00a5c581 --rw-------     sfmdb     users     sfmdb     users      4   11345920 14440 14485  2:10:27  2:10:41  2:10:27
m          7 0x5e0e822b --rw-------      root      root      root      root      1        512  3182  3182  8:40:06 no-entry  8:40:06
m          8 0x02fb07f1 --rw-------    hpsmdb     users    hpsmdb     users      1   10469376  3331  3335  8:40:07  8:40:07  8:40:07
m          9 0x06347849 --rw-rw-rw-      root      root      root      root      2      65544  3410  3421  8:40:15  8:40:12  8:40:10
m      32778 0x0c6629c9 --rw-r-----      root      root      root      root      3   18121528  3423 21075 16:16:41 16:17:29  8:40:12
m         11 0x010eb2ff --rw-rw-r--      root      root      root      root      1       4192  3426  3426  8:40:12  8:40:12  8:40:12
m         12 0xa90c5105 -----------      root      root      root      root      1          1  4003  4004  8:40:34 no-entry  8:40:34
m      65549 0x6c0ef197 --rw-------    zabbix     users    zabbix     users      6     995944 28067 28067 13:28:14 no-entry 13:28:14
m         14 0x010eb2d3 --rw-rw-r--      root      root      root      root      1       4192  4095  4095  8:40:40  8:40:40  8:40:40
m         15 0x010eb278 --rw-rw-r--      root      root      root      root      1       4192  4102  4102  8:40:41 no-entry  8:40:41
m         16 0x010eb2e8 --rw-rw-r--      root      root      root      root      1       4192  4104  4117  8:40:45  8:40:51  8:40:43
m      32785 0x010d8c05 --rw-rw-r--      root      root      root      root      1       4192  4123  4123  8:40:51 no-entry  8:40:51
m   41779218 0x3564e93c --rw-rw----    oracle  oinstall    oracle  oinstall     16 1593901056  8212 22012 23:04:36 23:04:36  5:25:32
m     458771 0x000001df --rw-------   doctest     dinst   doctest     dinst     10    6470528   479   479 17:37:11 no-entry 17:37:11
m     393236 0x00000000 D-rw-------   doctest     dinst   doctest     dinst     10       4096   479   479 17:37:19 no-entry 17:37:19
m    5308437 0x00000000 D-rw-------   doctest     dinst   doctest     dinst     10       4096   479   479 17:37:19 no-entry 17:37:19
m     196630 0x720f2ab9 --rw-------   doctest     dinst   doctest     dinst     16    2293760   479   509 17:37:24 no-entry 17:37:19
m     655383 0x00000000 --rw-r-----    oracle  oinstall    oracle  oinstall     79   50331648 14472 21866 23:03:51 23:04:11 15:39:41
m     360472 0x00000000 --rw-r-----    oracle  oinstall    oracle  oinstall     79 3539992576 14472 21866 23:03:51 23:04:11 15:39:41
m     294937 0x1faae8ec --rw-r-----    oracle  oinstall    oracle  oinstall     79      65536 14472 21866 23:03:51 23:04:11 15:39:41
m     589850 0x54397118 --rw-rw----    oracle  oinstall    oracle  oinstall     24 1291911168  1538 22028 23:04:43 23:04:43  9:43:25
m     131099 0x35c2cf88 --rw-rw----    oracle  oinstall    oracle  oinstall     24 1107361792  2370 22038 23:04:46 23:04:46  9:44:37
m      32800 0xc3d6ab78 --rw-rw----    oracle  oinstall    oracle  oinstall     25 2634088448 19751 21940 23:04:06 23:04:37 14:00:30
m      32801 0x510bd824 --rw-rw----    oracle  oinstall    oracle  oinstall     26 2466316288 23261 21994 23:04:33 23:05:25 14:12:18
m     131106 0x5fbab904 --rw-rw----    oracle  oinstall    oracle  oinstall     22 1728118784 27090 21454 23:00:07 23:00:08 14:23:51
m      32803 0x2e2918a0 --rw-rw----    oracle  oinstall    oracle  oinstall     18  599851008  2690 21519 23:03:20 23:04:41 14:43:54

and i sort 

 

oracle   12840     1 3967872 1212 ora_vkrm_tvetest
oracle   14636     1 3967872 1217 ora_psp0_tvetest
oracle   14668     1 3967872 1212 ora_vktm_tvetest
oracle   14676     1 3967872 1213 ora_gen0_tvetest
oracle   14686     1 3967872 1211 ora_diag_tvetest
oracle   14708     1 3967872 1215 ora_mman_tvetest
oracle   15376     1 3967872 1213 ora_qmnc_tvetest
oracle   16208     1 3967872 1217 ora_smco_tvetest
oracle   14632     1 3968000 1230 ora_pmon_tvetest
oracle   14734     1 3968128 1215 ora_ckpt_tvetest
oracle   15451     1 3968128 1215 ora_q001_tvetest
oracle    1135     1 3968320 1205 oracletvetest (LOCAL=NO)
oracle    4597     1 3968320 1228 oracletvetest (LOCAL=NO)
oracle   13022     1 3968320 1228 oracletvetest (LOCAL=NO)
oracle   20900     1 3968320 1226 ora_w000_tvetest
oracle   22237     1 3968320 1291 ora_w001_tvetest
oracle   12078     1 3968384 1205 oracletvetest (LOCAL=NO)
oracle   14749     1 3968640 1211 ora_reco_tvetest
oracle   14764     1 3969152 1224 ora_mmnl_tvetest
oracle   19563     1 3969152 1205 oracletvetest (LOCAL=NO)
oracle   20733     1 3969152 1205 oracletvetest (LOCAL=NO)
oracle   20737     1 3969152 1205 oracletvetest (LOCAL=NO)
oracle   21737     1 3969152 1229 oracletvetest (LOCAL=NO)
oracle   14774     1 3969280 1217 ora_d000_tvetest
oracle   14700     1 3969408 1214 ora_dia0_tvetest
oracle   20735     1 3969600 1205 oracletvetest (LOCAL=NO)
oracle   20891     1 3969600 1205 oracletvetest (LOCAL=NO)
oracle   15449     1 3969664 1211 ora_q000_tvetest
oracle   14692     1 3969984 1218 ora_dbrm_tvetest
oracle   14742     1 3972864 1229 ora_smon_tvetest
oracle   14758     1 3973120 1217 ora_mmon_tvetest
oracle   14716     1 3974272 1214 ora_dbw0_tvetest
oracle   15410     1 3978944 1218 ora_cjq0_tvetest
oracle   14726     1 3983680 1216 ora_lgwr_tvetest
oracle   15319     1 3991872 1216 ora_arc2_tvetest
oracle   15303     1 4000064 1215 ora_arc0_tvetest
oracle   15313     1 4000064 1212 ora_arc1_tvetest
oracle   15331     1 4000064 1215 ora_arc3_tvetest

i think tvetest DB eat a lot of memory?

0rb1t0
Advisor

Re: RED Global swap space is nearly full

i shutdown this tvetest db.

and sort command show^

 

oracle   19873     1 3017216 1625 ora_mman_UPDB
oracle   20060     1 3017216 1624 ora_qmnc_UPDB
oracle   22015     1 3017216 1623 ora_smco_UPDB
oracle    4494     1 3017408 1622 ora_vkrm_UPDB
oracle   19783     1 3017408 1643 ora_pmon_UPDB
oracle   17584     1 3017600 1625 ora_q003_UPDB
oracle   19871     1 3017728 1624 ora_dia0_UPDB
oracle   19881     1 3017984 1625 ora_ckpt_UPDB
oracle   19869     1 3018048 1627 ora_dbrm_UPDB
oracle   20250     1 3018112 1624 ora_w000_UPDB
oracle   22275     1 3018112 1625 ora_w001_UPDB
oracle   19889     1 3018752 1629 ora_mmnl_UPDB
oracle   19885     1 3019008 1622 ora_reco_UPDB
oracle   19883     1 3020160 1628 ora_smon_UPDB
oracle   19887     1 3022464 1630 ora_mmon_UPDB
oracle   21465     1 3023616 1646 oracleUPDB (LOCAL=NO)
oracle   15712     1 3024320 1635 ora_j000_UPDB
oracle    4886     1 3028608 1636 ora_cjq0_UPDB
oracle   19879     1 3033024 1626 ora_lgwr_UPDB
oracle   19875     1 3054336 1624 ora_dbw0_UPDB
oracle   19877     1 3054336 1624 ora_dbw1_UPDB
doctest   2082     1 3614464  100 /opt/java1.5/bin/IA64N/java -Djava.awt.headless=true -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.u
#> ipcs -moab
IPC status from /dev/kmem as of Fri Jun 19 23:10:16 2015
T         ID     KEY        MODE        OWNER     GROUP   CREATOR    CGROUP NATTCH      SEGSZ  CPID  LPID   ATIME    DTIME    CTIME 
Shared Memory:
m          0 0x410c185d --rw-rw-rw-      root      root      root      root      0        348  1341  1341  8:37:58  8:37:58  8:37:53
m          1 0x4e0c0002 --rw-rw-rw-      root      root      root      root      2      61760  1341 19728 10:50:19 11:07:39  8:37:53
m          2 0x410efccd --rw-rw-rw-      root      root      root      root      2       8192  1341 11059  1:33:09  1:33:09  8:37:53
m          3 0x410c40cb --rw-r--r--      root      root      root      root      3    2850096  2388  4202 23:10:15 23:10:15  8:38:46
m          4 0x490ded32 --rw-r--r--      root      root      root      root      1      22908  2388  3496 23:10:10 23:10:10  8:38:46
m          5 0x500c40ca --rw-r--r--      root   hpvmsys      root   hpvmsys      1   17115608  2394  3496 23:10:10 23:10:10  8:38:47
m     622598 0x00a5c581 --rw-------     sfmdb     users     sfmdb     users      4   11345920 14440 14485  2:10:27  2:10:41  2:10:27
m          7 0x5e0e822b --rw-------      root      root      root      root      1        512  3182  3182  8:40:06 no-entry  8:40:06
m          8 0x02fb07f1 --rw-------    hpsmdb     users    hpsmdb     users      1   10469376  3331  3335  8:40:07  8:40:07  8:40:07
m          9 0x06347849 --rw-rw-rw-      root      root      root      root      2      65544  3410  3421  8:40:15  8:40:12  8:40:10
m      32778 0x0c6629c9 --rw-r-----      root      root      root      root      3   18121528  3423 21075 16:16:41 16:17:29  8:40:12
m         11 0x010eb2ff --rw-rw-r--      root      root      root      root      1       4192  3426  3426  8:40:12  8:40:12  8:40:12
m         12 0xa90c5105 -----------      root      root      root      root      1          1  4003  4004  8:40:34 no-entry  8:40:34
m      65549 0x6c0ef197 --rw-------    zabbix     users    zabbix     users      6     995944 28067 28067 13:28:14 no-entry 13:28:14
m         14 0x010eb2d3 --rw-rw-r--      root      root      root      root      1       4192  4095  4095  8:40:40  8:40:40  8:40:40
m         15 0x010eb278 --rw-rw-r--      root      root      root      root      1       4192  4102  4102  8:40:41 no-entry  8:40:41
m         16 0x010eb2e8 --rw-rw-r--      root      root      root      root      1       4192  4104  4117  8:40:45  8:40:51  8:40:43
m      32785 0x010d8c05 --rw-rw-r--      root      root      root      root      1       4192  4123  4123  8:40:51 no-entry  8:40:51
m   41779218 0x3564e93c --rw-rw----    oracle  oinstall    oracle  oinstall     15 1593901056  8212 22012 23:04:36 23:05:37  5:25:32
m     458771 0x000001df --rw-------   doctest     dinst   doctest     dinst     10    6470528   479   479 17:37:11 no-entry 17:37:11
m     393236 0x00000000 D-rw-------   doctest     dinst   doctest     dinst     10       4096   479   479 17:37:19 no-entry 17:37:19
m    5308437 0x00000000 D-rw-------   doctest     dinst   doctest     dinst     10       4096   479   479 17:37:19 no-entry 17:37:19
m     196630 0x720f2ab9 --rw-------   doctest     dinst   doctest     dinst     16    2293760   479   509 17:37:24 no-entry 17:37:19
m     589850 0x54397118 --rw-rw----    oracle  oinstall    oracle  oinstall     24 1291911168  1538 22278 23:06:06 23:06:06  9:43:25
m     131099 0x35c2cf88 --rw-rw----    oracle  oinstall    oracle  oinstall     25 1107361792  2370 22470 23:07:24 23:07:25  9:44:37
m      32800 0xc3d6ab78 --rw-rw----    oracle  oinstall    oracle  oinstall     28 2634088448 19751  4537 23:10:00 23:10:00 14:00:30
m      32801 0x510bd824 --rw-rw----    oracle  oinstall    oracle  oinstall     27 2466316288 23261  4021 23:09:55 23:09:55 14:12:18
m     131106 0x5fbab904 --rw-rw----    oracle  oinstall    oracle  oinstall     22 1728118784 27090 22294 23:06:13 23:06:13 14:23:51
m      32803 0x2e2918a0 --rw-rw----    oracle  oinstall    oracle  oinstall     18  599851008  2690 21519 23:03:20 23:04:41 14:43:54
0rb1t0
Advisor

Re: RED Global swap space is nearly full

maybe oracle not correctly configured?

Patrick Wallek
Honored Contributor

Re: RED Global swap space is nearly full

So what does the memory usage from 'glance' look like now and what does 'swapinfo -tam' show now?

0rb1t0
Advisor

Re: RED Global swap space is nearly full

glance say memory util 100 %

 

#> swapinfo -tam
                 Mb          Mb           Mb   PCT      START/          Mb
TYPE          AVAIL        USED         FREE  USED       LIMIT     RESERVE  PRI  NAME
dev           32768       13812        18956   42%           0           -    1  /dev/vg00/lvol2
reserve           -        6706        -6706
memory       130496      124449         6047   95%
total        163264      144967        18297   89%       -           0    -
Dennis Handly
Acclaimed Contributor

Re: Global swap space is nearly full

>maybe oracle not correctly configured?

 

It seems you have too much SGA space configured.

You need to get Oracle to release those shared memory segments.

0rb1t0
Advisor

Re: Global swap space is nearly full

Thank you Dennis Handly.

It seems that, as is.

Thank you very much/

0rb1t0
Advisor

Re: Global swap space is nearly full

I shutdown all oracle db and restart the hp-ux , but

 

#> swapinfo -tam
                 Mb          Mb           Mb   PCT      START/          Mb
TYPE          AVAIL        USED         FREE  USED       LIMIT     RESERVE  PRI  NAME
dev           32768           0        32768    0%           0           -    1  /dev/vg00/lvol2
reserve           -        2035        -2035
memory       130496      121065         9431   93%
total        163264      123100        40164   75%       -           0    -

i dont know, what happen (

0rb1t0
Advisor

Re: Global swap space is nearly full

#> kctune|grep -i shmm*
core_addshmem_read                           0  Default             Immed
core_addshmem_write                          0  Default             Immed
shmmax                            137256417362  137256417362        Immed
shmmni                                    7440  7440                Immed
shmseg                                    7340  7340                Immed
-----------------------------------------------
#> /usr/contrib/bin/machinfo  | grep -i Memory
Memory: 130896 MB (127.83 GB)

 

#> ipcs -m
IPC status from /dev/kmem as of Mon Jun 22 13:59:27 2015
T         ID     KEY        MODE        OWNER     GROUP
Shared Memory:
m          0 0x410c185d --rw-rw-rw-      root      root
m          1 0x4e0c0002 --rw-rw-rw-      root      root
m          2 0x410efdc8 --rw-rw-rw-      root      root
m          3 0x410c40cb --rw-r--r--      root      root
m          4 0x490ded32 --rw-r--r--      root      root
m          5 0x500c40ca --rw-r--r--      root   hpvmsys
m      32774 0x00a5c581 --rw-------     sfmdb     users
m          7 0x5e0e822b --rw-------      root      root
m          8 0x02fb07f1 --rw-------    hpsmdb     users
m          9 0x06347849 --rw-rw-rw-      root      root
m      32778 0x0c6629c9 --rw-r-----      root      root
m         11 0x010eb2ff --rw-rw-r--      root      root
m         12 0xa90c5105 -----------      root      root
m     327693 0x00000000 --rw-rw----    oracle  oinstall
m         14 0x010eb2d3 --rw-rw-r--      root      root
m         15 0x010eb278 --rw-rw-r--      root      root
m         16 0x010eb2e8 --rw-rw-r--      root      root
m      32785 0x010d8c05 --rw-rw-r--      root      root
m     426002 0x00000000 --rw-rw----    oracle  oinstall
m      98323 0x1faae8ec --rw-rw----    oracle  oinstall
m      65556 0x00003d97 --rw-------   doctest     dinst
m         21 0x00000000 D-rw-------   doctest     dinst
m         22 0x00000000 D-rw-------   doctest     dinst
m         23 0x720f0e2e --rw-------   doctest     dinst

 

Dennis Handly
Acclaimed Contributor

Re: Global swap space is nearly full

> shmmax 137256417362

 

I would say that's way too much, more than your memory.

0rb1t0
Advisor

Re: Global swap space is nearly full

i calculate this by formula:

 

127,83*1024*1024*1024=137256417362

 

sorry I dont understand, right value

Dennis Handly
Acclaimed Contributor
Solution

Re: Global swap space is nearly full

>I calculate this by formula:

 

I would cut it in half, since you can't really get a segment that big, without swapping.

0rb1t0
Advisor

Re: Global swap space is nearly full

Thank you very much  Dennis Handly