RED Global swap space is nearly full

[0rb1t0]

Hello to all!

I read all internet sites but not undestand whe botllneck on server HPUX/

Glance say to me 14:15:15 RED Global swap space is nearly full

And i have:

Disk Util F FV V |100% 81% 100%
Mem Util S SU U |100% 100% 100%
Swap Util U UR R | 96% 96% 96%

Please help me

[0rb1t0]

#> vmstat 1 5
procs memory page faults cpu
r b w avm free re at pi po fr de sr in sy cs us sy id
1 2 0 196842 7986 13 0 26 13 7 0 338 5416 233880 678 3 1 96
1 2 0 196842 7982 186 0 184 14 31 0 2017 6905 13859 1326 1 5 94
1 2 0 196842 8106 197 0 194 11 24 0 2035 6848 16564 1299 3 8 89
1 2 0 196842 8130 198 0 186 8 19 0 1998 7065 16232 1436 1 8 91
1 2 0 196842 8167 189 0 177 6 15 0 1926 6919 16247 1356 3 2 95

[Patrick Wallek]

What does the command:

# swapinfo -tam

show?

My guess is that you are, or have been, swapping to such an extent that your swap space is nearly used up.

[0rb1t0]

#> swapinfo -tam
                 Mb          Mb           Mb   PCT      START/          Mb
TYPE          AVAIL        USED         FREE  USED       LIMIT     RESERVE  PRI  NAME
dev           32768       15005        17763   46%           0           -    1  /dev/vg00/lvol2
reserve           -       10414       -10414
memory       130496      124372         6124   95%
total        163264      149791        13473   92%       -           0    -

I dont know what happen, not new applications be added  to him

[Patrick Wallek]

You've got something that is using lots of memory and causing the system to page out.

Perhaps something opened some new share memory segments?  Check the output of 'ipcs -moab' to see shared memory.

To see what processes are using lots of memory you can do:

# UNIX95= ps -eo user,pid,ppid,vsz,sz,args | sort -nk 4

The 4th column will show the virtual memory size of the process and the list will be sorted in ascending order.

[0rb1t0]

In this server a lot of process opened by oracle.

#> ipcs -moab
IPC status from /dev/kmem as of Fri Jun 19 23:05:31 2015
T         ID     KEY        MODE        OWNER     GROUP   CREATOR    CGROUP NATTCH      SEGSZ  CPID  LPID   ATIME    DTIME    CTIME 
Shared Memory:
m          0 0x410c185d --rw-rw-rw-      root      root      root      root      0        348  1341  1341  8:37:58  8:37:58  8:37:53
m          1 0x4e0c0002 --rw-rw-rw-      root      root      root      root      2      61760  1341 19728 10:50:19 11:07:39  8:37:53
m          2 0x410efccd --rw-rw-rw-      root      root      root      root      2       8192  1341 11059  1:33:09  1:33:09  8:37:53
m          3 0x410c40cb --rw-r--r--      root      root      root      root      3    2850096  2388  4202 23:05:30 23:05:30  8:38:46
m          4 0x490ded32 --rw-r--r--      root      root      root      root      1      22908  2388  3496 23:05:30 23:05:30  8:38:46
m          5 0x500c40ca --rw-r--r--      root   hpvmsys      root   hpvmsys      1   17115608  2394  3496 23:05:30 23:05:30  8:38:47
m     622598 0x00a5c581 --rw-------     sfmdb     users     sfmdb     users      4   11345920 14440 14485  2:10:27  2:10:41  2:10:27
m          7 0x5e0e822b --rw-------      root      root      root      root      1        512  3182  3182  8:40:06 no-entry  8:40:06
m          8 0x02fb07f1 --rw-------    hpsmdb     users    hpsmdb     users      1   10469376  3331  3335  8:40:07  8:40:07  8:40:07
m          9 0x06347849 --rw-rw-rw-      root      root      root      root      2      65544  3410  3421  8:40:15  8:40:12  8:40:10
m      32778 0x0c6629c9 --rw-r-----      root      root      root      root      3   18121528  3423 21075 16:16:41 16:17:29  8:40:12
m         11 0x010eb2ff --rw-rw-r--      root      root      root      root      1       4192  3426  3426  8:40:12  8:40:12  8:40:12
m         12 0xa90c5105 -----------      root      root      root      root      1          1  4003  4004  8:40:34 no-entry  8:40:34
m      65549 0x6c0ef197 --rw-------    zabbix     users    zabbix     users      6     995944 28067 28067 13:28:14 no-entry 13:28:14
m         14 0x010eb2d3 --rw-rw-r--      root      root      root      root      1       4192  4095  4095  8:40:40  8:40:40  8:40:40
m         15 0x010eb278 --rw-rw-r--      root      root      root      root      1       4192  4102  4102  8:40:41 no-entry  8:40:41
m         16 0x010eb2e8 --rw-rw-r--      root      root      root      root      1       4192  4104  4117  8:40:45  8:40:51  8:40:43
m      32785 0x010d8c05 --rw-rw-r--      root      root      root      root      1       4192  4123  4123  8:40:51 no-entry  8:40:51
m   41779218 0x3564e93c --rw-rw----    oracle  oinstall    oracle  oinstall     16 1593901056  8212 22012 23:04:36 23:04:36  5:25:32
m     458771 0x000001df --rw-------   doctest     dinst   doctest     dinst     10    6470528   479   479 17:37:11 no-entry 17:37:11
m     393236 0x00000000 D-rw-------   doctest     dinst   doctest     dinst     10       4096   479   479 17:37:19 no-entry 17:37:19
m    5308437 0x00000000 D-rw-------   doctest     dinst   doctest     dinst     10       4096   479   479 17:37:19 no-entry 17:37:19
m     196630 0x720f2ab9 --rw-------   doctest     dinst   doctest     dinst     16    2293760   479   509 17:37:24 no-entry 17:37:19
m     655383 0x00000000 --rw-r-----    oracle  oinstall    oracle  oinstall     79   50331648 14472 21866 23:03:51 23:04:11 15:39:41
m     360472 0x00000000 --rw-r-----    oracle  oinstall    oracle  oinstall     79 3539992576 14472 21866 23:03:51 23:04:11 15:39:41
m     294937 0x1faae8ec --rw-r-----    oracle  oinstall    oracle  oinstall     79      65536 14472 21866 23:03:51 23:04:11 15:39:41
m     589850 0x54397118 --rw-rw----    oracle  oinstall    oracle  oinstall     24 1291911168  1538 22028 23:04:43 23:04:43  9:43:25
m     131099 0x35c2cf88 --rw-rw----    oracle  oinstall    oracle  oinstall     24 1107361792  2370 22038 23:04:46 23:04:46  9:44:37
m      32800 0xc3d6ab78 --rw-rw----    oracle  oinstall    oracle  oinstall     25 2634088448 19751 21940 23:04:06 23:04:37 14:00:30
m      32801 0x510bd824 --rw-rw----    oracle  oinstall    oracle  oinstall     26 2466316288 23261 21994 23:04:33 23:05:25 14:12:18
m     131106 0x5fbab904 --rw-rw----    oracle  oinstall    oracle  oinstall     22 1728118784 27090 21454 23:00:07 23:00:08 14:23:51
m      32803 0x2e2918a0 --rw-rw----    oracle  oinstall    oracle  oinstall     18  599851008  2690 21519 23:03:20 23:04:41 14:43:54

and i sort 

oracle   12840     1 3967872 1212 ora_vkrm_tvetest
oracle   14636     1 3967872 1217 ora_psp0_tvetest
oracle   14668     1 3967872 1212 ora_vktm_tvetest
oracle   14676     1 3967872 1213 ora_gen0_tvetest
oracle   14686     1 3967872 1211 ora_diag_tvetest
oracle   14708     1 3967872 1215 ora_mman_tvetest
oracle   15376     1 3967872 1213 ora_qmnc_tvetest
oracle   16208     1 3967872 1217 ora_smco_tvetest
oracle   14632     1 3968000 1230 ora_pmon_tvetest
oracle   14734     1 3968128 1215 ora_ckpt_tvetest
oracle   15451     1 3968128 1215 ora_q001_tvetest
oracle    1135     1 3968320 1205 oracletvetest (LOCAL=NO)
oracle    4597     1 3968320 1228 oracletvetest (LOCAL=NO)
oracle   13022     1 3968320 1228 oracletvetest (LOCAL=NO)
oracle   20900     1 3968320 1226 ora_w000_tvetest
oracle   22237     1 3968320 1291 ora_w001_tvetest
oracle   12078     1 3968384 1205 oracletvetest (LOCAL=NO)
oracle   14749     1 3968640 1211 ora_reco_tvetest
oracle   14764     1 3969152 1224 ora_mmnl_tvetest
oracle   19563     1 3969152 1205 oracletvetest (LOCAL=NO)
oracle   20733     1 3969152 1205 oracletvetest (LOCAL=NO)
oracle   20737     1 3969152 1205 oracletvetest (LOCAL=NO)
oracle   21737     1 3969152 1229 oracletvetest (LOCAL=NO)
oracle   14774     1 3969280 1217 ora_d000_tvetest
oracle   14700     1 3969408 1214 ora_dia0_tvetest
oracle   20735     1 3969600 1205 oracletvetest (LOCAL=NO)
oracle   20891     1 3969600 1205 oracletvetest (LOCAL=NO)
oracle   15449     1 3969664 1211 ora_q000_tvetest
oracle   14692     1 3969984 1218 ora_dbrm_tvetest
oracle   14742     1 3972864 1229 ora_smon_tvetest
oracle   14758     1 3973120 1217 ora_mmon_tvetest
oracle   14716     1 3974272 1214 ora_dbw0_tvetest
oracle   15410     1 3978944 1218 ora_cjq0_tvetest
oracle   14726     1 3983680 1216 ora_lgwr_tvetest
oracle   15319     1 3991872 1216 ora_arc2_tvetest
oracle   15303     1 4000064 1215 ora_arc0_tvetest
oracle   15313     1 4000064 1212 ora_arc1_tvetest
oracle   15331     1 4000064 1215 ora_arc3_tvetest

i think tvetest DB eat a lot of memory?

[0rb1t0]

i shutdown this tvetest db.

and sort command show^

oracle   19873     1 3017216 1625 ora_mman_UPDB
oracle   20060     1 3017216 1624 ora_qmnc_UPDB
oracle   22015     1 3017216 1623 ora_smco_UPDB
oracle    4494     1 3017408 1622 ora_vkrm_UPDB
oracle   19783     1 3017408 1643 ora_pmon_UPDB
oracle   17584     1 3017600 1625 ora_q003_UPDB
oracle   19871     1 3017728 1624 ora_dia0_UPDB
oracle   19881     1 3017984 1625 ora_ckpt_UPDB
oracle   19869     1 3018048 1627 ora_dbrm_UPDB
oracle   20250     1 3018112 1624 ora_w000_UPDB
oracle   22275     1 3018112 1625 ora_w001_UPDB
oracle   19889     1 3018752 1629 ora_mmnl_UPDB
oracle   19885     1 3019008 1622 ora_reco_UPDB
oracle   19883     1 3020160 1628 ora_smon_UPDB
oracle   19887     1 3022464 1630 ora_mmon_UPDB
oracle   21465     1 3023616 1646 oracleUPDB (LOCAL=NO)
oracle   15712     1 3024320 1635 ora_j000_UPDB
oracle    4886     1 3028608 1636 ora_cjq0_UPDB
oracle   19879     1 3033024 1626 ora_lgwr_UPDB
oracle   19875     1 3054336 1624 ora_dbw0_UPDB
oracle   19877     1 3054336 1624 ora_dbw1_UPDB
doctest   2082     1 3614464  100 /opt/java1.5/bin/IA64N/java -Djava.awt.headless=true -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.u

#> ipcs -moab
IPC status from /dev/kmem as of Fri Jun 19 23:10:16 2015
T         ID     KEY        MODE        OWNER     GROUP   CREATOR    CGROUP NATTCH      SEGSZ  CPID  LPID   ATIME    DTIME    CTIME 
Shared Memory:
m          0 0x410c185d --rw-rw-rw-      root      root      root      root      0        348  1341  1341  8:37:58  8:37:58  8:37:53
m          1 0x4e0c0002 --rw-rw-rw-      root      root      root      root      2      61760  1341 19728 10:50:19 11:07:39  8:37:53
m          2 0x410efccd --rw-rw-rw-      root      root      root      root      2       8192  1341 11059  1:33:09  1:33:09  8:37:53
m          3 0x410c40cb --rw-r--r--      root      root      root      root      3    2850096  2388  4202 23:10:15 23:10:15  8:38:46
m          4 0x490ded32 --rw-r--r--      root      root      root      root      1      22908  2388  3496 23:10:10 23:10:10  8:38:46
m          5 0x500c40ca --rw-r--r--      root   hpvmsys      root   hpvmsys      1   17115608  2394  3496 23:10:10 23:10:10  8:38:47
m     622598 0x00a5c581 --rw-------     sfmdb     users     sfmdb     users      4   11345920 14440 14485  2:10:27  2:10:41  2:10:27
m          7 0x5e0e822b --rw-------      root      root      root      root      1        512  3182  3182  8:40:06 no-entry  8:40:06
m          8 0x02fb07f1 --rw-------    hpsmdb     users    hpsmdb     users      1   10469376  3331  3335  8:40:07  8:40:07  8:40:07
m          9 0x06347849 --rw-rw-rw-      root      root      root      root      2      65544  3410  3421  8:40:15  8:40:12  8:40:10
m      32778 0x0c6629c9 --rw-r-----      root      root      root      root      3   18121528  3423 21075 16:16:41 16:17:29  8:40:12
m         11 0x010eb2ff --rw-rw-r--      root      root      root      root      1       4192  3426  3426  8:40:12  8:40:12  8:40:12
m         12 0xa90c5105 -----------      root      root      root      root      1          1  4003  4004  8:40:34 no-entry  8:40:34
m      65549 0x6c0ef197 --rw-------    zabbix     users    zabbix     users      6     995944 28067 28067 13:28:14 no-entry 13:28:14
m         14 0x010eb2d3 --rw-rw-r--      root      root      root      root      1       4192  4095  4095  8:40:40  8:40:40  8:40:40
m         15 0x010eb278 --rw-rw-r--      root      root      root      root      1       4192  4102  4102  8:40:41 no-entry  8:40:41
m         16 0x010eb2e8 --rw-rw-r--      root      root      root      root      1       4192  4104  4117  8:40:45  8:40:51  8:40:43
m      32785 0x010d8c05 --rw-rw-r--      root      root      root      root      1       4192  4123  4123  8:40:51 no-entry  8:40:51
m   41779218 0x3564e93c --rw-rw----    oracle  oinstall    oracle  oinstall     15 1593901056  8212 22012 23:04:36 23:05:37  5:25:32
m     458771 0x000001df --rw-------   doctest     dinst   doctest     dinst     10    6470528   479   479 17:37:11 no-entry 17:37:11
m     393236 0x00000000 D-rw-------   doctest     dinst   doctest     dinst     10       4096   479   479 17:37:19 no-entry 17:37:19
m    5308437 0x00000000 D-rw-------   doctest     dinst   doctest     dinst     10       4096   479   479 17:37:19 no-entry 17:37:19
m     196630 0x720f2ab9 --rw-------   doctest     dinst   doctest     dinst     16    2293760   479   509 17:37:24 no-entry 17:37:19
m     589850 0x54397118 --rw-rw----    oracle  oinstall    oracle  oinstall     24 1291911168  1538 22278 23:06:06 23:06:06  9:43:25
m     131099 0x35c2cf88 --rw-rw----    oracle  oinstall    oracle  oinstall     25 1107361792  2370 22470 23:07:24 23:07:25  9:44:37
m      32800 0xc3d6ab78 --rw-rw----    oracle  oinstall    oracle  oinstall     28 2634088448 19751  4537 23:10:00 23:10:00 14:00:30
m      32801 0x510bd824 --rw-rw----    oracle  oinstall    oracle  oinstall     27 2466316288 23261  4021 23:09:55 23:09:55 14:12:18
m     131106 0x5fbab904 --rw-rw----    oracle  oinstall    oracle  oinstall     22 1728118784 27090 22294 23:06:13 23:06:13 14:23:51
m      32803 0x2e2918a0 --rw-rw----    oracle  oinstall    oracle  oinstall     18  599851008  2690 21519 23:03:20 23:04:41 14:43:54

[0rb1t0]

maybe oracle not correctly configured?

[Patrick Wallek]

So what does the memory usage from 'glance' look like now and what does 'swapinfo -tam' show now?

[0rb1t0]

glance say memory util 100 %

#> swapinfo -tam
                 Mb          Mb           Mb   PCT      START/          Mb
TYPE          AVAIL        USED         FREE  USED       LIMIT     RESERVE  PRI  NAME
dev           32768       13812        18956   42%           0           -    1  /dev/vg00/lvol2
reserve           -        6706        -6706
memory       130496      124449         6047   95%
total        163264      144967        18297   89%       -           0    -