System Administration
cancel
Showing results for 
Search instead for 
Did you mean: 

Server is **bleep** slow and taking too much in login

 
rock_singh
Occasional Advisor

Server is **bleep** slow and taking too much in login

Hi,

 

It is regarding with one of HP-UX box and issue is that server is very slow and taking too much time in login.

What could be the possible issue? As i am new to HP-UX. How to trace the issue.

 

 

#-> bdf /

 

Filesystem          kbytes    used   avail %used Mounted on
/dev/vg00/lvol3    1048576  417160  626520   40% /

 

Load averages: 1.12, 1.17, 1.18
319 processes: 218 sleeping, 69 running, 32 zombies

 

Appreciate the response.

 


 

4 REPLIES
rock_singh
Occasional Advisor

Re: Server is **bleep** slow and taking too much in login

Some more findings:--

 

 

#-> dmesg

 

Dec 13 07:40
...
.

SCSI: Async write error -- dev: b 31 0x048002, errno: 126, resid: 8192,
        blkno: 36680320, sectno: 74384674, offset: 37560647680, bcount: 8192.

SCSI: Async write error -- dev: b 31 0x048002, errno: 126, resid: 8192,
        blkno: 36680312, sectno: 74384658, offset: 37560639488, bcount: 819

 
SCSI: Write error -- dev: b 31 0x048002, errno: 126, resid: 2048,
        blkno: 2728, sectno: 1029490, offset: 2793472, bcount: 2048.
SCSI Ultra320 2/0/4/1/0 instance 4:       IO Type : SCSI IO has timed-out.       Target ID: 8, LUN ID: 0.       Inquiry Command - CDB: 12 00 00 00 ff 00
LVM: VG 64 0x000000: PVLink 31 0x048002 Failed! The PV is not accessible.
SCSI Ultra320 2/0/4/1/0 instance 4:       An IO timeout condition was detected.       Condition cleared, no intervention required.
SCSI Ultra320 2/0/4/1/1 instance 5:       Driver initiating SCSI bus reset.       Condition cleared, no intervention required.
 
 
 
------------------------------------------------------------------------------------------------------------------------------------------------------
Cpu  Util   S                                                                            SARRU                                 U     | 97%   73%   97%
Disk Util   F                       F                                                                                                | 21%   29%   33%
Mem  Util   S                  SU                                                                     UB            B                | 88%   88%   88%
Swap Util   U          UR                                        R                                                                   | 46%   46%   46%
------------------------------------------------------------------------------------------------------------------------------------------------------
                                                                      IO BY DISK                                                          Users=    2
                                                      Logl       Phys       Serv
Idx   Device        Util    Qlen       KB/Sec          IO         IO        Time
--------------------------------------------------------------------------------
   1 c4t8d0s2      0/  0     0.0     0.0/    0.0    na/   na   0.0/  0.0     0.0
   2 c10t8d0s2    21/ 28     0.0  1802.4/ 1935.7    na/   na  62.2/ 73.7     8.6
   3 c16t0d0       0/  0     0.0     0.0/    0.0    na/   na   0.0/  0.0     0.0
   4 c18t0d0       0/  0     0.0     0.0/    0.0    na/   na   0.0/  0.0     0.0
   5 c18t0d1       0/  0     0.0     1.6/    0.7    na/   na   0.2/  0.0     0.0
   6 c16t0d1       0/  0     0.0     0.0/    0.0    na/   na   0.0/  0.0     0.0
   7 c16t0d2       0/  0     0.0     0.0/    0.0    na/   na   0.0/  0.0     0.0
   8 c18t0d2       0/  0     0.0     0.0/    0.0    na/   na   0.0/  0.0     0.0
 
------------------------------------------------------------------------------------------------------------------------------------------------------
CPU  Util   S                                                              SR  RU          U                                         | 67%   71%   97%
Disk Util   F     F                                                                                                                  |  6%   30%   60%
Mem  Util   S                  SU                                                                     UB            B                | 88%   88%   88%
Swap Util   U          UR                                         R                                                                  | 46%   46%   46%
------------------------------------------------------------------------------------------------------------------------------------------------------
                                                                  IO BY FILE SYSTEM                                                       Users=    2
Idx   File System            Device               Type     Logl IO     Phys IO
--------------------------------------------------------------------------------
   1 /                       /dev/vg00/lvol3      vxfs      na/   na   0.4/  8.9
   2 /stand                  /dev/vg00/lvol1      vxfs      na/   na   0.0/  0.6
   3 /work/XJ2               */vg_XJ2/lv_workXJ2  vxfs      na/   na   0.0/  0.0
   4 /var                    /dev/vg00/lvol9      vxfs      na/   na   0.0/  2.0
   5 /var/adm/crash          /dev/vg00/lvol10     vxfs      na/   na   0.0/  0.0
   6 /usr                    /dev/vg00/lvol7      vxfs      na/   na   0.2/  0.8
   7 /usr/sap/trans          */lv_usrsaptrans     vxfs      na/   na   0.0/  0.0
Patrick Wallek
Honored Contributor

Re: Server is **bleep** slow and taking too much in login

More information from /var/adm/syslog/syslog.log would be helpful.  As would the output of 'ioscan -fn'.  If you can attach that output, rather than pasting it into this thread that would help readability.

 

That being said, judging from your dmesg output it appears you may have a disk that is going bad.

 

rock_singh
Occasional Advisor

Re: Server is **bleep** slow and taking too much in login

#-> tail -100 syslog.log

Dec 13 06:47:40 jpnqxic1 adclient[24430]: WARN  <fd:33 sudo(3919)> Open session for user 'root': mapping misconfiguration. Passing user to next service module.
Dec 13 06:47:40 jpnqxic1 adclient[24430]: INFO  <fd:28 PAMIsUserAllowedAccess> audit User 'mbhatia1' is authorized
Dec 13 06:48:39 jpnqxic1 su: + tty?? root-sentrigo
Dec 13 06:48:39 jpnqxic1 adclient[24430]: WARN  <fd:34 sudo(3919)> Close session for user 'root': mapping misconfiguration. Passing user to next service module.
Dec 13 06:48:39 jpnqxic1 su: + 1 mbhatia1-root
Dec 13 06:54:57 jpnqxic1 su: Can't rewrite protected password entry for user sentrigo
Dec 13 06:55:30 jpnqxic1 sshd[4496]: fatal: Timeout before authentication for 10.150.163.105
Dec 13 06:56:25 jpnqxic1 adclient[24430]: WARN  <fd:32 NSSGetPasswdDataByName> base.bind.cache LDAP search CN=Users,CN=JPNQXIC1-HPUX-NA,CN=Zones,CN=Centrify,CN=Program Data,DC=jnj,DC=com:(&(keywords=upn:(unknown user))(displayName=$CimsUserVersion3)) threw unexpected exception: search : Bad search filter
Dec 13 06:56:25 jpnqxic1 adclient[24430]: WARN  <fd:32 NSSGetPrPasswdDataByName> base.bind.cache LDAP search CN=Users,CN=JPNQXIC1-HPUX-NA,CN=Zones,CN=Centrify,CN=Program Data,DC=jnj,DC=com:(&(keywords=upn:(unknown user))(displayName=$CimsUserVersion3)) threw unexpected exception: search : Bad search filter
Dec 13 06:58:27 jpnqxic1 sshd[5032]: fatal: Timeout before authentication for 10.150.163.105
Dec 13 06:59:36 jpnqxic1 adclient[24430]: WARN  <fd:33 login(5087)> Authentication for user 'root': mapping misconfiguration. Passing user to next service module.
Dec 13 06:59:36 jpnqxic1 adclient[24430]: WARN  <fd:33 login(5087)> Account management for user 'root': mapping misconfiguration. Passing user to next service module.
Dec 13 06:59:36 jpnqxic1 adclient[24430]: WARN  <fd:33 login(5087)> Set credentials for user 'root': mapping misconfiguration. Passing user to next service module.
Dec 13 06:59:26 jpnqxic1 su: + tty?? root-sentrigo
Dec 13 06:59:36 jpnqxic1  above message repeats 11 times
Dec 13 06:59:36 jpnqxic1 adclient[24430]: WARN  <fd:34 login(5087)> Open session for user 'root': mapping misconfiguration. Passing user to next service module.
Dec 13 07:00:30 jpnqxic1 sshd[23235]: ERROR: Unexpected error in conversation: (6)
Dec 13 07:00:30 jpnqxic1 sshd[23235]: WARNING: Failed to get password for 'get_first_pass' option
Dec 13 07:02:28 jpnqxic1 su: + tty?? root-sentrigo
Dec 13 07:02:30 jpnqxic1 adclient[24430]: WARN  <fd:32 NSSNextPasswd> base.objecthelper.user AD user 'icoenegr'[uid=26081] conflicts with user 'icoenegr'[uid=21521] in local protected password database. Continuing without overwriting.
Dec 13 07:02:30 jpnqxic1 adclient[24430]: WARN  <fd:32 NSSNextPasswd> base.objecthelper.user AD user 'rparsons'[uid=24540] conflicts with user 'rparsons'[uid=171] in local protected password database. Continuing without overwriting.
Dec 13 07:02:28 jpnqxic1 su: + tty?? root-sentrigo
Dec 13 07:12:30 jpnqxic1 su: + tty?? root-sentrigo
Dec 13 07:12:30 jpnqxic1 su: + tty?? root-sfmdb
Dec 13 07:18:13 jpnqxic1 su: Can't rewrite protected password entry for user sentrigo
Dec 13 07:18:46 jpnqxic1 su: + tty?? root-sfmdb
Dec 13 07:19:48 jpnqxic1 su: + tty?? root-sentrigo
Dec 13 07:20:31 jpnqxic1  above message repeats 16 times
Dec 13 07:24:04 jpnqxic1 su: + tty?? root-sentrigo
Dec 13 07:33:40 jpnqxic1 adclient[24430]: WARN  <fd:32 PAMVerifyPassword> audit User 'sverma8' not authenticated: bad password
Dec 13 07:31:22 jpnqxic1 su: + tty?? root-sentrigo
Dec 13 07:34:39 jpnqxic1  above message repeats 11 times
Dec 13 07:34:39 jpnqxic1 adclient[24430]: INFO  <fd:32 PAMIsUserAllowedAccess> audit User 'sverma8' is authorized
Dec 13 07:36:38 jpnqxic1 su: + tty?? root-sentrigo
Dec 13 07:37:49 jpnqxic1 sshd[11058]: fatal: Timeout before authentication for 10.150.163.105
Dec 13 07:39:57 jpnqxic1 su: + tty?? root-sentrigo
Dec 13 07:40:31 jpnqxic1  above message repeats 6 times
Dec 13 07:41:59 jpnqxic1 su: + tty?? root-sentrigo
Dec 13 07:42:59 jpnqxic1 sshd[13074]: fatal: Timeout before authentication for 10.15.23.159
Dec 13 07:43:03 jpnqxic1 adclient[24430]: WARN  <fd:32 NSSGetPasswdDataByName> base.bind.cache LDAP search CN=Users,CN=JPNQXIC1-HPUX-NA,CN=Zones,CN=Centrify,CN=Program Data,DC=jnj,DC=com:(&(keywords=upn:(unknown user))(displayName=$CimsUserVersion3)) threw unexpected exception: search : Bad search filter
Dec 13 07:43:03 jpnqxic1 adclient[24430]: WARN  <fd:32 NSSGetPrPasswdDataByName> base.bind.cache LDAP search CN=Users,CN=JPNQXIC1-HPUX-NA,CN=Zones,CN=Centrify,CN=Program Data,DC=jnj,DC=com:(&(keywords=upn:(unknown user))(displayName=$CimsUserVersion3)) threw unexpected exception: search : Bad search filter
Dec 13 07:46:39 jpnqxic1 adclient[24430]: WARN  <fd:32 NSSGetPrPasswdDataByName> base.objecthelper.user AD user 'rparsons'[uid=24540] conflicts with user 'rparsons'[uid=171] in local protected password database. Continuing without overwriting.
Dec 13 07:51:40 jpnqxic1 adclient[24430]: ERROR <gpworker> gp.monitored Group Policy:  Timeout waiting for /usr/share/centrifydc/mappers/runmappers to complete
Dec 13 07:52:44 jpnqxic1 runmappers[14454]: /usr/share/centrifydc/mappers/machine/centrifydc.conf.pl taking too long, killing
Dec 13 07:52:47 jpnqxic1 runmappers[14454]: /usr/share/centrifydc/mappers/machine/sudoers.pl map refresh: Exit status 17. A problem may have occurred.
Dec 13 08:00:25 jpnqxic1 LVM[15485]: lvlnboot -v
Dec 13 07:56:41 jpnqxic1 su: + tty?? root-sentrigo
Dec 13 08:00:31 jpnqxic1  above message repeats 13 times
Dec 13 08:00:35 jpnqxic1 su: + tty?? root-sentrigo
Dec 13 08:01:52 jpnqxic1 adclient[24430]: WARN  <fd:32 NSSNextPasswd> base.objecthelper.user AD user 'icoenegr'[uid=26081] conflicts with user 'icoenegr'[uid=21521] in local protected password database. Continuing without overwriting.
Dec 13 08:01:55 jpnqxic1 adclient[24430]: WARN  <fd:32 NSSNextPasswd> base.objecthelper.user AD user 'tfoster9'[uid=26078] conflicts with user 'tfoster9'[uid=23070] in local protected password database. Continuing without overwriting.
Dec 13 08:20:29 jpnqxic1 su: + tty?? root-sfmdb
Dec 13 08:20:29 jpnqxic1 su: + tty?? root-sentrigo
Dec 13 08:20:31 jpnqxic1  above message repeats 23 times
Dec 13 08:22:28 jpnqxic1 su: + tty?? root-sentrigo
Dec 13 08:22:28 jpnqxic1 su: + tty?? root-sfmdb
Dec 12 18:24:45 jpnqxic1 syslog: icapd: ERROR: The following low-level error occurred:ERROR: The following low-level WBEM error occurred: PGS00414: Connection timed out.
Dec 13 08:37:11 jpnqxic1 su: + tty?? root-sentrigo
Dec 13 08:40:31 jpnqxic1  above message repeats 15 times
Dec 13 08:40:39 jpnqxic1 su: + tty?? root-sentrigo
Dec 13 08:41:18 jpnqxic1 su: Can't rewrite protected password entry for user sentrigo
Dec 13 08:44:10 jpnqxic1 sshd[25701]: reverse mapping checking getaddrinfo for vpnclient.wks.jnj.com failed - POSSIBLE BREAK-IN ATTEMPT!
Dec 13 08:44:14 jpnqxic1 adclient[24430]: INFO  <fd:32 PAMVerifyPassword> audit User 'rreddy14' authenticated based on Kerberos exchange to AD
Dec 13 08:44:14 jpnqxic1 adclient[24430]: INFO  <fd:33 PAMIsUserAllowedAccess> audit User 'rreddy14' is authorized
Dec 13 08:44:00 jpnqxic1 su: + tty?? root-sentrigo
Dec 13 08:44:15 jpnqxic1  above message repeats 6 times
Dec 13 08:44:15 jpnqxic1 sshd[25701]: Accepted keyboard-interactive/pam for rreddy14 from 172.17.129.130 port 1740 ssh2
Dec 13 08:47:20 jpnqxic1 su: + tty?? root-sentrigo
Dec 13 08:52:41 jpnqxic1 su: + tty?? root-sfmdb
Dec 13 08:53:41 jpnqxic1 su: + tty?? root-sfmdb
Dec 13 08:58:47 jpnqxic1 su: + tty?? root-sentrigo
Dec 13 09:00:31 jpnqxic1  above message repeats 13 times
Dec 13 09:00:59 jpnqxic1 su: + tty?? root-sentrigo
Dec 13 09:05:32 jpnqxic1 sshd[580]: fatal: Timeout before authentication for 10.150.163.105
Dec 13 09:10:29 jpnqxic1 sshd[884]: warning: can't get client address: Connection reset by peer
Dec 13 09:10:29 jpnqxic1 sshd[884]: Could not write ident string to UNKNOWN
Dec 13 09:11:16 jpnqxic1 vmunix: SCSI Ultra320 2/0/4/1/0 instance 4:       IO Type : SCSI IO has timed-out.       Target ID: 8, LUN ID: 0.       Write Command - CDB: 2a 00 03 cc 61 72 00 00 10 00
Dec 13 09:05:40 jpnqxic1 su: + tty?? root-sentrigo
Dec 13 09:11:18 jpnqxic1  above message repeats 6 times
Dec 13 09:11:18 jpnqxic1 sshd[890]: Did not receive identification string from 10.150.163.105
Dec 13 09:11:18 jpnqxic1 sshd[1017]: Did not receive identification string from 10.150.163.105
Dec 13 09:11:18 jpnqxic1 su: + tty?? root-sentrigo
Dec 13 09:11:44 jpnqxic1 su: Can't rewrite protected password entry for user sentrigo
Dec 13 09:20:30 jpnqxic1 su: + tty?? root-sentrigo
Dec 13 09:20:31 jpnqxic1  above message repeats 13 times
Dec 13 09:12:10 jpnqxic1 su: Can't rewrite protected password entry for user sentrigo
Dec 13 09:20:31 jpnqxic1  above message repeats 2 times
Dec 13 09:25:46 jpnqxic1 sshd[2527]: Did not receive identification string from 10.150.163.105
Dec 13 09:27:37 jpnqxic1 su: + tty?? root-sentrigo
Dec 13 09:28:37 jpnqxic1 sshd[2535]: fatal: Timeout before authentication for 10.150.163.105
Dec 13 09:29:37 jpnqxic1 adclient[24430]: WARN  <fd:34 NSSGetPasswdDataByName> base.bind.cache LDAP search CN=Users,CN=JPNQXIC1-HPUX-NA,CN=Zones,CN=Centrify,CN=Program Data,DC=jnj,DC=com:(&(keywords=upn:(unknown user))(displayName=$CimsUserVersion3)) threw unexpected exception: search : Bad search filter
Dec 13 09:29:37 jpnqxic1 adclient[24430]: WARN  <fd:33 NSSGetPrPasswdDataByName> base.bind.cache LDAP search CN=Users,CN=JPNQXIC1-HPUX-NA,CN=Zones,CN=Centrify,CN=Program Data,DC=jnj,DC=com:(&(keywords=upn:(unknown user))(displayName=$CimsUserVersion3)) threw unexpected exception: search : Bad search filter
Dec 13 09:27:38 jpnqxic1 su: + tty?? root-sentrigo
Dec 13 09:30:35 jpnqxic1  above message repeats 4 times
Dec 13 09:29:37 jpnqxic1 adclient[24430]: WARN  <fd:32 NSSGetPasswdDataByName> base.bind.cache LDAP search CN=Users,CN=JPNQXIC1-HPUX-NA,CN=Zones,CN=Centrify,CN=Program Data,DC=jnj,DC=com:(&(keywords=upn:(unknown user))(displayName=$CimsUserVersion3)) threw unexpected exception: search : Bad search filter
Dec 13 09:30:36 jpnqxic1 su: + tty?? root-sentrigo
Dec 13 09:37:50 jpnqxic1 sshd[4770]: Did not receive identification string from 10.150.163.105
Dec 13 09:38:49 jpnqxic1 su: Can't rewrite protected password entry for user sentrigo
Dec 13 09:39:47 jpnqxic1 su: + tty?? root-sentrigo
Dec 13 09:40:35 jpnqxic1  above message repeats 8 times
[root@jpnqxic1:/var/adm/syslog]#
#->
[root@jpnqxic1:/var/adm/syslog]#
#->

Dennis Handly
Acclaimed Contributor

Re: Server is **bleep** slow and taking too much in login

glance shows your system is 97% CPU.  I'm not sure this is due to your bad disk.

What does top show as the top busy processes?

What model server do you have and how many CPUs?