- Community Home
- >
- Servers and Operating Systems
- >
- Operating Systems
- >
- Operating System - Linux
- >
- Re: Need Help Interpreting I/O Error Messages
Categories
Company
Local Language
Forums
Discussions
Forums
- Data Protection and Retention
- Entry Storage Systems
- Legacy
- Midrange and Enterprise Storage
- Storage Networking
- HPE Nimble Storage
Discussions
Discussions
Discussions
Forums
Forums
Discussions
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
- BladeSystem Infrastructure and Application Solutions
- Appliance Servers
- Alpha Servers
- BackOffice Products
- Internet Products
- HPE 9000 and HPE e3000 Servers
- Networking
- Netservers
- Secure OS Software for Linux
- Server Management (Insight Manager 7)
- Windows Server 2003
- Operating System - Tru64 Unix
- ProLiant Deployment and Provisioning
- Linux-Based Community / Regional
- Microsoft System Center Integration
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Community
Resources
Forums
Blogs
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО04-07-2011 09:09 AM
тАО04-07-2011 09:09 AM
Linux rsnperf 2.6.18-92.1.1.el5 #1 SMP Thu May 22 09:01:47 EDT 2008 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 5.2 (Tikanga)
*********************************************
Problem:
This server seems to be throwing out
scsi errors. It is currently hooked to a
SAN via a SAN switch. Most recently, the firmware to the SAN switch was updated. A review of the server logs doesn't seem to indicate any I/O errors *BEFORE* the firmware
update. The following log message occurred
*AFTER* the switch firmware upgrade:
SCSI error: return code = 0x00010000
end_request: I/O error, dev sdg, sector 524287992
sd 1:0:1:3: SCSI error: return code = 0x00010000
end_request: I/O error, dev sdg, sector 0
sd 1:0:1:4: SCSI error: return code = 0x00010000
end_request: I/O error, dev sdh, sector 0
sd 1:0:1:4: SCSI error: return code = 0x00010000
end_request: I/O error, dev sdh, sector 0
sd 1:0:1:4: SCSI error: return code = 0x00010000
end_request: I/O error, dev sdh, sector 524287992
sd 1:0:1:4: SCSI error: return code = 0x00010000
end_request: I/O error, dev sdh, sector 524287992
sd 1:0:1:4: SCSI error: return code = 0x00010000
end_request: I/O error, dev sdh, sector 0
Also, the following partial vgdisply -v output
seems to confirm the error messages from the logs (listed above):
# vgdisplay -v
Finding all volume groups
/dev/sda: read failed after 0 of 4096 at 0: Input/output error
/dev/sdb: read failed after 0 of 4096 at 0: Input/output error
/dev/sdc: read failed after 0 of 4096 at 0: Input/output error
/dev/sdd: read failed after 0 of 4096 at 0: Input/output error
/dev/sde: read failed after 0 of 4096 at 0: Input/output error
/dev/sdf: read failed after 0 of 4096 at 0: Input/output error
/dev/sdg: read failed after 0 of 4096 at 0: Input/output error
/dev/sdh: read failed after 0 of 4096 at 0: Input/output error
Found duplicate PV xjjRbOG92EcZA4tIWdCgCTg1hfYPl2w1: using /dev/sdm not /dev/sdi
Found duplicate PV 0MKA80XY7DWOKcmIgohS6l8IENebjUgT: using /dev/sdn not /dev/sdj
Found duplicate PV u2VZKuhxRDOVCXunN0czzOvaF4liqmZg: using /dev/sdo not /dev/sdk
Found duplicate PV LEYBZCSyOltHJhpojFE7vaZruMDpXBio: using /dev/sdp not /dev/sdl
Partial output of dmesg:
end_request: I/O error, dev sdf, sector 0
sd 1:0:1:2: SCSI error: return code = 0x00010000
end_request: I/O error, dev sdf, sector 0
sd 1:0:1:2: SCSI error: return code = 0x00010000
end_request: I/O error, dev sdf, sector 524287992
Has anyone seen this before or can someone point me in the right direction to begin troubleshooting this? Thanks in advance!
robs
Solved! Go to Solution.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО04-07-2011 10:19 AM
тАО04-07-2011 10:19 AM
Re: Need Help Interpreting I/O Error Messages
Also, if you haven't configured multipath, during firmware upgrade and controller reboots, you will lose access to your disks.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО04-07-2011 12:57 PM
тАО04-07-2011 12:57 PM
Re: Need Help Interpreting I/O Error Messages
Thanks for your earlier reply!
There was an eva firmware upgrade at the time of the incident. Several other servers encountered the event but were able to recover. Not knowing the specifics of how or if multipath was configured, it appears that multipath is not configured or not configured properly on the server.
Among other things:
the multipath daemon is not configured to start on startup and is not running
log $ /sbin/chkconfig --list multipathd
multipathd 0:off 1:off 2:off 3:off 4:off 5:off 6:off
# /sbin/service multipathd status
multipathd is stopped
the /etc/multipath.conf file does not appear to have been touched and blacklists all devices:
etc $ more multipath.conf
# This is a basic configuration file with some examples, for device mapper
. . .
blacklist {
devnode "*"
}
. . .
not sure, but ├в multipath ├в ll ├в showing no output also indicates that multipath is not configured .
Is it possible that a reboot of the system would re-acquire the LUN connections?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО04-08-2011 10:29 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО04-08-2011 02:23 PM
тАО04-08-2011 02:23 PM
Re: Need Help Interpreting I/O Error Messages
kernel: rport-1:0-7: blocked FC remote port time out: saving binding
kernel: rport-1:0-6: blocked FC remote port time out: saving binding
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО04-14-2011 03:28 PM
тАО04-14-2011 03:28 PM