topic Strange DD performance issue in Operating System - HP-UX

Strange DD performance issue

Chetan_5 — Fri, 08 Sep 2006 14:13:39 GMT

Recently ran into something peculiar that almost got system to a halt. I have 2 PA-RISC dome partitions in a cluster with large number of disk devices.

A monitoring job that runs through OVO, checks disk connectivity by running the dd command in this format

dd count=1 if=/dev/dsk/cXtYDZ of=/dev/null

Everytime the dd runs on a device which is active on the other node, resource utilization would shoot up dramatically. For disks active on the current node, it works like a charm.

I was just curious to know why this behaviour.

The other question that I have is what is a good way to check disk connectivity.

Acc. to HP, ioscan is not a good way to check it. At first I couldnt beleive this response until yesterday when we found out a PV status had changed to an unavailable status, with EMS pouring out error messages and ioscan showed the disk as CLAIMED.

So the rationale behind us using the DD command was to send IO to check the disk status. But it now seems that it has some overheads in a cluster environment.

Re: Strange DD performance issue

Steven E. Protter — Fri, 08 Sep 2006 14:18:55 GMT

Shalom,

I recommend using EMS to monitor disks. You can use sam to set it up and it won't touch resources not allocated to the partition its running in.

I also used to run a script that simply checked the syslog or dmesg output for the string lbolt.

That is a sign that either a hot swap was hot swapped or a disk has gone bad.

SEP

Re: Strange DD performance issue

IT_2007 — Fri, 08 Sep 2006 14:21:07 GMT

dd command won't tell you 100% sure that disk is bad or good. Sometimes it says good but disk may be bad due bad blocks.

It seems dd runs on the passive node and disk activity may be busy during IO activity from active node.

Re: Strange DD performance issue

A. Clay Stephenson — Fri, 08 Sep 2006 15:03:02 GMT

When multiple hosts access common SCSI devices all the hosts have to play by the rules. Each of the host's must use SCSI reserve/release commands to control access. This locks the drive for access by only one host. The other host gets a SCSI reservation conflict status until the current host sends a SCSI release cmd (or a SCSI bus reset is done). It's the waiting for the release that you are seeing.

It makes much more sense to use EMS and/or syslog to feed OV/O and that is much less intrusive.

Re: Strange DD performance issue

Chetan_5 — Fri, 08 Sep 2006 16:01:57 GMT

Thanks for all the replies, but Clays response is what I was looking for.

We already have EMS, syslog monitoring in place, but this script was born to quickly check SAN connectivity for disks in a complex environment like ours which had gone thru a couple of migrations.

Re: Strange DD performance issue

Chetan_5 — Fri, 08 Sep 2006 16:02:42 GMT