Trying to capture dd and pv output in a file

[Tom Wolf_3]

Hello everyone.

I'm trying to capture all output of the following command to a file.

dd if=/dev/zero | pv | dd of=/xtra/dd_bench_1.img bs=512000k count=400000

Tried re-directing standard output but that didn't work.

dd if=/dev/zero | pv | dd of=/xtra/dd_bench_1.img bs=512000k count=400000 >/var/tmp/output.txt 2>&1

I'm sure I'm overlooking something simple.

Please advise.

Thank you.

[Steven Schweda]

> Tried re-directing standard output but that didn't work.

   "didn't work" is not a particularly useful description of what
happened, or what you didn't like about it.

   Given that you have a pipeline of three commands (dd, pv, dd), and
only the last stage has any redirection, I'll guess that you're seeing
output to stderr from the first two stages.  You might try using a
sub-shell to collect the output from all the stages.  For example:

   ( dd blah1 | pv | dd blah2 ) > file 2>&1

[Tom Wolf_3]

Hello Steven.

Thanks for replying.

I apologize.

You're right, my comment "didn't work" is rather vague.

Specifically, when we run this command without redirecting anything it writes the following to stdout (the screen).

# dd if=/dev/zero | pv | dd of=/xtra/dd_bench_1.img bs=512000k count=400000
0444MiB 0:00:19 [23.4MiB/s] [ <=> ]
+400000 records in.
0+400000 records out.

We want ALL of that output, especially the MiB/s data, to be written to a file.

When we try your suggestion, something like this:

# (dd if=/dev/zero | pv | dd of=/xtra/dd_bench_1.img bs=512000k count=400000) > /var/tmp/output.file 2>&1

Only the "records in" and "records out" data is written to the output file /var/tmp/output.file as shown here:

# cat /var/tmp/output.file
0+400000 records in.
0+400000 records out.
dd: 0511-041 The process was killed by signal number 13.

The MiB/s data is discarded.

How do we write that MiB/s data to a file as well as the "records in" and "records out" info.

Please let me know if you have any suggestions.

Thanks again for replying.

-Tom

[Steven Schweda]

> 0444MiB 0:00:19 [23.4MiB/s] [ <=> ]

   I hate to sound ignorant, but, what produces that?  Also, what's
"pv"?  All I expect to see from "dd" are the "records in" and "records
out" messages.  Are you using some fancy "dd", or is that stuff coming
from "pv" (whatever that is), or what?

      type dd

   There are programs whose behavior depends on whether stdout/stderr is
a terminal.  Redirect them, and you get different results.

[Tom Wolf_3]

Hello again.

The dd command is standard.

# type dd
dd is /usr/bin/dd

The pv command shows the progress of data going through a pipeline.

That's the MiB output.

My organization has us benchmarking disk I/O (as well as CPU metrics) both before and after applying Unix OS spectre/meltdown fixes.

This dd command is being used to capture disk I/O benchmarking.

I'm sure there are better ways to do this but I'm just doing as instructed.

I'm trying to script this dd command and that's when I found I couldn't capture the pv output.

From the command line it's just fine, but trying to assign the dd/pv output to a file or variable does not work for me.

Any other suggestions are welcome.

Thanks.

[Steven Schweda]

> The pv command shows the progress of data going through a pipeline.

   Ok.  I found the source, and built one.  Its bouncing "<=>" thing
uses cursor positioning, which wouldn't look good when stored in a file.
Having only glanced at the source, I know nothing, but I'd bet a small
sum that if isatty(STDERR_FILENO) fails, then you won't see any of that
output, which was always intended for (only) an interactive user.
Certainly, "pv 2> file" seems to disable it.

> [...] trying to assign the dd/pv output to a file or variable does not
> work for me.

   You can redirect stderr from "dd" without data loss, but, apparently,
the "pv" progress stuff is provided only to a user at a terminal.
That's not unusual; For example, Wget has a similar bouncing "<=>" thing
for an interactive user, but a different-format progress report for
redirected output.

> Any other suggestions are welcome.

   Why not just time the command, and if you want a speed, divide the
byte count (which you're specifying) by the measured time?  It's not as
if you'd be watching the "pv" output while it's running, which is what
that eye-candy was intended for.

   man time

[Dennis Handly]

As Steven said, pv seems to be detecting how it's run.

Perhaps script(1) will fool it and capture the output?

Note: Using pv and pipelines will kill performance.  Pipes are limited to an 8 KB buffer, not your 512 MB writes.

>  both before and after applying Unix OS spectre/meltdown fixes.

This is a waste of time on HP-UX, no X86-64 here, unless over NFS.

[Tom Wolf_3]

Thanks again for the feedback.

I'm going to try the time command and divide the bytes by time of dd execution.

This is actually for an AIX server and not HP-UX.