topic Re: discovered an issue when performing a remote tr (translate) command on hpux 11v3. in Operating System - HP-UX

discovered an issue when performing a remote tr (translate) command on hpux 11v3.

Manuel Contreras — Fri, 20 Aug 2010 15:51:16 GMT

recently discovered an issue when performing a remote tr (translate) command on hpux 11v3.

test results listed below (same test.file exists on both boxes)...

$ remsh goodBOX -n " head -1 test.file | cut -c60-69 | tr '[a-z]' '[A-Z]'>
OABMIS02

$ remsh badBOX -n " head -1 test.file | cut -c60-69 | tr '[a-z]' '[A-Z]'"
��hr02

without the tr:
$ remsh badBOX -n " head -1 165700047_20100728_095914 | cut -c60-69 "
OABMIS02

the above command works correctly when performed locally...

already checked the tr executable...same on both boxes:

> ll /usr/bin/tr
-r-xr-xr-x 1 bin bin 76508 Feb 15 2007 /usr/bin/tr*

> cksum /usr/bin/tr
3992625266 76508 /usr/bin/tr

largest difference between boxes is goodBOX has a 2007 quality pack, and the badBOX is 2008.

input is appreciated...
manny

Re: discovered an issue when performing a remote tr (translate) command on hpux 11v3.

Manuel Contreras — Fri, 20 Aug 2010 16:31:56 GMT

thought I would provide another example:

$ remsh badBOX "echo abCd | tr '[a-z]' '[A-Z]'"
ABbD

$ remsh goodBOX "echo abCd | tr '[a-z]' '[A-Z]'"
ABCD

Re: discovered an issue when performing a remote tr (translate) command on hpux 11v3.

Dennis Handly — Fri, 20 Aug 2010 16:55:14 GMT

>the above command works correctly when performed locally.

In the same directory? Also, try using the absolute path to tr(1).

Since you are using '', it can't be filename completion that is expanding your strings.

It could possibly be your locale settings?

Re: discovered an issue when performing a remote tr (translate) command on hpux 11v3.

Manuel Contreras — Fri, 20 Aug 2010 17:04:48 GMT

I like the latest echo example, as it simplifies the test.

locally works fine:
> echo abCd | /usr/bin/tr '[a-z]' '[A-Z]'
ABCD

remotely does not :(
$ remsh badBOX "echo abCd | /usr/bin/tr '[a-z]' '[A-Z]'"
ABbD

does work on goodBOX:
$ remsh goodBOX "echo abCd | /usr/bin/tr '[a-z]' '[A-Z]'"
ABCD

both boxes are 11.31

Re: discovered an issue when performing a remote tr (translate) command on hpux 11v3.

Dennis Handly — Fri, 20 Aug 2010 17:14:45 GMT

What happens if you invoke a remote script like:
#!/usr/bin/sh
env
echo "The tr output ====="
echo 'abCd' | /usr/bin/tr '[a-z]' '[A-Z]'

Re: discovered an issue when performing a remote tr (translate) command on hpux 11v3.

Manuel Contreras — Fri, 20 Aug 2010 17:25:52 GMT

$ remsh badBOX "/home/is/testuser/test.sh"
_=/usr/bin/env
LANG=en_US.iso88591
PATH=/usr/bin:/usr/ccs/bin:/usr/bin/X11:/usr/contrib/bin:/usr/local/bin:
LOGNAME=testuser
SHELL=/usr/bin/ksh
HOME=/home/is/testuser
PWD=/home/is/testuser
TZ=PST8PDT
The tr output =====
ABbD
$

Re: discovered an issue when performing a remote tr (translate) command on hpux 11v3.

Matti_Kurkela — Fri, 20 Aug 2010 20:16:29 GMT

Several possible causes:

1.) tr makes a wrong conversion on plain US-ASCII character "C", which would be pretty weird indeed.

2.) tr works OK, but something in the data transmission chain converts the uppercase C to lowercase b (or to something else that ultimately ends up displayed as lowercase b). Strange but possible, if your terminal/emulator includes key macro features.

3.) something else, more information needed.

To test (and possibly exclude) hypothesis 2) and to get more information, we need a way to express the characters without directly inputting them. The "printf" command and the ASCII code table (see "man 5 ascii") are helpful here.

Connection sanity check in remote -> local direction:

remsh badBOX 'printf "\\141\\142\\103\\144\\n"'

(output should be "abCd"; if it's something else, the output of badBOX is being modified on the way from the badBOX to the local display.)

Connection sanity check in local -> remote direction:
remsh badBOX "echo abCd | od -b"

(output should include "141 142 103 144 012" which is abCd + line termination character expressed in octal ASCII codes; if the codes are different, your input is being modified on the way from local to badBOX.)

Explicit remote tr sanity check:

remsh badBOX 'printf "\\141\\142\\103\\144\\n" | /usr/bin/tr "[a-z]" "[A-Z]" | od -b'

(output should include "101 102 103 104 012" which is ABCD + line termination character expressed in octal ASCII codes)

MK

Re: discovered an issue when performing a remote tr (translate) command on hpux 11v3.

Manuel Contreras — Fri, 20 Aug 2010 20:37:45 GMT

To test (and possibly exclude) hypothesis 2) and to get more information, we need a way to express the characters without directly inputting them. The "printf" command and the ASCII code table (see "man 5 ascii") are helpful here.

Connection sanity check in remote -> local direction:

remsh badBOX 'printf "\\141\\142\\103\\144\\n"'

(output should be "abCd"; if it's something else, the output of badBOX is being modified on the way from the badBOX to the local display.)

$ remsh badBOX 'printf "\\141\\142\\103\\144\\n"'
abCd

Connection sanity check in local -> remote direction:
remsh badBOX "echo abCd | od -b"

(output should include "141 142 103 144 012" which is abCd + line termination character expressed in octal ASCII codes; if the codes are different, your input is being modified on the way from local to badBOX.)

$ remsh badBOX "echo abCd | od -b"
0000000 141 142 103 144 012
0000005

Explicit remote tr sanity check:

remsh badBOX 'printf "\\141\\142\\103\\144\\n" | /usr/bin/tr "[a-z]" "[A-Z]" | od -b'

(output should include "101 102 103 104 012" which is ABCD + line termination character expressed in octal ASCII codes)

$ remsh badBOX 'printf "\\141\\142\\103\\144\\n" | /usr/bin/tr "[a-z]" "[A-Z]" | od -b'
0000000 101 102 142 104 012
0000005

$ remsh goodBOX 'printf "\\141\\142\\103\\144\\n" | /usr/bin/tr "[a-z]" "[A-Z]" | od -b'
0000000 101 102 103 104 012
0000005

Matti, thanks for your Extended analysis.

Re: discovered an issue when performing a remote tr (translate) command on hpux 11v3.

Dennis Handly — Fri, 20 Aug 2010 23:25:32 GMT

>MK: Connection sanity check in local -> remote direction:
>remsh badBOX "echo abCd | od -b"

A better output is hex, not octal: echo abCd | xd -tx1 -tc"

What architecture/model are the two boxes?

Perhaps the next check is to do all this on the badBOX in your home directory. Invoke the script with:
env -i ./script

#!/usr/bin/ksh

export _=/usr/bin/env
export LANG=en_US.iso88591
export PATH=/usr/bin:/usr/ccs/bin:/usr/bin/X11:/usr/contrib/bin:/usr/local/bin:
export LOGNAME=testuser
export SHELL=/usr/bin/ksh
export HOME=/home/is/testuser
export PWD=/home/is/testuser
export TZ=PST8PDT

echo 'abCd' | /usr/bin/tr '[a-z]' '[A-Z]' | xd -tx1 tc

Other next steps could be putting "abCd" in a file, rather than using echo:
/usr/bin/tr '[a-z]' '[A-Z]' < file | xd -tx1 tc

Re: discovered an issue when performing a remote tr (translate) command on hpux 11v3.

Raj D. — Fri, 20 Aug 2010 23:42:43 GMT

Manuel,

What happens if you try with ssh , instead of remsh :

$ ssh -q badBOX "echo abCd | tr '[a-z]' '[A-Z]'"

do you get ABCD , the correct output.
- may be some flaws over remsh services.

Hth,
Raj.

Re: discovered an issue when performing a remote tr (translate) command on hpux 11v3.

Matti_Kurkela — Sat, 21 Aug 2010 08:33:35 GMT

The connection sanity checks were OK, so it probably isn't a remsh problem.

$ remsh badBOX 'printf "\\141\\142\\103\\144\\n" | /usr/bin/tr "[a-z]" "[A-Z]" | od -b'
0000000 101 102 142 104 012
0000005

OK, so it definitely looks like a problem with tr. But on second thought, it might be a locale problem too.

One more test:

remsh badBOX 'echo abCd | LC_ALL=C tr "[a-z]" "[A-Z]"'

Setting LC_ALL=C makes all locale-specific processing fall back to POSIX standrd US-ASCII. If this makes the problem go away, then perhaps the locale files for your default locale en_US.iso88591 (located in /usr/lib/nls/loc/) are broken on badBOX.

Perhaps someone with root access has experimented with the "localedef" command and made a little mistake?

You might also copy /usr/bin/tr from goodBOX to /tmp/tr on the badBOX, then run the tests using /tmp/tr instead of /usr/bin/tr. If this fixes the problem, then it conclusively proves that badBOX's /usr/bin/tr is broken.

------

Dennis> A better output is hex, not octal: echo abCd | xd -tx1 -tc"

Maybe, assuming that you are already familiar with hex values of ASCII characters. I would have used it if printf(1) could accept hex escapes.

But as it can't, octal is unavoidable. And as I'm already potentially confusing people, having input in octal and output in hex would make it even harder to figure out what's going on if the reader is not already familiar with all the tools and concepts I used.

MK

Re: discovered an issue when performing a remote tr (translate) command on hpux 11v3.

Dennis Handly — Sun, 22 Aug 2010 05:42:26 GMT

>MK: it might be a locale problem too.

That's why I had the ENVs printed.

>You might also copy /usr/bin/tr from goodBOX to /tmp/tr on the badBOX

Manny said the cksum matched. But libc and the locale libs could be different.

>assuming that you are already familiar with hex values of ASCII characters.

That's why there is ascii(5) with octal and hex.

>octal is unavoidable.

You could put the chars in a file and do a hex dump on them to prove they're correct.

Re: discovered an issue when performing a remote tr (translate) command on hpux 11v3.

Manuel Contreras — Wed, 25 Aug 2010 16:25:38 GMT

your not going to believe the root cause:

# remsh badBOX -l testuser "echo abCd | tr '[a-z]' '[A-Z]'"
ABbD

# remsh badBOX -l testuser "unset LANG;echo abCd | tr '[a-z]' '[A-Z]'"
ABCD
#

thank you all...your input is always appreciated :)

Re: discovered an issue when performing a remote tr (translate) command on hpux 11v3.

Manuel Contreras — Wed, 25 Aug 2010 16:27:08 GMT

the LANG setting was the root cause for my remote tr issues.

# remsh badBOX -l testuser "echo abCd | tr '[a-z]' '[A-Z]'"
ABbD

# remsh badBOX -l testuser "unset LANG;echo abCd | tr '[a-z]' '[A-Z]'"
ABCD

the following setting does not exist on the goodBOX:

> cat /etc/rc.config.d/LANG
#!/sbin/sh
# @(#)B.11.31_LR
# Language preference. See lang(5), hpnls(5)
#
# LANG: Locale name
#
# Note: if using the default C locale, many commands will execute faster if
# LANG is not set.
#
#LANG=C
#export LANG
LANG=en_US.iso88591

Re: discovered an issue when performing a remote tr (translate) command on hpux 11v3.

Dennis Handly — Thu, 26 Aug 2010 07:16:35 GMT

>the LANG setting was the root cause for my remote tr issues.
echo abCd | tr '[a-z]' '[A-Z]'

Ah. If you want to do this, you have to be in the American Nerd locale, not this goofy dictionary sort locale.

Basically what you asked for was:
a B b .. Z z converted to:
A AE a ... Z

See /usr/lib/nls/loc/src/en_US.iso1.src LC_COLLATE category.

From the man page, you probably want:
tr '[:lower:]' '[:upper:]'