problem with accented characters : ksh ok, sh ko.

[Joel Girot]

hi all
On my hp9000 11iv1, French accented characters are correctly display whith ksh shell, not with sh.
Does someone know the reason ?
Here is example :
$ locale
LANG=
LC_CTYPE="C"
LC_COLLATE="C"
LC_MONETARY="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_MESSAGES="C"
LC_ALL=
$ locale LC_CTYPE
"0"
"0"
"roman8"
""
$ echo $SHELL
/usr/bin/sh
$ echo "^?^?^? : three e accent" > toto
$ cat toto
ééé : three e accent
$ echo "^?^?^?^?^?^?"
éééààà
$
$ ksh
gvr/joel$ echo "éééààà"
éééààà
gvr/joel$ ééé
ksh: ééé: not found
gvr/joel$ exit
$ ^?^?^?
sh: ^?^?^?: not found.
$

many thanks

Joel

[Muthukumar_5]

Just do as,

# stty erase ^?

hth.

[Luk Vandenbussche]

Hi,

Try with stty cs7 or stty cs8

Your emulation must be able to handle with this

[Joel Girot]

Windows terminal emulation is wrq reflection. Terminal type hp70092, hp70096.
Host characters set HP Roman 8.

on hpux :
$ echo $TERM
70092

$ stty -a
speed 9600 baud; line = 0;
rows = 24; columns = 80
min = 4; time = 0;
intr = ^C; quit = ^\; erase = ^H; kill = ^U
eof = ^D; eol = ^@; eol2 = ^@; swtch
stop = ^S; start = ^Q; susp = ^Z; dsusp
werase = ^W; lnext = ^V
-parenb -parodd cs8 -cstopb hupcl cread -clocal -loblk -crts
-ignbrk brkint ignpar -parmrk -inpck -istrip -inlcr -igncr icrnl -iuclc
ixon -ixany ixoff -imaxbel -rtsxoff -ctsxon -ienqak
isig icanon -iexten -xcase echo echoe echok echonl noflsh
-echoctl -echoprt -echoke -flusho -pendin
opost -olcuc onlcr -ocrnl -onocr -onlret -ofill -ofdel -tostop tab3

with stty cs7 or cs8 or erase ^ no change for the better

with shell sh éééààà (three eeeaaa accent) echoed as :
$ ^ ^ ^ ^ ^ ^ > toto 2>&1
$ cat toto
sh: ^ ^ ^ ^ ^ ^ : not found.
$ od -xc toto
0000000 7368 3a20 5e85 5e85 5e85 5e88 5e88 5e88
s h : ^ 205 ^ 205 ^ 205 ^ 210 ^ 210 ^ 210
0000020 3a20 206e 6f74 2066 6f75 6e64 2e0a
: n o t f o u n d . \n
0000036

with shell ksh it's good :
$ ksh
$
$ éééààà > toto 2>&1
$ od -xc toto
0000000 6b73 683a 20c5 c5c5 c8c8 c83a 2020 6e6f
k s h : 305 305 305 310 310 310 : n o
0000020 7420 666f 756e 640a
t f o u n d \n
0000030

[Stephen Keane]

Try

# echo '\0305\0305\0305\0310\0310\0310' > toto

# cat toto
# od -xc toto

works in ksh and sh on my server.

[Stephen Keane]

Also, HOW are you generating the extended character set characters? Perhaps the key sequence you are using has special meaning in sh but not so in ksh?

[CAS_2]

For me, the output of your commands posted here is a mess.

Please, in order to debug this problem, run then following command lines and post their outputs:

# (1) using sh
echo "é" | xd -tx1

# (2) using ksh
echo "é" | xd -tx1

[Stephen Keane]

I can't do that because I don't have a Spanish keyboard. The characters you are typing in probably display differently on the screen when I see your posts too as I don't have a Spanish language pack loaded on my PC either.

What I think you are typing is:

o
A

i.e. A with a small circle above it or inOctal 0305

and

^
E

i.e. E with a carat '^' above it.

What I actually see on my screen is

/
e

i.e. a small e with an acute accent.

[CAS_2]

Anyway... I knew this issue of the POSIX shell.
I tested in my box.

The issue is related to LANG environment variable. One you are logged run:

/usr/bin/ksh
echo "é" # é is stressed 'e' (I noticed this forum only works using 7bit chars)
exit # returning to original shell

export LANG=fr_FR.roman8
/usr/bin/sh
echo "é"

[Joel Girot]

tested in my box :

# with sh :

$ echo "^ " | xd -tx1
0000000 c5 a
0000002
$
$ echo "^ "
é
$

# e with accent is echoed like ^ when I type the echo command
# but the display on stdout off this command is correct (e with accent) ?
# when I use vi with sh : all is correct


# with ksh, all is correct :

$ ksh
$ echo "é" | xd -tx1
0000000 c5 a
0000002
$

same observation with a basic terminal hp 700/96

(Note : when I preview this post my browser display A with tilde + the copyright small circle in place off e with accent on my hpux screen)

Joel

[Stephen Keane]

HOW are you generating the extended characters? Which key sequence are you using?

Try

# echo '\0351'
é

I get a lowercase e with an acute accent, octal character 351 which in ISO8859 is correct. You seem to be typing octal character 305 which in ISO8859 is NEVER a lowercase e with an acute accent. It is usually an uppercase A with various symbols above it (umlauts, small circle etc.)

[Joel Girot]

I generate the extended characters by typing on the letter e with an acute accent on my french keyboard. I dont know which key sequence is generated but it is probably
\0305.

when I try
$ echo "\0351"
Ã
I see on my hpux screen an uppershift O with a curious hat.

when I try
$ echo "\0305"
é

I see on my screen a correct e with an acute accent.

(On this forum my browser display A with tilde + the copyright small circle)

[CAS_2]

Please, Joel.

Have you tested my second proposal ?

I use Reflection HP terminal emulation to connect to LAN consoles of my boxes.
I tested this issue using the LAN console of my boxes. It happens as you explained:

<< Bear in mind that ITRC forums don't handle properly the 8 bit chars. So, in the lines below 'é' is quoted stressed e >>

Run:

$ stty -parenb -istrip cs8
echo 'é' # At this point, stressed e is displayed in the Reflection HP terminal emulator as a ^ char followed by a blank.

/usr/bin/ksh
echo 'é' # stressed e is properly displayed
exit

export LANG=fr_FR.roman8
echo 'é' # stressed e SHOULD BE PROPERLY displayed

[CAS_2]

Excuse me. The last pair of lines should be:

export LANG=fr_FR.roman8
/usr/bin/sh
echo 'à ©' # stressed e SHOULD BE PROPERLY displayed

[Joel Girot]

That's right CAS, but set LANG at other value than the default (C) is the cause of others disadvantages, french notation of some commands disrupt some program. example
août in place of aug ...
$ ll
total 477320
-rw-r----- 1 joel gvr 90 août 17 16:31 alias_a_teste

I just want to understand why ksh work whith
default LANG=C and sh don't...

[CAS_2]

Once the sh is properly running, you should type:

unset LANG

and execute those programs.

[Joel Girot]

Thank you all