Reading UTF16 input from STDIN using fgetws()

[Srikrishna Erra]

Hi all,

I am trying to read UTF16 encoded data from stdin(as of now only thru file re-direction). I am writing a C program for this on HPUX.

I am using fgetws() for reading UTF16 Data.

Man pages of fgetws() says,
fgetws() converts the input string from the encoding either specified by "LC_CTYPE" or specified in fopen() to wide character string.

My intention is to read UTF16 input from STDIN.

For this i have written following C Program.

#include
#include
#include
#include

int main()
{
wchar_t buf[512]={0};
FILE* res=NULL;

res=fopen("/dev/stdin","r,ccs=UTF-16LE");
perror("File");
fgetws(buf,sizeof(buf),res);
printf("%s",buf);
return(0);
}

I was sucessfull in reading UTF16 from stdin on LINUX by passing special file "/dev/stdin".

Now I want this code to be ported to HPUX.

The problem is HPUX doesn't have "/dev/stdin" device file.

I have checked /dev/ directory and not able to find any reference of STDIN device file.

Later i have checked some commands.

I have checked man page of "mkfifo" command but it generates only file descriptor but i need filename i.e. pathname for STDIN device file as fopen() takes filename.

I have also checked man page of mknod command but there was no info about where the special files are created.

Please let me know the location of device file for STDIN in HPUX. So that i can use it in fopen().

regards,
Srikrishna Erra.

[James R. Ferguson · Accepted solution]

Hi:

> The problem is HPUX doesn't have "/dev/stdin" device file.

No, but you do have '/dev/tty' which is your controlling terminal.

Regards!

...JRF...

[Srikrishna Erra]

Hi,
Thanks for your reply.

I don't have expertise on controlling tty's.

I have executed my C program considering "/dev/tty" as the device file for stdin so that i can give iteractive input and provide input from a file re-direction.

Here is modified program

int main()
{
wchar_t buf[512]={0};
FILE* res=NULL;

res=fopen("/dev/tty","r,ccs=UTF-16LE");
perror("File");

fgetws(buf,sizeof(buf),res);
wprintf(buf,"%s");

return(0);

}


when exectuted, i am able to give interactive input. But if provided file re-direction then also it is prompting for interactive input.

# ./a.out
File: Error 0
Srikrishna
Srikrishna

# ./a.out < input_BE.txt
File: Error 0
srikrishna
srikrishna
#

Please let me know what should be done for supporting re-direction with having device filename "/dev/tty".

Thanks in Advance.

regards,
Srikrishna Erra.

[Dennis Handly]

res=fopen("/dev/stdin","r,ccs=UTF-16LE");
perror("File");
fgetws(buf,sizeof(buf),res);

Replace by:
You must export LANG or LC_CTYPE to the appropriate locale. Which is NOT UTF-16LE.
There are a bunch of *.utf8 locales.

Then you must call setlocale(3).

fgetws(buf, sizeof(buf), stdin); // use open file

>The problem is HP-UX doesn't have "/dev/stdin" device file.

You don't need it nor call fopen, just refer to stdin.

>Please let me know what should be done for supporting re-direction with having device filename "/dev/tty".

Redirection is done by the shell. If you want to read from stdin, read from stdin.

[Srikrishna Erra]

Hi,

I want to read the data in UTF16 form.
I didn't find any locale for UTF16.

That's why i am using above calls i.e. fopen function with passing ccs= parameter so that the input will considered in that form.

Please let me know how to set locale for UTF-16.

Thanks

regards,
Srikrishna Erra.

[Raj D.]

Srikrishna,


The doc mentioed UTF-8 support ptf driver , not mentioned about STDIN , please check out this document, hope this helps,

http://docs.hp.com/en/5187-1361/ch16s03.html



There is already question asked about supporting UTF-16 on hp-ux (what is your os ver.) , check this out:

http://forums11.itrc.hp.com/service/forums/questionanswer.do?threadId=1282647



On 11iv3 , it is mentioned that updated iconv converter added with 11iv3:
Several Asian character sets now have iconv
conversion support added to convert data directly from these character sets to and from various transformation formats of Unicode such as UTF-8, UTF-16 and UTF-32.
http://docs.hp.com/en/5992-4183/5992-4183.pdf



Hth,
Raj.

[Dennis Handly]

>That's why i am using above calls i.e. fopen function with passing ccs= parameter so that the input will considered in that form.

There is no such "ccs=" parm on HP-UX.

>Please let me know how to set locale for UTF-16.

Why do you care about utf16? Just use utf8 and convert to wchar_t with iconv(3).

[Dennis Handly]

>I am trying to read UTF16 encoded data from stdin (as of now only thru file re-direction).

You have a data file with UTF16 chars?
Then you'll need to fread(3) a large buffer then call iconv(3) to convert it.
Either to wchar_t or multibyte strings.

You may have to convert from your ucs2 codeset to some utf8 codeset, then from that to ucs4 for wchar_t.

[Srikrishna Erra]

Hi,

I care about only UTF16 input (either from redirection or interactive)because our customer is trying to read UTF16 input from both file re-drection and interactively.

fread() will not work if the input is interactively. With the prompt still wait for more input (count passed to fread) even though the Enter key is pressed. So fread will not work for STDIN input handling.

If there is a way to selocale for UTF-16 then my problem will resolve very easliy.

[Dennis Handly]

>fread will not work if the input is interactively. With the prompt still wait for more input (count passed to fread) even though the Enter key is pressed. So fread will not work for STDIN input handling.

Why do you say that? You should be able to terminate fread from a terminal by enter.
How are you reading UTF16 data from what type of terminal? Is this some Asian language?

>If there is a way to setlocale for UTF-16 then my problem will resolve very easily.

I don't think there are any locales with UTF16. Why aren't you using a UTF8 locale and data?

[Srikrishna Erra]

>>Why do you say that? You should be able to terminate fread from a terminal by enter.
How are you reading UTF16 data from what type of terminal? Is this some Asian language?

Input Data is in UTF16 form (saved using Notepad in windows and ftped to HPUX) contains all Unicode characters in UTF16 range.(2^16)

Here is the code used for reading the data.

int COUNT=512;
int bcnt=0;
bcnt=(int)read((char *)dst,
sizeof(char),
COUNT,
stdin);

From man pages of fread:

size_t fread(void *ptr, size_t size, size_t nitems, FILE *stream);

The fread() function reads into the array pointed to by ptr up to nitems elements whose size is specified by size in
bytes, from the stream pointed to by stream.

How can we terminate by pressing Enter Key?

[Srikrishna Erra]

Hi
please note that for now i am trying to test by re-direction. I able to handle Re-direction from file i.e. giving the string only till \n (0x000a) even though the strings returned by fread in one iteration is still more.

For interactive, it is not like that. I have terminate reading when \n(0x000a) i.e Enter key is pressed.

[Dennis Handly]

>How can we terminate by pressing Enter Key?

Only by having a 8 bit newline at the end as you mentioned below.

>I able to handle Re-direction from file i.e. giving the string only till \n (0x000a)

How does Notepad terminate your lines?

>For interactive, it is not like that. I have terminate reading when \n(0x000a) i.e Enter key is pressed.

I assume this should work too.