topic REMOVING UNICODE CHARACTERS from file in Operating System - HP-UX

REMOVING UNICODE CHARACTERS from file

MBacc — Tue, 08 Sep 2009 18:46:12 GMT

Does anyone know how to remove unicode characters from a file in unix?

Re: REMOVING UNICODE CHARACTERS from file

Steven E. Protter — Tue, 08 Sep 2009 19:30:40 GMT

Shalom,

It would help to know what characters specifically and how they got there. Samba? FTP transfer. Email as an attachment? If so how was the file transmitted.

dos2unix

See the man page, it might help.

SEP

Re: REMOVING UNICODE CHARACTERS from file

Matti_Kurkela — Tue, 08 Sep 2009 19:45:15 GMT

Your problem can be re-phrased as "remove everything that is not an ASCII control character or an ASCII printable character".

When a problem is presented in this way, it's easy to find a solution using the standard "tr" command.

Example: file.utf8 contains Unicode UTF8 characters, and file.txt will be the stripped version.

export LC_ALL=C
tr -dc '[:cntrl:][:print:]' < file.utf8 > file.txt
unset LC_ALL

Setting the environment variable LC_ALL to C for the duration of this command is important: it explicitly switches off the Unicode support and tells tr that only ASCII characters are considered to be "printable".

This command can be run as an one-liner too:

LC_ALL=C tr -dc '[:cntrl:][:print:]' < file.utf8 > file.txt

MK