grep for asian characters in UTF8 file

[Mark H Smith]

I have a UTF-8 text file containing four languages: English, Chinese, Japanese, and Korean.

Each line has 2 columns: an english sentence in the first column and its C,J,or K translation in the second column. (tab separated)

However some lines contain only English and are of no use to me. So I want to locate and discard the english-only lines containing no valid asian data.

cat -A allows me to see certain escape codes in this file, such as tab (^I), and carriage returns (^M$) and also the -A flag shows me that each asian sentence begins with an uppercase M. I'm assuming that represents some code signalling a switch to asian text.

I guess what I am really looking for is a way to grep for asian characters..

[D Block 2]

Mark- how about sending us an example output.. Also, have your tried using the:

cat -v

or have your also tried using:

strings

Also, you might want to verify your Shell Environment variables:

$ set

Just to verify your settings for UTF.

[Mark H Smith]

I have attached a snippet of my file.

cat -v gives the same result. (just not sure how to interpret escape codes, if any)

running strings (debian binutils) on it only gives me the english-- no asian characters, just empty space..

Output of locale:

LANG=en_US.utf8
LC_CTYPE="en_US.utf8"
LC_NUMERIC="en_US.utf8"
LC_TIME="en_US.utf8"
LC_COLLATE="en_US.utf8"
LC_MONETARY="en_US.utf8"
LC_MESSAGES="en_US.utf8"
LC_PAPER="en_US.utf8"
LC_NAME="en_US.utf8"
LC_ADDRESS="en_US.utf8"
LC_TELEPHONE="en_US.utf8"
LC_MEASUREMENT="en_US.utf8"
LC_IDENTIFICATION="en_US.utf8"
LC_ALL=en_US.utf8


snippet of my file: (see attachment if this appears garbled)
--------------------------
LS-3108 LS-3108

LS-3114 LS-3114
LS-3108 LS-3108

LS-3114 LS-3114
LS-3108 LS-3108

LS-3114 LS-3114
It must be Easter!
It must be Easter!

It's a 60's shindig...
It's a 60's shindig...

It's a baby bash!

It's a baby bash!
It's a birthday fete! èª ç ä¼ ã ®ã æ¡ å
It's a birthday fete! ì ì ¼ í í °
It's a party! å­ ã ©ã ä¼ ã ®ã ã ¼ã 㠣㠼
It's a party! í í °!
It's the 4th of July.

It's the 4th of July.
It's the beginning of post-adolescent neuroses!

It's the beginning of post-adolescent neuroses!
It's time for a shower. çµ å© å¼ äº æ¬¡ä¼ ã ®ã æ¡ å

It's time for a shower. ì²­ì²©ì ¥ 3
It's a picnic... ã ã ¯ã ã ã ¯ã «çµ¶å¥½ã ®å­£ç¯ ã §ã ...
It's a picnic...
I love my phone! æ ç ±æ ç ç µè¯ !
I love my phone! í ´ë í °ì ´ ë 무 ì¢ ì ì !
Running æ­£å ¨è¿ è¡
And I've been head over heels ever since. And I've been head over heels ever since.

And I've gone gaga over you.

file and linker output. í ì ¼ì ë§ ë ¤ê³  ì ì µë ë ¤.

And may you hop with happiness.

And Mom's never forgiven you for it!
Recipient Postal Code æ ¶ä¿¡äººé ®æ ¿ç¼ ç 

Recipient qualifier æ ¶ä»¶äººé å® ç¬¦

Recipient queue name length æ ¶ä»¶äººé å å ç§°é ¿åº¦

Recipient Queue æ ¶ä»¶äººé å

Recipient routing address æ ¶ä»¶äººè·¯ç ±å °å

{Recipient's Home Telephone #} {æ ¶ä»¶äººå®¶åº­ç µè¯ å ·ç  }

{Recipient's Name} {æ ¶ä»¶äººå§ å }

{Recipient's Office Location} {æ ¶ä»¶äººå å ¬å °ç ¹}

{Recipient's Office Telephone #} {æ ¶ä»¶äººå å ¬å®¤ç µè¯ å ·ç  }
--------------------------

[Vibhor Kumar Agarwal]

Will this work:

grep -v "^[a-z,A-Z,tabs,spaces]*$

[Mark H Smith]

afraid not.. I still get all those unwanted ASCII-only lines in the output..

[Vibhor Kumar Agarwal]

I missed a word "something like this"

Don't know the exact syntax but i think you got the meaning.

Just grep -v all lines which contain only a-z,A-Z,spaces,tabs,nos.

[Mark H Smith]

I want to filter out all ASCII-only lines. In order to accomplish this, don't I want to grep for the asian characters, and not those between A-Z?

[Vibhor Kumar Agarwal]

Have a look into this:

I think Gnu grep has a option to look for ascii values of characters.

[Karthick K S]

Hi Mark,

This is not related to ur question but i need some information regarding korean language setup.

1.If i using ICONV cmd it converting english to Korean but if i am trying thro'Keyboard(I changed Korean fonts in windows),not getting i/p from kbd(only english taking)so pls help me for this

[john korterman]

Hi Mark,


perhaps you could instead look for a certain line structure, e.g. lines ending with a certain code not followed by any text.
For instance, based on your attachment, start by discarding lines ending in either "" or "" followed by only a single space
- and afterwards throw away the rest, among other lines, those ending in "LS31" followed by two digits.
If that is an idea, try the below single-line example as a starting point:

# grep -vE ".*[|] $" ./infile| grep -v "LS-31[0-9][0-9]"


regards,
John K.