How to use a hex-00 as a seperator in a ksh

[Dirk Rosenblatt]

I have some database export files. As field seperator a "\0" (hex-00) was used. Now we have some problems with this files and I need to do a few checks like
cut -d"\0" -f1-7 file
grep "^500\0" file
but nothing works...

[Dennis Handly]

You can use tr(1) to change that NUL to another char that is shell friendly.

[Jose Mosquera]

Hi,

At export process you can force to use a specific field separator (eg: pipe symbol).

With your current export file please:
#cat -t your_export_file
What is the translation of characters that you visualize as a field separator?

About ASCII map on your system please:
#man ascii

Rgds.

[Dirk Rosenblatt]

@Dennis Handly: the same problem. How to deal with hex-00 in tr?

@Jose Mosquera: it is a big table and it is locked during export so I can only use sundays for it if ever. So I have to deal with the silly character.

-t for cat was new for me. thx.
The output looks like:

500^@1^@2^@10151998^@58100^@

Did not find the ^@ in "man ascii" but google helped me out. It is a hex-00. Now ctrl-v, ctrl-a is a hex-01 but how to get hex-00? ctrl-v, ctrl-shift-@ does not work.

[James R. Ferguson]

Hi Dirk:

Consider:

# echo "hello\000there\000dirk"|cat -etv
hello^@there^@dirk$

Now:

# echo "hello\000there\000dirk"|tr -s "\000" ":"
hello:there:dirk

You must quote the \000 (null).

Regards!

...JRF...

[Matti_Kurkela · Accepted solution]

"man tr" says:
-----
The escape character \ can be used as in the shell to remove special meaning from any character in a string. In addition, \ followed by 1, 2, or 3 octal digits represents the character whose ASCII code is given by those digits.

An ASCII NUL character in string1 or string2 can be represented only as an escaped character; i.e. as \000, but is treated like other characters and translated correctly if so specified. NUL characters in the input are not stripped out unless the option -d "\000" is given.
-----
NUL is the ASCII standard name for the hex-00 character.

In other words: the tr command understands "\000" means the hex-00 character. The man page also explicitly says tr will manipulate hex-00 characters correctly if commanded to do so.

Choose a separator character that is suitable to you (i.e. one that you know is not used in the exported data).

For example, if a colon ":" is a suitable separator for your data, then do this:

tr "\000" ":" < export.hex00 > export.colons

MK

[Jose Mosquera]

Hi,

Try this one:
#cat -t current_export_file|sed 's/\^@/|/g' > new_formated_file
#cat new_formated_file

Rgds.

[Dennis Handly]

>the same problem. How to deal with hex-00 in tr?

Read the man page? ;-)
tr "\00" ":" < file | grep "^500:"

If you actually want to echo a NUL you could try to do:
grep "^500$(echo '\00\c')" file | xd -tc

Unfortunately this won't work because argv[x] is null terminated, so there is no way for the process to see that NUL.

[Dirk Rosenblatt]

mea culpa, mea culpa, mea maxima culpa

sorry, read the only head and the tail of the man page.
But I had expected that the shell does the work. And so I learned, that it only works with the tr command.

Thanks...

[Dirk Rosenblatt]

.

[Matti_Kurkela]

Most Unix shells and command line utilities are written in "C" programming language.

When character strings are processed in C language, the hex-00 character is typically used to indicate end-of-string.

http://en.wikipedia.org/wiki/C_string

If a C program must process strings that contain hex-00 characters, the standard C library functions for string manipulation cannot be used. Instead, the programmer must use enhanced string manipulation routines (either written from scratch, or using an enhanced string manipulation library).

This means the programmer must do more work and the resulting program will be more complex, so the ability to process hex-00 characters is usually not implemented, unless there is a clear requirement for it.

MK

[James R. Ferguson]

HI:

...and examples didn't help you either (?).

...JRF...