topic Re: need to remove carriage return from end of last record in a file in Operating System - HP-UX

need to remove carriage return from end of last record in a file

Kevin Medlin — Thu, 21 Feb 2002 21:16:32 GMT

Howdy,

I get files with "non-printable carriage return characters" (\n in Textpad) at the end of each row. On the last row this causes an "extra row" when looked at by non-UNIX text editors. This last carriage return needs to be stripped somehow. Any ideas?

In vi I don't see anything except my normal rows. I saw that some folks see ^M, but I don't. How do I see the non printable characters in vi?

Re: need to remove carriage return from end of last record in a file

S.K. Chan — Thu, 21 Feb 2002 21:23:13 GMT

In vi, type the following:

:1,$s/^V^M//

The key sequence is CTRL+V and CTRL+M, only the "^M" will appear on screen.
This will remove th ^M character.

Re: need to remove carriage return from end of last record in a file

SHABU KHAN — Thu, 21 Feb 2002 21:23:47 GMT

Hi Kevin,

There are many ways to get rid of ^M from a file here a few:

in VI
:%s/^V^M//g

^V is control V and ^M is control M or Enter

USING SED:
sed 's/^V^M//g' foo > foo.new

Also:
strings foo > foo.new
should work

-Shabu

Re: need to remove carriage return from end of last record in a file

A. Clay Stephenson — Thu, 21 Feb 2002 21:27:59 GMT

I suspect what is happening is that you are sharing text files between UNIX and Windows. The Windows guys expect each line to end with a CR/LF pair while the UNIX guys expect only a LF. There are a couple of utilities that will do the conversion for you. dos2ux and ux2dos. Man ux2dos and dos2ux for details.

A good way to see all the characters in the file is via the od command. od -c myfile will display the stuff in a readable form.

Re: need to remove carriage return from end of last record in a file

Alan Riggs — Thu, 21 Feb 2002 22:16:10 GMT

:set list
can be used to display non-printing characters in vi.

Re: need to remove carriage return from end of last record in a file

Deepak Extross — Fri, 22 Feb 2002 05:51:58 GMT

Hi,
Have these files been ftp'd to a Unix filesystem fro a DOS/Windows system? If so, it's likely that the ftp was done in binary mode.
While you can remove those pesky ^M characters using sed or dos2ux, I find that these CRs do not appear at all if the file is ftp'd in ascii mode.

Re: need to remove carriage return from end of last record in a file

Kevin Medlin — Fri, 22 Feb 2002 14:46:04 GMT

Thanks for the great responses.

I am creating the file on UNIX and then ftp'ing to a Windows server. The problem is when they try to load the file in SQL server it blows there script on that last non-printable character. It tries to load an empty row.

I did ':set list' command and I can see a $ (dollar sign) at the end of each row. There are no ^Ms or ^Vs in the file.

I did 'ux2dos myfile > newfile'. This put ^Ms in newfile. It also added a new row to the end of newfile that has '^Z$' only.

I don't want to remove the non-printable character from every row. I just want it removed from the last row. Also, I would like to be able to add the command string to my ftp script so this would not be a manual process.

Re: need to remove carriage return from end of last record in a file

Darrell Allen — Fri, 22 Feb 2002 15:33:51 GMT

Hi Kevin,

When you ftp the file from UNIX to Windows (or vice versa), you need to use ASCII mode. This will take care of the "line delimiting" difference between UNIX and Windows.

As info, UNIX uses a newline (actually the same ascii code as a linefeed) character to delimit lines where Windows uses a linefeed and a carriage return.

If I read your post correctly, you have a blank line at the end of the file when you create it in UNIX. You need to delete that blank line before loading the file into SQL Server. Maybe you can modify how the file is created to not append the last blank line.

To better see what the last line is, use od. If the last line is blank, you would see:
tail -1 file | od -cb
0000000 \n
012
0000001

man ascii for more info on the ascii character set.

Darrell

Re: need to remove carriage return from end of last record in a file

Kirk Gardner — Fri, 22 Feb 2002 15:42:50 GMT

And just to add to the mix:

tr -d \\015 < filename > newfilename

or

dos2ux filename > newfilename

Re: need to remove carriage return from end of last record in a file

Darrell Allen — Fri, 22 Feb 2002 21:59:59 GMT

Hi again Kevin,

This is an ugly little script but it should work for you. At least it did on my test file of ~500 lines. The result is all lines print as is except the last which has the newline striped off.

while read line
do
echo "$line\c"
while read line
do
echo "\n$line\c"
done
done outfile

Darrell