topic Re: sed help in Operating System - Linux

sed help

slydmin — Mon, 04 Feb 2008 16:30:53 GMT

I can use some help using sed. I have a file that has a lot of trailing spaces.
I use vi to edit the file, and search and replace all the trailing space like so
:%s/[ ]*$//

Although, when I try to use sed, I am not able to edit the file at all. Here is how I am trying to do this

cat filename | sed -e 's/[ ]*$//'

This does not edit the file at all. What could I be doing wrong? Any help is greatly appreciated.
-S

Re: sed help

slydmin — Mon, 04 Feb 2008 16:48:06 GMT

I think I may have a solution using tr

cat fielname | tr -d '\032* $'

Although, I need to confirm it with the QA ppl to make sure the file works for them.

Re: sed help

Alexander Chuzhoy — Mon, 04 Feb 2008 18:24:34 GMT

You need to append g (to allow sed to fix all the matched cases and not just the first one). So it's:
sed -e 's/[ ]*$//g' filename

Re: sed help

Eric SAUBIGNAC — Mon, 04 Feb 2008 18:46:25 GMT

Bonsoir -S,

It should work asis, that is without 'g' flag. Under HP-UX it works well.

As far as I know, regular expressions are satisfied with the largest number of characters that can match the expression. If it was not the case, and IMHO, '*' in a regular expression would have no sens. And '[ ]*$' clearly means : 0 or more spaces before end of line ! So it shoul work.

Don't know Linux as well as HP-UX but maybe there is a limitation on implementation of sed. Maybe there is a special option to tell 'sed' to use extended regular expressions ? I have this beahavior under HP-UX with 'grep' : if you don't use -E option, 'grep' doesn't treat correctly all regular expressions.

An other idea : '[]' is used to match a list of characters. As you have only one character, space, may be '[ ]' is not correctly interpreted ? So try this :

cat filename | sed -e 's/ *$//'

Hope this will help

Regards

Eric

Re: sed help

Stuart Browne — Mon, 04 Feb 2008 20:16:32 GMT

As far as I'm aware, none of these commands 'edit' the file in place, but alter an output stream in the pipe.

If you want to edit a file in place, you can use:

perl -pi -e 's/\s+$//' filename

Re: sed help

Alexander Chuzhoy — Tue, 05 Feb 2008 06:43:29 GMT

Good point Stuart.

slydmin, in order to actually update the content of the file you can redirect the output:
sed -e 's/[ ]*$//g' > newfile; mv newfile origfile;

make sure you have a backup of the original file.

Re: sed help

slydmin — Tue, 05 Feb 2008 16:31:01 GMT

Alexander,
I was actually doing that, but did not mention in my original email.

sed -e 's/[ ]*$//'outputfile

I tried everyone's suggestion but none gave desired output.

I have a csv file, with say about 14 comma separated fields. The last field has a lot of spaces in it.
vim works like a charm (:%s![ ]*$!), although we are going to have a large number of files to process and manually doing it does not make sense.

So this attempt to use same regex with sed does not work. tr -d \032 works, although it removes white space from everywhere in the file.
I will copy 3 lines from the file and attache it herewith, although the data is changed.

I am stumped at this point.

Re: sed help

Eric SAUBIGNAC — Tue, 05 Feb 2008 16:59:38 GMT

Bonsoir,

Just found this at http://pwet.fr/man/linux/commandes/sed :

-r, --regexp-extended
use extended regular expressions in the script

It sounds like like -E option for grep with HP-UX, no ?

Could you try to add -r option and tell us ?

Regards

Eric

Re: sed help

slydmin — Tue, 05 Feb 2008 17:16:51 GMT

Eric,
I thought I had tried that (sed -r). But I gave it a try and it did not work.

What is interesting though, the file I uploaded , if that were to be manipulated with sed -r 's/[ ]*$//g' , I get the desired output but not on the original file.

The original file has a line count of about 25000. I tried splitting it up in to 5000 line files, but still am not having success.

I am going to try and use perl to see if I can use the expression posted to my OP (it removed all spaces, just like tr -d\032).

Thanks,
-S

Re: sed help

slydmin — Tue, 05 Feb 2008 17:37:32 GMT

Actually I was wrong about the perl expression, it does not remove all white spaces like tr does.

It removes the white space(s) at the end of the line, but not all new lines are lost.

2nd line is not a continuation of the 1st.

Will try a few more things, but I stand corrected on the tr /perl comparison.

Re: sed help

slydmin — Tue, 05 Feb 2008 17:55:30 GMT

I may have the solution after all.

Stuart's command line perl expression works very well with one minor change

perl -pi -e 's/\s+$/\n/g' filename

The addition of \n -newline ensures that lines are kept separate.

Thanks all for the help

Re: sed help

Eric SAUBIGNAC — Fri, 08 Feb 2008 19:04:43 GMT

Bonsoir,

It must work with sed ... grrrr ! And I the solution ... IMHO ;-)

You said "What is interesting though, the file I uploaded , if that were to be manipulated with sed -r 's/[ ]*$//g' , I get the desired output but not on the original file"

Here is the difference : NOT ON THE ORIGINAL FILE.

Why does it work with perl and not with sed ?You use [ ] with sed wich means space. But you use \s with perl wich means blankspace : blankspace is characters producing white-space in displayed text : spaces and tabulations.

Here again is the difference : THE ORIGINAL FILE INCLUDES SPACES AND TABULATIONS.

So the answer is very simple : all lines are not necessary spaces terminated in your original file. Make a ontrol with 'od -c' ...

For detailed examples see the attached file.

Here is the definitive answer to work with sed. 2 solutions :

sed -r 's/[[:space:]]*$//' filename

or

SPACE=`echo " \t\c"`
sed -r 's/['"$SPACE"']*$//' filename

What about it ? :-)

Have a nice WE

Regards

Eric