topic Re: simplest way to strip a character from a filename in Operating System - Linux

simplest way to strip a character from a filename

Daavid Turnbull — Tue, 21 Feb 2006 19:52:36 GMT

I have a gazzilion tiny files each with multiple ':' in the file name. I need to strip these out.

A sample file name is: x_0041257026_102110535974_2006-01-26T20:33:31.272Z.xml.28_00_05.Z

I wrote a clumbsy perl script to do it which is working but it soooo slooooow.

Is there a mean, easy and efficient way to do this?

Re: simplest way to strip a character from a filename

James R. Ferguson — Tue, 21 Feb 2006 20:18:00 GMT

Hi David:

# perl -ple 's/\.//g;s/Z$/\.Z/'

As for example:

# echo "12.3\nabc_def.xyz\ndavid.Z"|perl -ple 's/\.//g;s/Z$/\.Z/'

(or):

# perl -ple 's/\.//g;s/Z$/\.Z/' fileofnames

This (crudely) preserves the ".Z" extension if present.

Regards!

...JRF...

Re: simplest way to strip a character from a filename

Geoff Wild — Tue, 21 Feb 2006 20:19:12 GMT

Don't have a solution for the renaming, but as far as generation of the files - did you fix that so that it doesn't do the :

Looks like a time stamp...

Could do something like:

date '+%FT%H%M%S'

Rgds...Geoff

Re: simplest way to strip a character from a filename

Sandman! — Tue, 21 Feb 2006 20:23:26 GMT

Hi,

Try this awk construct...

# ls -1 | awk '{gsub(":",""); print $0}'

cheers!

Re: simplest way to strip a character from a filename

Daavid Turnbull — Tue, 21 Feb 2006 21:36:07 GMT

I am not sure that this is particularly efficient because it invokes perl for each mv but this is currently doing what I want:

for file in *
> do
> mv $file `echo $file | perl -ple 's/\://g'`
> done

Note that the char I wished to replace was a : and not a .

Re: simplest way to strip a character from a filename

A. Clay Stephenson — Tue, 21 Feb 2006 22:02:11 GMT

Something like this should work leveraging the tr command:

ls | while read FNAME
do
FNAME2=$(echo "${FNAME}" | tr -d ":")
if [[ "${FNAME}" != "${FNAME2}" && -n "${FNAME2}" ]]
then
if [[ -r "${FNAME2}" ]]
then
echo "${FNAME2} exists; can't mv" >&2
else
mv "${FNAME}" "${FNAME2}"
fi
fi
done

The idea is that we use tr -d to strip the ':'s from the filename. Next we check to see if the filenames are then diffirent and also that the filename still has a non-zero length. Next we make sure that the new filename does not already exist; if so, that file is skipped. Finally, after all the tests have passed we mv the old name to the new.

Note the "'s around each filename. Whitespace is perfectly legal (if dumb) in UNIX pathnames.

This should be a robust solution.

Re: simplest way to strip a character from a filename

Muthukumar_5 — Tue, 21 Feb 2006 22:50:38 GMT

for file in *
> do
> mv $file `echo $file | perl -ple 's/\://g'`
> done
==

Simply as,

for file in `ls`
do

mv ${file} $(echo ${file} | perl -pe 's/://g')

done

--
Muthu

Re: simplest way to strip a character from a filename

Muthukumar_5 — Tue, 21 Feb 2006 22:53:42 GMT

There is no need to negate with \ for : character. IT is needed for * ? \ / characters. Simply use s/://g is enough for pattern change.

--
Muthu

Re: simplest way to strip a character from a filename

Daavid Turnbull — Tue, 21 Feb 2006 23:00:23 GMT

Because it needs to be done a gazzillion times (well somewhere between 100000 and 1000000) am I correct in assuming that the over head of starting perl as opposed to tr would be a factor in the load it places on the machine?

Re: simplest way to strip a character from a filename

Arunvijai_4 — Tue, 21 Feb 2006 23:11:59 GMT

Hi David,

For processing large number of files, perl is the best way to go. It has proven ability when it comes to large number.

-Arun

Re: simplest way to strip a character from a filename

Daavid Turnbull — Tue, 21 Feb 2006 23:25:54 GMT

Dear Arun - If the whole script was Perl I know you would be right but in this instance the majority of the file handling is done by the shell which invokes perl for every file name. Hence my question about the overhead of "invoking" perl over tr. I had assumed that tr would have less overhead because it is smaller but this is not the only factor.

In the script that was processing these files I have changed the line that moves the file to read:

mv $file $(echo "$dest/$file.$dayTimeStr" | tr ":" "_")

(Looking at the script as a whole these days I would have written the whole script in perl which would no doubt have solved a lot of problems and made it a lot easier to maintain.)

Re: simplest way to strip a character from a filename

Muthukumar_5 — Tue, 21 Feb 2006 23:34:07 GMT

Perl is utilizing more cpu resource. It is good to go with default utilities like sed or tr or awk also.

mv $file $(echo "$dest/$file.$dayTimeStr" | tr ":" "_")

It is not good always.

Use as,

mv ${file} $(echo "${dest}/${file}.${dayTimeStr}" | tr ":" "_")

It is good.

PS: Do you want to remove : or change to _ ?

--
Muthu

Re: simplest way to strip a character from a filename

Muthukumar_5 — Tue, 21 Feb 2006 23:36:53 GMT

If you want to maintain load then use tr method or sed instead of perl.

mv ${file} $(echo ${file}| sed -e 's/://g'

If you want to have SPEED use perl always.

mv ${file} $(echo ${file}| perl -pe 's/://g'

--
Muthu

Re: simplest way to strip a character from a filename

Arunvijai_4 — Tue, 21 Feb 2006 23:39:15 GMT

Hi David,

You said, If the whole script was Perl I know you would be right but in this instance the majority of the file handling is done by the shell which invokes perl for every file name. Hence my question about the overhead of "invoking" perl over tr. I had assumed that tr would have less overhead because it is smaller but this is not the only factor.

In the script that was processing these files I have changed the line that moves the file to read:

mv $file $(echo "$dest/$file.$dayTimeStr" | tr ":" "_")

When it comes to handle lot of files, i dont think unix default utilities will play a big part. I am not sure "tr" and "mv" are multithreaded as well. Perl with combination Unix shell utilities is a good way to strike.

-Arun

Re: simplest way to strip a character from a filename

Sandman! — Wed, 22 Feb 2006 11:38:11 GMT

Hi Daavid,

After careful deliberation, it looks like you need to replace the colon characters and rename the files.

Here's an awk construct that would help. It assumes that your curent working dir is the one that has a guzzillion of those tiny files.

# ls -1 | awk '{x=$0;gsub(":","");system("mv "x" "$0)}'

cheers!

Re: simplest way to strip a character from a filename

A. Clay Stephenson — Wed, 22 Feb 2006 12:47:45 GMT

You need to recognize that your real bottleneck in this case is not Perl/Shell/external commands but rather the large (I assume a gazzilion is a large number) number of entries in a directory. The overhead of safely rewriting each directory entry is quite high. UNIX has to make certain that no other processes are altering the directory at the same time. Even if written in very tight C this would still be a time-consuming operation. On the other hand, this sounds like a one-time deal so let it run all night or over a weekend and declare victory.

Re: simplest way to strip a character from a filename

Daavid Turnbull — Wed, 22 Feb 2006 19:21:51 GMT

Dear All,

Thanks for the responses.

I managed to replace the colons in about 90000 file names and fix the script that was responsible for creating them.

Clay, your last post improved my understanding of why it was taking so long a lot.

It did not help that the machine was out of disk space (hence my renaming requirement - I needed to archive the offending files to a Windows environment where colons in file names are verboten.) At one point I was getting errors doing the actual rename because of the lack of disk space.

I would probably still be trying to pull my hair out if if it were not for this forum.