Operating System - Linux
1745924 Members
3984 Online
108723 Solutions
New Discussion юеВ

Re: simplest way to strip a character from a filename

 
SOLVED
Go to solution
Daavid Turnbull
Frequent Advisor

Re: simplest way to strip a character from a filename

Dear Arun - If the whole script was Perl I know you would be right but in this instance the majority of the file handling is done by the shell which invokes perl for every file name. Hence my question about the overhead of "invoking" perl over tr. I had assumed that tr would have less overhead because it is smaller but this is not the only factor.

In the script that was processing these files I have changed the line that moves the file to read:

mv $file $(echo "$dest/$file.$dayTimeStr" | tr ":" "_")

(Looking at the script as a whole these days I would have written the whole script in perl which would no doubt have solved a lot of problems and made it a lot easier to maintain.)
Behold the turtle for he makes not progress unless he pokes his head out.
Muthukumar_5
Honored Contributor

Re: simplest way to strip a character from a filename

Perl is utilizing more cpu resource. It is good to go with default utilities like sed or tr or awk also.

mv $file $(echo "$dest/$file.$dayTimeStr" | tr ":" "_")


It is not good always.

Use as,

mv ${file} $(echo "${dest}/${file}.${dayTimeStr}" | tr ":" "_")

It is good.

PS: Do you want to remove : or change to _ ?

--
Muthu
Easy to suggest when don't know about the problem!
Muthukumar_5
Honored Contributor

Re: simplest way to strip a character from a filename

If you want to maintain load then use tr method or sed instead of perl.

mv ${file} $(echo ${file}| sed -e 's/://g'

If you want to have SPEED use perl always.

mv ${file} $(echo ${file}| perl -pe 's/://g'

--
Muthu
Easy to suggest when don't know about the problem!
Arunvijai_4
Honored Contributor

Re: simplest way to strip a character from a filename

Hi David,

You said, If the whole script was Perl I know you would be right but in this instance the majority of the file handling is done by the shell which invokes perl for every file name. Hence my question about the overhead of "invoking" perl over tr. I had assumed that tr would have less overhead because it is smaller but this is not the only factor.

In the script that was processing these files I have changed the line that moves the file to read:

mv $file $(echo "$dest/$file.$dayTimeStr" | tr ":" "_")

When it comes to handle lot of files, i dont think unix default utilities will play a big part. I am not sure "tr" and "mv" are multithreaded as well. Perl with combination Unix shell utilities is a good way to strike.

-Arun
"A ship in the harbor is safe, but that is not what ships are built for"
Sandman!
Honored Contributor

Re: simplest way to strip a character from a filename

Hi Daavid,

After careful deliberation, it looks like you need to replace the colon characters and rename the files.

Here's an awk construct that would help. It assumes that your curent working dir is the one that has a guzzillion of those tiny files.

# ls -1 | awk '{x=$0;gsub(":","");system("mv "x" "$0)}'

cheers!
A. Clay Stephenson
Acclaimed Contributor

Re: simplest way to strip a character from a filename

You need to recognize that your real bottleneck in this case is not Perl/Shell/external commands but rather the large (I assume a gazzilion is a large number) number of entries in a directory. The overhead of safely rewriting each directory entry is quite high. UNIX has to make certain that no other processes are altering the directory at the same time. Even if written in very tight C this would still be a time-consuming operation. On the other hand, this sounds like a one-time deal so let it run all night or over a weekend and declare victory.
If it ain't broke, I can fix that.
Daavid Turnbull
Frequent Advisor

Re: simplest way to strip a character from a filename

Dear All,

Thanks for the responses.

I managed to replace the colons in about 90000 file names and fix the script that was responsible for creating them.

Clay, your last post improved my understanding of why it was taking so long a lot.

It did not help that the machine was out of disk space (hence my renaming requirement - I needed to archive the offending files to a Windows environment where colons in file names are verboten.) At one point I was getting errors doing the actual rename because of the lack of disk space.

I would probably still be trying to pull my hair out if if it were not for this forum.
Behold the turtle for he makes not progress unless he pokes his head out.