topic Re: sorting files in Operating System - Linux

sorting files

Gyankr — Wed, 09 Apr 2008 10:07:26 GMT

Hello,

I want to sort below files in a particular order

currently it is like below:
>ls
fancy_LANG_STD_AU_2008-03-05.dat
fancy_LANG_STD_HK_2008-03-06.dat
fancy_LANG_STD_NZ_2008-03-05.dat
fancy_STD_AU_2008-03-05.dat
fancy_STD_HK_2008-03-06.dat
fancy_STD_NZ_2008-03-05.dat

i want to sort it like this

fancy_STD_AU_2008-03-05.dat
fancy_LANG_STD_AU_2008-03-05.dat
fancy_STD_HK_2008-03-06.dat
fancy_LANG_STD_HK_2008-03-06.dat
fancy_STD_NZ_2008-03-05.dat
fancy_LANG_STD_HK_2008-03-06.dat

i.e i want the standard files for a country first ( fancy_STD_HK...) and then its corresponding language files(fancy_STD_LANG_HK_...).

Please let me know how it can be done

Regards,
Gyan

Re: sorting files

Dennis Handly — Wed, 09 Apr 2008 10:29:42 GMT

You'll need to break up the filenames using awk then use sort:
ls | awk -F_ '
{
if ($2 == "LANG") {
# combine fields
country = $4
lang = "STDLANG"
} else {
country = $3
lang = $2
}
print country, lang, $1, $0
}' | sort | awk '{print $4}'

Did you want to also sort by the first "fancy" or the date?

Re: sorting files

Aashique — Wed, 09 Apr 2008 10:34:15 GMT

Hi,
first create a file with all your country code
lets say test1

vi test1
-----------
AU
HK
NZ

then write another sctipt name "test2"

vi test2
--------
set `cat test1`
for i in $*
do
grep $i test|sort -rn>>test3

$ more test3
fancy_STD_AU_2008-03-05.dat
fancy_LANG_STD_AU_2008-03-05.dat
fancy_STD_HK_2008-03-06.dat
fancy_LANG_STD_HK_2008-03-06.dat
fancy_STD_NZ_2008-03-05.dat
fancy_LANG_STD_NZ_2008-03-05.dat

Thanks & Regards

Aashique

Re: sorting files

Aashique — Wed, 09 Apr 2008 10:36:43 GMT

Hi,
i missed to write something.in the "test" file

vi test
----------
fancy_LANG_STD_AU_2008-03-05.dat
fancy_LANG_STD_HK_2008-03-06.dat
fancy_LANG_STD_NZ_2008-03-05.dat
fancy_STD_AU_2008-03-05.dat
fancy_STD_HK_2008-03-06.dat
fancy_STD_NZ_2008-03-05.dat

Re: sorting files

Hein van den Heuvel — Wed, 09 Apr 2008 11:16:34 GMT

At first is seems you can just sort specifying -t_ to make underscore be a field terminator. But the field number also changes.

So you need a sort with a twist.
Here is a perl solution:

$ cat x
fancy_LANG_STD_AU_2008-03-05.dat
fancy_LANG_STD_HK_2008-03-06.dat
fancy_LANG_STD_NZ_2008-03-05.dat
fancy_STD_AU_2008-03-05.dat
fancy_STD_HK_2008-03-06.dat
fancy_STD_NZ_2008-03-05.dat

$ perl -e 'sub twist {$x=@_[0]; $x=~s/_LANG(_STD_..)_/${1}x/; return $x}; print sort {twist($a) cmp twist($b)} <>' x
fancy_STD_AU_2008-03-05.dat
fancy_LANG_STD_AU_2008-03-05.dat
fancy_STD_HK_2008-03-06.dat
fancy_LANG_STD_HK_2008-03-06.dat
fancy_STD_NZ_2008-03-05.dat
fancy_LANG_STD_NZ_2008-03-05.dat

Just to clarify, here is that twist function alone at work:

$ perl -e 'sub twist {$x=@_[0]; $x=~s/_LANG(_STD_..)_/${1}x/; return $x}; foreach (<>) {print twist($_)}' x
fancy_STD_AUx2008-03-05.dat
fancy_STD_HKx2008-03-06.dat
fancy_STD_NZx2008-03-05.dat
fancy_STD_AU_2008-03-05.dat
fancy_STD_HK_2008-03-06.dat
fancy_STD_NZ_2008-03-05.dat

fwiw,
Hein.

Re: sorting files

Gyankr — Wed, 09 Apr 2008 13:09:10 GMT

Hi again,
I am not so familiar with perl.
I am trying something like this
local filelist="$@"
where filelist contains all the list of files passed from someother file.

Now how do i sort it,i tried Dennis' solution
local filelist="$@"
local filesortedlist=`echo $filelist | awk -F_ '
{
if ($2 == "LANG") {
# combine fields
country = $4
lang = "STDLANG"
} else {
country = $3
lang = $2
}
print country, lang, $1, $0
}' | sort | awk '{print $4}'`;

but filesortedlist contains only the first of the sorted file(i.e fancy_STD_AU_2008-03-05.dat).
I want all the sorted files into a variable(filesortedlist).

Regards,
Gyan

Re: sorting files

James R. Ferguson — Wed, 09 Apr 2008 13:42:41 GMT

Hi:

If you want to use Dennis' solution, you began with:

> local filelist="$@"
> local filesortedlist=`echo $filelist | awk -F_ '

This would work if you changed the 'echo' to 'cat' in order to read the file represented by 'filelist'. You don't need to create a separate process with 'cat' to pipe the output to 'awk'. That is wasteful!

Using Dennis's solution and your surrounding shell logic (as written):

#!/usr/bin/sh
local filelist="$@"
local filesortedlist=`awk -F_ '
{
if ($2 == "LANG") {
# combine fields
country = $4
lang = "STDLANG"
} else {
country = $3
lang = $2
}
print country, lang, $1, $0
}' $filelist | sort | awk '{print $4}'` ;
echo ${filesortedlist}

...That is, 'awk' takes the first argument supplied to it as a FILE to be opened and read. The output of the 'awk' process is then piped to a 'sort'.

Perl makes this process-efficient.

Regards!

...JRF...

Re: sorting files

Hein van den Heuvel — Wed, 09 Apr 2008 14:06:06 GMT

well, this seams a perfect excuse te learn perl to me!

If you do, be sure to think outside the immediate question and ask yourself how the input came to be, and what is going to be done with the output.
You may well find that those steps are in fact better done in perl as well.

Anyway... the awk solution failed because all data line were really words in a string.
So you'd have to split those first.

This calls for a minor variation on Dennis's suggestion:

Too bad hpux does not have an asorti function like 'gawk'

$ cat x
fancy_LANG_STD_AU_2008-03-05.dat
fancy_LANG_STD_HK_2008-03-06.dat
fancy_LANG_STD_NZ_2008-03-05.dat
fancy_STD_AU_2008-03-05.dat
fancy_STD_HK_2008-03-06.dat
fancy_STD_NZ_2008-03-05.dat
$ local filelist=$(cat x)
$ echo $filelist
...
$ echo $filelist | awk '{while (++ifancy_STD_AU_2008-03-05.dat
fancy_LANG_STD_AU_2008-03-05.dat
fancy_STD_HK_2008-03-06.dat
fancy_LANG_STD_HK_2008-03-06.dat
fancy_STD_NZ_2008-03-05.dat
fancy_LANG_STD_NZ_2008-03-05.dat

core:
while (++isplit($i,x,"_") # each field is split into array x by "_"
y=(x[2]=="LANG")?4:3 # is sub-field 2 = LANG?
# then language is in 4, not 3.
print x[y],y,$i} # print language,helper,file ready for sorting

Re: sorting files

Gyankr — Thu, 10 Apr 2008 05:57:03 GMT

Thanks everybody.I guess its time for me to start cramping perl

Regards,
Gyan

Re: sorting files

Gyankr — Thu, 01 May 2008 16:32:57 GMT

Hi everybody,

Just when things were going fine,i got a big hiccup, the filelist has more than 100 files and when i use the below code

local flist="$@"
local filelist=`echo $flist | awk '{
while (++i<=NF) {
split($i,x,"_");
y=(x[2]=="LANG")?4:3;
print x[y],y,$i}
}' | sort -t"-" +1 +2 | awk '{print $3}'`;

I am getting an error like

: Input line FANCY_LANG_STD_AU_20 cannot be longer than 3,000 bytes.

I tried something like

local filelist=`echo $flist | fold -w | awk '{.... but the last the sorting of the files fails :(

Regards,
Gyan

Re: sorting files

James R. Ferguson — Thu, 01 May 2008 16:51:16 GMT

Hi Gyan:

> Just when things were going fine,i got a big hiccup, the filelist has more than 100 files

This is yet another reason to use Perl. I suggest you use Hein's Perl solution. You can specific mulitple files as the argument:

# perl -e ' ... ' file1 file2 ...

OR you could do:

# perl -e ' ... ' $(cat /home/filelist)

...where '/home/filelist' contains the list of filenames (one per line) that you want to process.

Regards!

...JRF...

Re: sorting files

Gyankr — Thu, 01 May 2008 17:05:17 GMT

Hi JRF,

I agree with you but the problem is the scripts were already written using bash.

Regards,
Gyan

Re: sorting files

James R. Ferguson — Thu, 01 May 2008 17:11:52 GMT

Hi (again) Gyan:

> I agree with you but the problem is the scripts were already written using bash.

Well, 'awk' has limits that Perl doesn't. If by 'bash' you mean a shell script that invokes 'awk', then I'd ask you why a shell script that invokes a Perl script is any different. After all, isn't the objective to get the job done?

Regards!

...JRF...

Re: sorting files

Dennis Handly — Thu, 01 May 2008 20:17:32 GMT

>i got a big hiccup, the filelist has more than 100 files and when i use the below code
>I am getting an error like
Input line FANCY_LANG_STD_AU_20 cannot be longer than 3,000 bytes.

My original file based script works fine since it only has one file per line.

So you need to change your flist and write each filename to a file:
for i in $flist; do
echo $i
done > file