how to split file and give prefix name

[Fransisco Aris]

Hi Guys,

I wanna ask about shell scripting.
I have file move_data.sql and it has 600 line command, how to split that file per 100 line and I expect result is, move_data1.sql, move_data2.sql, etc... so I can run it at the same time.

I have try with split, but it always give default prefix aa, ab, etc after file name. please give me an enlightenment guys..
thanks.

[Kapil Jha]

m not sure why you want to run a scripts in pieces.

but i support cut and paste command can help you.

BR,
Kapil+

[Dennis Handly]

You can use awk to split the file for every 100 lines.

Or you can just use a shell while read loop:
(( limit = 100 ))
(( n = 1 ))
(( count = 0 ))
file="move_data$n.sql"
while read line; do
echo "$line" >> $file
(( count += 1 ))
if (( count >= limit )); then
(( count = 0 ))
(( n += 1 ))
file="move_data$n.sql"
fi
done < move_data.sql

[rariasn]

Hi,

man csplit


csplit -k -f move_data.sql move_data.sql 100 "{100}"

rgs,

[Raj D.]

Francisco,

Remember csplit will result the first file to 99 , and it will add 00 suffix, after the extention i.e it will result move_data.sql00 , move_Data.sql01 ...and so on.



The best way perhaps , and perfect with the 600 line command file:


# cat move_data.sql | sed -n '1,100p' > move_data.1.sql
# cat move_data.sql | sed -n '101,200p' > move_data.2.sql
# cat move_data.sql | sed -n '201,300p' > move_data.3.sql
# cat move_data.sql | sed -n '301,400p' > move_data.4.sql
# cat move_data.sql | sed -n '401,400p' > move_data.5.sql
# cat move_data.sql | sed -n '401,500p' > move_data.5.sql
# cat move_data.sql | sed -n '501,600p' > move_data.6.sql



# wc -l *.sql
100 move_data.1.sql
100 move_data.2.sql
100 move_data.3.sql
100 move_data.4.sql
100 move_data.5.sql
100 move_data.6.sql
600 move_data.sql

-----------------------------


Cheers, Have Fun!,
Raj.

[Raj D. · Accepted solution]

Fransisco,

>I have file move_data.sql and it has 600 line command, how to split that file per 100 line and I expect result is, move_data1.sql, move_data2.sql, etc


The best way , you can do as below:


#-------------------------------------------
j=1 ; for i in 1 2 3 4 5 6
do
k=`expr $i \* 100` ; cat move_data.sql | sed -n "$j,$k"'p' > move_data$i.sql
j=`expr $j + 100` ; ls -l move_data$i.sql
done
#------------------------------------------


-rw-r--r-- 1 root root 4533 May 26 20:10 move_data1.sql
-rw-r--r-- 1 root root 4518 May 26 20:10 move_data2.sql
-rw-r--r-- 1 root root 4649 May 26 20:10 move_data3.sql
-rw-r--r-- 1 root root 4533 May 26 20:10 move_data4.sql
-rw-r--r-- 1 root root 4518 May 26 20:10 move_data5.sql
-rw-r--r-- 1 root root 4234 May 26 20:10 move_data6.sql


# wc -l move_data?.sql
100 move_data1.sql
100 move_data2.sql
100 move_data3.sql
100 move_data4.sql
100 move_data5.sql
100 move_data6.sql
600 total
#



Enjoy, Have fun!,
Raj.

[James R. Ferguson]

Hi:

> Raj: The best way , you can do as below:
#-------------------------------------------
j=1 ; for i in 1 2 3 4 5 6
do
k=`expr $i \* 100` ; cat move_data.sql | sed -n "$j,$k"'p' > move_data$i.sql
j=`expr $j + 100` ; ls -l move_data$i.sql
done
#------------------------------------------

This is hardly the "best" from a performance standpoint.

First, you don't need the extra 'cat' process since 'sed' can read thusly:

# sed -n "$j,$k"'p' move_data.sql > move_data$i.sql

# k=$(( i * 100 ))
# j=$(( j + 100 ))

Second, the use of 'expr' to do arithmetic that the shell could do (as Dennis showed) is a waste of another, very slow process.

Third, if you were doing repetitive arithmetic, using 'typeset -i ' to declare your variables makes the arithmetic faster.

TMTOWTDI

Regards!

...JRF...

[Raj D.]

James,
Thanks! for pointing out the draw back of the code, but I think for the above problem job done is more important than the performance stand point, .
I checked for a 5000 line to split with the shell arithmetic, to 50 file it takes 1.24 sec for a single cpu system., with typeset -i set , it takes 1.22sec , where as the split -l command splits in real quick (real 0m0.15s ) (6 times faster, that makes sense. However the desired output cannot be achieved without compromising little time in this case to process it in the loop seems to be. Thank you ..TMTOWTDI agree..
Raj.

[Fransisco Aris]

fortunately this case already done by other vendor, but i've done some experiment, split command looks more simpler and efficient I think. anyway as usual, superb thanks for great support guys..