Operating System - HP-UX
1830050 Members
29053 Online
109998 Solutions
New Discussion

how to split file and give prefix name

 
SOLVED
Go to solution
Fransisco Aris
Frequent Advisor

how to split file and give prefix name

Hi Guys,

I wanna ask about shell scripting.
I have file move_data.sql and it has 600 line command, how to split that file per 100 line and I expect result is, move_data1.sql, move_data2.sql, etc... so I can run it at the same time.

I have try with split, but it always give default prefix aa, ab, etc after file name. please give me an enlightenment guys..
thanks.
8 REPLIES 8
Kapil Jha
Honored Contributor

Re: how to split file and give prefix name

m not sure why you want to run a scripts in pieces.

but i support cut and paste command can help you.

BR,
Kapil+
I am in this small bowl, I wane see the real world......
Dennis Handly
Acclaimed Contributor

Re: how to split file and give prefix name

You can use awk to split the file for every 100 lines.

Or you can just use a shell while read loop:
(( limit = 100 ))
(( n = 1 ))
(( count = 0 ))
file="move_data$n.sql"
while read line; do
echo "$line" >> $file
(( count += 1 ))
if (( count >= limit )); then
(( count = 0 ))
(( n += 1 ))
file="move_data$n.sql"
fi
done < move_data.sql
rariasn
Honored Contributor

Re: how to split file and give prefix name

Hi,

man csplit


csplit -k -f move_data.sql move_data.sql 100 "{100}"

rgs,
Raj D.
Honored Contributor

Re: how to split file and give prefix name

Francisco,

Remember csplit will result the first file to 99 , and it will add 00 suffix, after the extention i.e it will result move_data.sql00 , move_Data.sql01 ...and so on.



The best way perhaps , and perfect with the 600 line command file:


# cat move_data.sql | sed -n '1,100p' > move_data.1.sql
# cat move_data.sql | sed -n '101,200p' > move_data.2.sql
# cat move_data.sql | sed -n '201,300p' > move_data.3.sql
# cat move_data.sql | sed -n '301,400p' > move_data.4.sql
# cat move_data.sql | sed -n '401,400p' > move_data.5.sql
# cat move_data.sql | sed -n '401,500p' > move_data.5.sql
# cat move_data.sql | sed -n '501,600p' > move_data.6.sql



# wc -l *.sql
100 move_data.1.sql
100 move_data.2.sql
100 move_data.3.sql
100 move_data.4.sql
100 move_data.5.sql
100 move_data.6.sql
600 move_data.sql

-----------------------------


Cheers, Have Fun!,
Raj.
" If u think u can , If u think u cannot , - You are always Right . "
Raj D.
Honored Contributor
Solution

Re: how to split file and give prefix name

Fransisco,

>I have file move_data.sql and it has 600 line command, how to split that file per 100 line and I expect result is, move_data1.sql, move_data2.sql, etc


The best way , you can do as below:


#-------------------------------------------
j=1 ; for i in 1 2 3 4 5 6
do
k=`expr $i \* 100` ; cat move_data.sql | sed -n "$j,$k"'p' > move_data$i.sql
j=`expr $j + 100` ; ls -l move_data$i.sql
done
#------------------------------------------


-rw-r--r-- 1 root root 4533 May 26 20:10 move_data1.sql
-rw-r--r-- 1 root root 4518 May 26 20:10 move_data2.sql
-rw-r--r-- 1 root root 4649 May 26 20:10 move_data3.sql
-rw-r--r-- 1 root root 4533 May 26 20:10 move_data4.sql
-rw-r--r-- 1 root root 4518 May 26 20:10 move_data5.sql
-rw-r--r-- 1 root root 4234 May 26 20:10 move_data6.sql


# wc -l move_data?.sql
100 move_data1.sql
100 move_data2.sql
100 move_data3.sql
100 move_data4.sql
100 move_data5.sql
100 move_data6.sql
600 total
#



Enjoy, Have fun!,
Raj.
" If u think u can , If u think u cannot , - You are always Right . "
James R. Ferguson
Acclaimed Contributor

Re: how to split file and give prefix name

Hi:

> Raj: The best way , you can do as below:
#-------------------------------------------
j=1 ; for i in 1 2 3 4 5 6
do
k=`expr $i \* 100` ; cat move_data.sql | sed -n "$j,$k"'p' > move_data$i.sql
j=`expr $j + 100` ; ls -l move_data$i.sql
done
#------------------------------------------

This is hardly the "best" from a performance standpoint.

First, you don't need the extra 'cat' process since 'sed' can read thusly:

# sed -n "$j,$k"'p' move_data.sql > move_data$i.sql

# k=$(( i * 100 ))
# j=$(( j + 100 ))

Second, the use of 'expr' to do arithmetic that the shell could do (as Dennis showed) is a waste of another, very slow process.

Third, if you were doing repetitive arithmetic, using 'typeset -i ' to declare your variables makes the arithmetic faster.

TMTOWTDI

Regards!

...JRF...

Raj D.
Honored Contributor

Re: how to split file and give prefix name


James,
Thanks! for pointing out the draw back of the code, but I think for the above problem job done is more important than the performance stand point, .
I checked for a 5000 line to split with the shell arithmetic, to 50 file it takes 1.24 sec for a single cpu system., with typeset -i set , it takes 1.22sec , where as the split -l command splits in real quick (real 0m0.15s ) (6 times faster, that makes sense. However the desired output cannot be achieved without compromising little time in this case to process it in the loop seems to be. Thank you ..TMTOWTDI agree..
Raj.
" If u think u can , If u think u cannot , - You are always Right . "
Fransisco Aris
Frequent Advisor

Re: how to split file and give prefix name

fortunately this case already done by other vendor, but i've done some experiment, split command looks more simpler and efficient I think. anyway as usual, superb thanks for great support guys..