script to split large file

[george_57]

Hi Experts,
Could someone help me with a script im trying to write to split a large file into smaller files of 450 lines each. The output files (split files) could have the same name as the input file with a .01 .02 .03 and so on extension. or it could be an xx00 or xx01 using csplit.... Here is something i tried but it works only until it reaches the first 450 lines...Could somebody help me to set a counter and keep splitting till end of file....
##############################
#!/bin/ksh
echo "Enter file name to split :"
read infile

line=$(cat $infile | wc -l)

if(($line>450))
then
csplit $infile 451
fi
#############################
Thanks
George

[Rainer von Bongartz · Accepted solution]

Why no use the build-in split command

see man split

Regards
Rainer

[Michael Tully]

You can use 'split' command and implement it into your script.

# /usr/bin/split -l 450

[john korterman]

Hi George,

you can insert a counter like this:


#!/bin/ksh

typeset -i TOTAL_LINES=0 SPLIT_LINES=0 LINE_LIMIT=0
OUTFILE=/tmp/georgeout

# No. of lines per file
LINE_LIMIT=450

echo "Enter file name to split :"
read INFILE

TOTAL_LINES=$(cat $INFILE | wc -l)

while [ $SPLIT_LINES -lt $TOTAL_LINES ]
do
split -l ${LINE_LIMIT} $INFILE $OUTFILE
SPLIT_LINES=$(( $SPLIT_LINES + $LINE_LIMIT ))
done


hope it helps.

Regards,

John K.

[Justo Exposito]

Hi George,

You can use split in order to do the split and then use mv to change the name:

/usr/bin/split -l 450 filename
integer counter=0
for i in filename.*
do
(( counter = $counter + 1 ))
nomfile=$(echo $i | cut -d"." -f1)
mv $i $nomfile.$counter
done

Regards,

Justo.

[Simon Abbott]

Hello,

I would do this with awk. Try this...

#!/bin/ksh

$filename = filename_to_be_split.txt

awk '
BEGIN {
count = 0
filecount = 1
outfile = ARGV[1] "." filecount
}
{
count++
if ( count >= 451 ) {
filecount++
outfile = ARGV[1] "." filecount
count = 1
}
print > outfile
}' $filename

I hope that helps,

Simon.

[Simon Abbott]

Oops!

I went into some perl/awk/shell combined mode on the second line there. It should read:

filename=file_to_be_split.txt

where file_to_be_split.txt is the file you want split.

Simon.

[Nick Wickens]

Hi George

I think your original option of using csplit may have worked if you added the -k option. (Were you getting "out of range" messages ?)

The option helps when you need to specify an arbitarily large repeat argument as otherwise new files are removed.

eg - csplit -k $infile 450 {99}

The {99} should allow up to 100 files called .xx00 thru .xx99 to be created.

[harry d brown jr]

Why not use "split"

split [-l line_count] [-a suffix_length] [file [name]]


do a man on split.

CSPLIT does split'ing on CONTEXT, aka string matches.


live free or die
harry

[Leif Halvarsson_2]

Hi
In most cases split will do this job, but if you for some reason will make your own "split" this can be done with the "exec" command. Here is one example (filesize is 10 lines but this can be changed at your own need).

let line=1
let cntr=1
file=x
exec >$file$cntr
while read a
do


echo $a
if [ $line -eq 10 ]
then
let cntr=$cntr+1
line=0
exec >$file$cntr

fi
let line=$line+1
done

[george_57]

Thanks everybody...One thing I love most about this forum is that the response is almost instant.... Thank You all once again.