parse and check file and starts a another sub script

[Billa-User]

i have a file like "xargs.lis" and check the file and starts another script like :

XARGS_LISFILE=xargs.lis

if [ `egrep -v '^([A-Za-z0-9-]+ )?[A-Za-z0-9_-]+(\.([A-Za-z]){1,3})?$' ${XARGS_LISFILE} | \
wc -l ` -eq 0 ]
then
xargs -l -x ./sub_xargs-test-lis.sh "TestVersion" < ${XARGS_LISFILE}
..
.
the entries of file "xargs.lis" can be :
filename
appli-directory filename

example:
OK : filename-1.sql
OK : filename_1
OK : filename
Wrong,because "" at the end: filename1
Wrong,because at the end: frm_filename.fmb^I
OK : frm_filename.fmb
OK : applidir filename.fmb
Wrong,because"^M" at the end: applidir filename1.fmb^M
Wrong,because" at the end: applidir filename2.fmb

i have to extend the entries "xargs.lis" with :

filename project-number project-description
appli-directory filename project-number project-description

Field Seperator should be " "

example:
frm_filename.fmb 4711 this a description of project test
testdir frm_filename.fmb 4711 this a description of project test

hmm , how can i handle this issue ?
the "project-description" should be for sub-script one parameter because i want to work a variable like project_description

regards

[James R. Ferguson]

Hi:

Why do you care if there is trailing white-space after your filenames?

Why not use 'awk' and let it split() record into component fields which you then assess? If you do that, you can treat a number of fields as a group, like:

# echo "a b c "|awk '{if (NF>3) {print $(NF-1),$NF} else {print $NF}}'

Regards!

...JRF...

[Bob_Vance · Accepted solution]

You can do a lot with a 'while read' loop.

Not sure where you're getting the project number and desc,
but here's a possible start

Strip off the trash first:
Note: carriage return, ^M, is entered with ctl-V ctl-M
in succession
Tab, ^I, is ctl-V ctl-I or just TAB key
it will just show as a clear tab not ^I

Note: the 1st -e deletes ALL CR
the 2nd -e deletes trailing whitespace
the 3rd -e deletes any residual empty lines

#
(sed < ${XARGS_LISFILE} \
-e 's/[^M]//g' \
-e 's/[ ^I]*$//' \
-e '/^[ ^I]*$/d' \
|while read \
dir file
do
echo $dir $file descr
done
) \
> outfile

Note that when there is only a file, then $dir is null
so $file will in effect start in col1.


bv

[Billa-User]

> (sed < ${XARGS_LISFILE} ...

copy/paste of "sed" and other commands produce a empty file "outfile"

i append a demo text file in the attachment.

regards

[Bob_Vance]

Your original description did not show anything on the input file but:

filename
or
appli-directory filename
.
If you have possible trash after those,
then you have a more complicated scenario.

What is the *exact* format of your input file
and where/how was this input file generated?

bv

[Bob_Vance]

You can't do a simple copy/paste from what I posted.
(I have attached my script)

Unfortunately, the HPUX 'sed' doesn't support matching octal or hex characters, so you must type in the special characters as I explained.

The ^M and ^I are *representative* of a ctl-M (which is a carriage return, "CR") and a ctl-I (which is a TAB character).
Normally, you would not have CRs in your data, unless the file was created on a Windows box, or is Printer output to a file, in which case you might also have Form Feeds in there.


To enter control chars directly is problematic, but possible in 'vi' for ctl chars below hex 0x20 (blank) with the sequence

ctl-V then ctl-Cchar

where Cchar is determined as follows:

. you must know the octal/hex/decimal code of the control char you want to enter:
((here is a conversion chart
http://www.robelle.com/smugbook/ascii.html
))

e.g., a TAB is hex 0x9 or decimal 9
a CR is hex 0xD or decimal 13
a FF (form feed) is hex 0xC or decimal 12

. counting the alphabet,

the 9th char is I, hence ctl-I
the 12th char is L, hence, ctl-L
the 13th char is M, hence, ctl-M
.

I should have explained that you should be in either
'vi' creating a script
*or*
in "vi" mode in the shell editor at command line.
If you are in posix shell, ksh, or Bourne shell,
then you can enter command-line "vi" mode with

set -o vi
.

So, you can paste my stuff into a script,
*BUT*, you must edit and delete the ^I and ^M and then insert the Tab with a ctl-V ctl-I
and the CR wirh a ctl-V ctl-M
.

bv

[James R. Ferguson]

Hi:

If you want to deal with carriage-returns and the end-of-file marker line that DOS/Windows inserts the easiest way is to use 'dos2ux' to eliminate them.

See the manpages for 'dos2ux(1)'. You can see the special characters in your attached file with:

# cat -etv file

Regards!

...JRF...

[Billa-User]

hello,

with external help (i am not a perl expert :-(( ) we created following perl script below.
it make all checks like above and print out the "invalid" lines. description is print out as MIME::Base64:encode_base64. it is easier to handle in shell env for me. hmm , i don't know reg expression in a shell env ?

parse_check_file.pl

when i use it in a shell script :

- this works well and process every line:

XARGS_LISFILE=xargs.lis

./parse_check_file.pl ${XARGS_LISFILE} | \
while read line
do
./sub-script.sh "TestVersion" "X" "${line}"
done

- this works with "xargs" not ok because it merge multiple lines, it ignores "\n" of perl script:

XARGS_LISFILE=xargs.lis

./parse_check_file.pl ${XARGS_LISFILE} | \
xargs -l -x ./sub-script.sh "TestVersion" "X"

regards

[James R. Ferguson]

Hi (again):

> this works with "xargs" not ok because it merge multiple lines, it ignores "\n" of perl script:

It is 'xargs' that is doing what you told it to do.

IF I understand your problem, try:

# ./parse_check_file.pl ${XARGS_LISFILE} | \
xargs -L1 ./sub-script.sh "TestVersion" "X"

Regards!

...JRF...

[Billa-User]

hello also again,

@ It is 'xargs' that is doing what you told it to do.

i tried several options and only one option works well.

i explain it:

in parse_check_file.pl i use:

print "$lisfile_var $File $Appldir $ProjNr $ProjDesc\n";

it produce an output like (in attachment):

filename.sql
fr_filename.fmb
fr_filename.fmb applidir
fr_filename.fmb 4711 dGhpcyBpcyBhIGRlc2NyaXB0aW9uIG9mIHByb2plY3QgNDcxMQ==
fr_filename.fmb applidir 4711 dGhpcyBpcyBhIGRlc2NyaXB0aW9uIG9mIHByb2plY3QgNDcxMQ==

when i use ./parse_check_file.pl ${XARGS_LISFILE} | \
xargs -L1 ./sub-script.sh "TestVersion" "X"

then i get :
with sub-script.sh

#!/usr/bin/ksh
exec 0echo "Number. Par: $#"
echo "Parameter: $*"

output :

Number. Par: 10
Parameter: TestVersion X L filename.sql fr_filename.fmb fr_filename.fmb applidir fr_filename.fmb 4711 dGhpcyBpcyBhIGRlc2NyaXB0aW9uIG9mIHByb2plY3QgNDcxMQ==
Number. Par: 7
Parameter: TestVersion X L fr_filename.fmb applidir 4711 dGhpcyBpcyBhIGRlc2NyaXB0aW9uIG9mIHByb2plY3QgNDcxMQ==

put not for example line 1 of output :
Number. Par: 3

xargs in combination with perl works well like this:

i use in perl for print ' (single quote) for the output like:
print "'$lisfile_var $File $Appldir $ProjNr $ProjDesc'\n";

./parse_check_file.pl ${XARGS_LISFILE} | \
xargs -l -x ./sub-script.sh "TestVersion" "X"

works well and line per line.

i put output and sub-script in attachment and you can see test it like:

cat attachment-file | \
xargs -l -x ./sub-script.sh "TestVersion" "X"

regards