topic Re: Search and replace script in Operating System - Linux

Search and replace script

Coolmar — Mon, 19 Jun 2006 09:58:34 GMT

Hi,

The /etc/hosts files are going to be changing on all our servers soon as all our IP address are changing. Each hosts file is different. I wonder if it is possible to create a script that will take each host and verify the current IP with an nslookup - if it matches, leave it - but if not, replace with the new address.

Thanks,
S.

Re: Search and replace script

Ian Vaughan — Mon, 19 Jun 2006 10:10:06 GMT

Hi Sally,
I would be tempted to make up a good "master" /etc/hosts that I know is right and overwrite the ones out in the field.
Do yuo have servers that are known by different names by other machines?
If you are confident in the info on your name servers (assuming that you have more than one for redundancy) you could ignore the hosts file altogether and manage everything through DNS by changing your nsswitch.conf.

regards
Ian

Re: Search and replace script

Steven E. Protter — Mon, 19 Jun 2006 10:13:12 GMT

Shalom Sally,

I would suggest creating a file list on each server and using sed

filelist

/etc/hosts
/etc/rc.config.d/netconf
# add what you need here for each server.

The script(outline)

while read -r filename
do
sed s/192.168.0.10/10.63.31.245/g $filename >filename.new
mv $filename.new $filename
done < filename

If you are careful, you can generate and edit the file list as follows

find / -exec grep -l '192.168.0.10' {} \; > filelist

edit it and get ready to run.

This same basic concept can be used for many kinds of editing.

SEP

Re: Search and replace script

Ninad_1 — Mon, 19 Jun 2006 10:23:15 GMT

Please use the below script

#!/usr/bin/ksh
egrep -v "^#|^$" /etc/hosts | awk '{print $1,$2}' | while read ipaddr hostnm rest
do
nslookupip=$(nslookup $hostnm 2>/dev/null | awk '/Trying DNS/,/zzz/' | grep -i address | awk '{print $NF}')
if [[ -z $nslookupip ]]
then
echo blank
else
if [[ "$ipaddr" = "$nslookupip" ]]
then
echo $hostnm ip address not changed >> newhosts.log
echo $ipaddr $hostnm $rest >> newhosts
else
echo $hostnm Change ip to $nslookupip >> newhosts.log
echo $nslookupip $hostnm $rest >> newhosts
fi
fi
done

The newly generated file will be newhosts and log file will be newhosts.log
Check before replacing the hosts file and always take a backup of the current hosts file before replacing.

Regards,
Ninad

Re: Search and replace script

Coolmar — Mon, 19 Jun 2006 10:46:04 GMT

Thanks Ninad...that is a great script. My only problem is that my nsswitch.conf (which I cannot change) has it looking at files first and then DNS second. So in that order, your script will not work unless I can do an nslookup with a force to DNS only.

S.

Re: Search and replace script

Mike Stroyan — Mon, 19 Jun 2006 11:09:28 GMT

The nslookup command can be given a dns server name as a second parameter. That will force it to us DNS when /etc/nsswitch.conf says to start with /etc/hosts.

Re: Search and replace script

James R. Ferguson — Mon, 19 Jun 2006 11:12:25 GMT

Hi Sally:

Here's a simple Perl script that will verify '/etc/hosts' files. The script will report (print) any entries where the hostname does *not* resolve to the listed IPaddress.

For example, given an entry like:

10.11.10.11 myserver

This line would be printed if the number and name were not equivalent.

# cat .hostproof
#!/usr/bin/perl
use strict;
use warnings;
use Socket;
while (<>) {
next if /^#/;
next if /^$/;
my @F = split;
my $derived_ip = gethostbyname($F[1]);
print if $F[0] ne inet_ntoa($derived_ip);
}
1;

...Run as :

# ./hostproof /etc/hosts

Regards!

...JRF...

Re: Search and replace script

James R. Ferguson — Mon, 19 Jun 2006 12:51:08 GMT

Hi (again) Sally:

OK, I reread your post. The first Perl script I suggested does a verification only of an '/etc/hosts' file to report lines that require change.

This version will recreate an 'etc/hosts' file fixing any IP addresses to match the currently derived address associated with each hostname.

The '/etc/hosts' file is preserved as '/etc/hosts.old' and a new file is generated.

# cat .hostfix
#!/usr/bin/perl -ni.old
use strict;
use warnings;
use Socket;
/^#/ and print, next;
/^\s*$/ and print, next;
my @F = split;
my $packed_ip = gethostbyname($F[1]);
my $derived_ip = inet_ntoa($packed_ip);
s/\s*\d+\.\d+\.\d+.\d+/$derived_ip/ if ($F[0] ne $derived_ip);
print;
1;

...Run as:

# ./hostfix /etc/hosts

Regards!

...JRF...

Re: Search and replace script

Ninad_1 — Tue, 20 Jun 2006 03:13:21 GMT

Hi Sally,

Sorry I had left office so couldnt reply.
Is your query been resolved ? Or you still need any other way. Do let us know, so that only if you still need something , can work out on a resolution.

Regards,
Ninad

Re: Search and replace script

Coolmar — Tue, 20 Jun 2006 06:54:44 GMT

Hi Ninad,
Still fighting with it. Our IP addresses have to change when our servers move. I figure relying on the DNS might not be a good idea either. Plus there are some systems that don't have DNS access. So my new idea is to create a flat file with the new addresses of the servers that are changing. Then scan each server for any of the addresses in the flat file and if it is there, take the new address. So here is how I modified your script below. The problem that I am having is that only the last entry actually gets changed. The others do but then go back to the original value and only the last one changes. The problem is in the sed line and to the filename.new.

if [ `whoami` != "root" ]; then
echo "This script must be run with root permissions!"
exit
fi
filename=/etc/hosts
d=$HOME/scripts/move
ips=$d/ips.list
if [ -f $d/$filename.new ]; then
rm $d/$filename.new
fi
egrep -v "^#|^$|localhost" /etc/hosts | awk '{print $1,$2}' | while read ipaddr hostnm
do
for host in `echo $hostnm |awk -F. '{print $1}'`; do
#host=`echo $hostnm |awk -F. '{print $1}'`
newip=`grep $host $ips |awk '{print $2}'`
if [[ -z $newip ]]
then
echo blank
else
if [[ "$ipaddr" != "$newip" ]]
then
cp $filename $filename.orig
# sed s/$ipaddr/$newip/g $filename >>$filename.new
sed s/$ipaddr/$newip/g $filename
# mv $filename.new $filename
fi
fi
done
done

Re: Search and replace script

Ninad_1 — Tue, 20 Jun 2006 07:32:15 GMT

OK, I think I am understanding your requirement in a better way now than I did before.
So you are saying that the ip addresses have not changed yet but are going to change - thus we cannot verify using either ping nor DNS.
So as you have found an alternative - write ip addresses to a flat file - I agree with you is the most suitable option for you.
Now as I understand what needs to be done is read hosts file , check ip address has changed or not from the flat file and update the entry if ip addr changed and leave as is if not changed.
I would propose you a simpler solution.
The logic is if you find entry in flat file - means ip address is changed - no need to check in hosts files again.
If no entry found in flat file - means ip address is unchanged.
Also I recommend you not to change the original hosts file directly. First create a newhosts file with updated entries and then replace the hosts file

Another thing - this flat file I suggest you create as following format
hostname changedipaddress
The reason behind this is in grep when we grep hostname it will match all occurances having that text
e.g. if you have server names as prod1 and prod1dr then when you grep prod - it will match both the entries, where as the method I have used is
grep "$hostnm " so I have added a space after the hostname so that it will be an exact match for one server only and this is more easy if you have the flatfile with
hostname first then a space and then ipaddress.

if [ `whoami` != "root" ]; then
echo "This script must be run with root permissions!"
exit
fi
d=$HOME/scripts/move
ips=$d/ips.list
if [ -f $d/hosts.new]; then
rm $d/$filename.new
fi
egrep -v "^#|^$|localhost" /etc/hosts | while read ipaddr hostnm remainingline
do
if [[ $(grep -c -i "$hostnm " $ips) != "0" ]]
then
echo $ipaddr $hostnm $remainingline >> $d/hosts.new
else
echo "$(grep "$hostnm " $ips | awk '{print $2}') $hostnm $remainingline" >> $d/hosts.new
fi
done
#cp /etc/hosts /etc/hosts.$(date +%Y%m%d%H%M)
#cp $d/hosts.new /etc/hosts

I have commented the last 2 entries - as I have not checked this script throughly - so request you to check first - if it is to your satisfaction remove the # and execute.

Pls let know if you need any other help.

Regards,
Ninad

Re: Search and replace script

James R. Ferguson — Tue, 20 Jun 2006 08:11:45 GMT

Hi (again) Sally:

Here's another Perl script to accomodate your changes.

Modify the __DATA__ section of the script below to specify lines of pairs of IP addresses as 'before' and 'after' replacements. The address pairs can be separated by blanks and/or tab characters.

Then simply run the script passing your '/etc/hosts' file as its argument. A backup copy of the unmodified file will be retained automatically as '/etc/hosts.old'.

# cat .hostfix
#!/usr/bin/perl -i.old
use strict;
use warnings;

my @tokens;
my ($i, $x, $y);
{
local $/ = undef;
@tokens = split(/\s+/,);
}
while (<>) {
for ($i=0; $i < scalar @tokens; $i+=2) {
($x, $y) = @tokens [ $i, $i+1 ];
s/$x/$y/;
}
print;
}
__DATA__
10.10.11.12 10.10.11.13
10.10.11.14 10.10.11.15
10.10.11.16 10.10.11.17

...after modifying the __DATA__ lines, run as:

# ./hostfix /etc/hosts

Regards!

...JRF...

Re: Search and replace script

H.Merijn Brand (procura — Tue, 20 Jun 2006 08:32:31 GMT

jrf, that script is very dangerous, as your DATA section lists the IP's with dots, and the dots being special in regular expressions and you did not anchor the words

my %change = qw{
10.10.11.12 10.10.11.13
10.10.11.14 10.10.11.15
10.10.11.16 10.10.11.17
};

while (<>) {
foreach my $org_ip (keys %change) {
s/\b\Q$org_ip\E\b/$change{$org_ip}/;
}
print;
}

might be a bit more safe :)

Enjoy, Have FUN! H.Merijn

Re: Search and replace script

Coolmar — Tue, 20 Jun 2006 08:46:04 GMT

Ok Ninad...I am almost there. My only proble now is that my flat file will look like the following:

10.l0.10.1 host
10.10.10.2 host-swc

Then the host files are like the following:

20.20.20.1 host.domain.com host
20.20.20.2 anotherhost
20.20.20.3 host-swc

So I go one step further in the script and echo $hostnm |awk -F. '{print $1}' so that I just have the name "host" .... but when the flat file is scanned for "host" it comes back with both host and host-swc and then the hosts.new gets screwed up.

Re: Search and replace script

Ninad_1 — Tue, 20 Jun 2006 08:59:59 GMT

It seems that you already have pretty much mastered shell scripting, so just explaning you the logic to be changed and not the whole script :)

Just wherever you are using grep to search the hostname occurance in flatfile, use
grep " $hostnm$" flatfilename

So that it will check for a space before the hostname and check that there are no other characters after the hostname [ The $ at the end meaning ending with ]

Hope this helps.

Regards,
Ninad

Re: Search and replace script

James R. Ferguson — Tue, 20 Jun 2006 09:00:21 GMT

Hi Merijn:

Yes, you're right, of course. I was relying entirely too much on decent data and failed to be rigouous. At the least, I would have made my substitution:

s/\b$x\b/$y/;

Your variation helps me, though. It addresses the dot character's ambiguity. Thanks, Merijn!

Regards!

...JRF...

Re: Search and replace script

Ninad_1 — Tue, 20 Jun 2006 09:07:18 GMT

Sorry,
One more thing - if you are using the modified script I gave - the latest script from my posts where I suggested flatfile format to be hostname and then ip address - change the $2 to $1 to get ip address from the flatfile as per your format.

Regards,
Ninad

Re: Search and replace script

Coolmar — Tue, 20 Jun 2006 09:38:16 GMT

Ninad...

I have the following (see below) and the hosts.new is good except for the entry with 10.10.10.1 host.domain.com host
That one doesn't change at all.

if [ `whoami` != "root" ]; then
echo "This script must be run with root permissions!"
exit
fi
d=$HOME/scripts/move
ips=$d/ips.list
if [ -f $d/hosts.new ]; then
rm $d/$filename.new
fi
egrep -v "^#|^$|localhost" /etc/hosts | while read ipaddr hostnm remainingline
do
if [[ $(grep -c -i "$hostnm$" $ips) != "0" ]]
then
echo `grep $hostnm$ $ips`>> $d/hosts.new
else
echo "$ipaddr $hostnm $remainingline" >> $d/hosts.new
fi
done
#cp /etc/hosts /etc/hosts.$(date +%Y%m%d%H%M)
#cp $d/hosts.new /etc/hosts

Re: Search and replace script

Coolmar — Tue, 20 Jun 2006 09:48:41 GMT

What I am thinking is that I want to somehow grep the hosts file for $hostnm and then awk out the hosts with no "." in them. So that would remove the host.domain.com strings and then it should work. Right? But I don't know how or if I can awk out the "." hosts.

Re: Search and replace script

Coolmar — Tue, 20 Jun 2006 09:56:06 GMT

Actually, what I suggested above would be no good because then I would lose the IP address...DOH!
So my next thought is possibly to force grep somehow to take the $hostnm *exactly* has it appears...so $hostnm would scan the file for "host" only and not the other variations like "host-swc". Is that possible?