Search and replace script

[Coolmar]

Hi,

The /etc/hosts files are going to be changing on all our servers soon as all our IP address are changing. Each hosts file is different. I wonder if it is possible to create a script that will take each host and verify the current IP with an nslookup - if it matches, leave it - but if not, replace with the new address.

Thanks,
S.

[Ian Vaughan]

Hi Sally,
I would be tempted to make up a good "master" /etc/hosts that I know is right and overwrite the ones out in the field.
Do yuo have servers that are known by different names by other machines?
If you are confident in the info on your name servers (assuming that you have more than one for redundancy) you could ignore the hosts file altogether and manage everything through DNS by changing your nsswitch.conf.

regards
Ian

[Steven E. Protter]

Shalom Sally,

I would suggest creating a file list on each server and using sed

filelist

/etc/hosts
/etc/rc.config.d/netconf
# add what you need here for each server.

The script(outline)

while read -r filename
do
sed s/192.168.0.10/10.63.31.245/g $filename >filename.new
mv $filename.new $filename
done < filename


If you are careful, you can generate and edit the file list as follows

find / -exec grep -l '192.168.0.10' {} \; > filelist

edit it and get ready to run.

This same basic concept can be used for many kinds of editing.

SEP

[Ninad_1 · Accepted solution]

Please use the below script

#!/usr/bin/ksh
egrep -v "^#|^$" /etc/hosts | awk '{print $1,$2}' | while read ipaddr hostnm rest
do
nslookupip=$(nslookup $hostnm 2>/dev/null | awk '/Trying DNS/,/zzz/' | grep -i address | awk '{print $NF}')
if [[ -z $nslookupip ]]
then
echo blank
else
if [[ "$ipaddr" = "$nslookupip" ]]
then
echo $hostnm ip address not changed >> newhosts.log
echo $ipaddr $hostnm $rest >> newhosts
else
echo $hostnm Change ip to $nslookupip >> newhosts.log
echo $nslookupip $hostnm $rest >> newhosts
fi
fi
done

The newly generated file will be newhosts and log file will be newhosts.log
Check before replacing the hosts file and always take a backup of the current hosts file before replacing.

Regards,
Ninad

[Coolmar]

Thanks Ninad...that is a great script. My only problem is that my nsswitch.conf (which I cannot change) has it looking at files first and then DNS second. So in that order, your script will not work unless I can do an nslookup with a force to DNS only.

S.

[Mike Stroyan]

The nslookup command can be given a dns server name as a second parameter. That will force it to us DNS when /etc/nsswitch.conf says to start with /etc/hosts.

[James R. Ferguson]

Hi Sally:

Here's a simple Perl script that will verify '/etc/hosts' files. The script will report (print) any entries where the hostname does *not* resolve to the listed IPaddress.

For example, given an entry like:

10.11.10.11 myserver

This line would be printed if the number and name were not equivalent.

# cat .hostproof
#!/usr/bin/perl
use strict;
use warnings;
use Socket;
while (<>) {
next if /^#/;
next if /^$/;
my @F = split;
my $derived_ip = gethostbyname($F[1]);
print if $F[0] ne inet_ntoa($derived_ip);
}
1;

...Run as :

# ./hostproof /etc/hosts

Regards!

...JRF...

[James R. Ferguson]

Hi (again) Sally:

OK, I reread your post. The first Perl script I suggested does a verification only of an '/etc/hosts' file to report lines that require change.

This version will recreate an 'etc/hosts' file fixing any IP addresses to match the currently derived address associated with each hostname.

The '/etc/hosts' file is preserved as '/etc/hosts.old' and a new file is generated.

# cat .hostfix
#!/usr/bin/perl -ni.old
use strict;
use warnings;
use Socket;
/^#/ and print, next;
/^\s*$/ and print, next;
my @F = split;
my $packed_ip = gethostbyname($F[1]);
my $derived_ip = inet_ntoa($packed_ip);
s/\s*\d+\.\d+\.\d+.\d+/$derived_ip/ if ($F[0] ne $derived_ip);
print;
1;

...Run as:

# ./hostfix /etc/hosts

Regards!

...JRF...

[Ninad_1]

Hi Sally,

Sorry I had left office so couldnt reply.
Is your query been resolved ? Or you still need any other way. Do let us know, so that only if you still need something , can work out on a resolution.

Regards,
Ninad

[Coolmar]

Hi Ninad,
Still fighting with it. Our IP addresses have to change when our servers move. I figure relying on the DNS might not be a good idea either. Plus there are some systems that don't have DNS access. So my new idea is to create a flat file with the new addresses of the servers that are changing. Then scan each server for any of the addresses in the flat file and if it is there, take the new address. So here is how I modified your script below. The problem that I am having is that only the last entry actually gets changed. The others do but then go back to the original value and only the last one changes. The problem is in the sed line and to the filename.new.

if [ `whoami` != "root" ]; then
echo "This script must be run with root permissions!"
exit
fi
filename=/etc/hosts
d=$HOME/scripts/move
ips=$d/ips.list
if [ -f $d/$filename.new ]; then
rm $d/$filename.new
fi
egrep -v "^#|^$|localhost" /etc/hosts | awk '{print $1,$2}' | while read ipaddr hostnm
do
for host in `echo $hostnm |awk -F. '{print $1}'`; do
#host=`echo $hostnm |awk -F. '{print $1}'`
newip=`grep $host $ips |awk '{print $2}'`
if [[ -z $newip ]]
then
echo blank
else
if [[ "$ipaddr" != "$newip" ]]
then
cp $filename $filename.orig
# sed s/$ipaddr/$newip/g $filename >>$filename.new
sed s/$ipaddr/$newip/g $filename
# mv $filename.new $filename
fi
fi
done
done

[Ninad_1]

OK, I think I am understanding your requirement in a better way now than I did before.
So you are saying that the ip addresses have not changed yet but are going to change - thus we cannot verify using either ping nor DNS.
So as you have found an alternative - write ip addresses to a flat file - I agree with you is the most suitable option for you.
Now as I understand what needs to be done is read hosts file , check ip address has changed or not from the flat file and update the entry if ip addr changed and leave as is if not changed.
I would propose you a simpler solution.
The logic is if you find entry in flat file - means ip address is changed - no need to check in hosts files again.
If no entry found in flat file - means ip address is unchanged.
Also I recommend you not to change the original hosts file directly. First create a newhosts file with updated entries and then replace the hosts file

Another thing - this flat file I suggest you create as following format
hostname changedipaddress
The reason behind this is in grep when we grep hostname it will match all occurances having that text
e.g. if you have server names as prod1 and prod1dr then when you grep prod - it will match both the entries, where as the method I have used is
grep "$hostnm " so I have added a space after the hostname so that it will be an exact match for one server only and this is more easy if you have the flatfile with
hostname first then a space and then ipaddress.

if [ `whoami` != "root" ]; then
echo "This script must be run with root permissions!"
exit
fi
d=$HOME/scripts/move
ips=$d/ips.list
if [ -f $d/hosts.new]; then
rm $d/$filename.new
fi
egrep -v "^#|^$|localhost" /etc/hosts | while read ipaddr hostnm remainingline
do
if [[ $(grep -c -i "$hostnm " $ips) != "0" ]]
then
echo $ipaddr $hostnm $remainingline >> $d/hosts.new
else
echo "$(grep "$hostnm " $ips | awk '{print $2}') $hostnm $remainingline" >> $d/hosts.new
fi
done
#cp /etc/hosts /etc/hosts.$(date +%Y%m%d%H%M)
#cp $d/hosts.new /etc/hosts


I have commented the last 2 entries - as I have not checked this script throughly - so request you to check first - if it is to your satisfaction remove the # and execute.

Pls let know if you need any other help.

Regards,
Ninad

[James R. Ferguson]

Hi (again) Sally:

Here's another Perl script to accomodate your changes.

Modify the __DATA__ section of the script below to specify lines of pairs of IP addresses as 'before' and 'after' replacements. The address pairs can be separated by blanks and/or tab characters.

Then simply run the script passing your '/etc/hosts' file as its argument. A backup copy of the unmodified file will be retained automatically as '/etc/hosts.old'.

# cat .hostfix
#!/usr/bin/perl -i.old
use strict;
use warnings;

my @tokens;
my ($i, $x, $y);
{
local $/ = undef;
@tokens = split(/\s+/,);
}
while (<>) {
for ($i=0; $i < scalar @tokens; $i+=2) {
($x, $y) = @tokens [ $i, $i+1 ];
s/$x/$y/;
}
print;
}
__DATA__
10.10.11.12 10.10.11.13
10.10.11.14 10.10.11.15
10.10.11.16 10.10.11.17

...after modifying the __DATA__ lines, run as:

# ./hostfix /etc/hosts

Regards!

...JRF...

[H.Merijn Brand (procura]

jrf, that script is very dangerous, as your DATA section lists the IP's with dots, and the dots being special in regular expressions and you did not anchor the words

my %change = qw{
10.10.11.12 10.10.11.13
10.10.11.14 10.10.11.15
10.10.11.16 10.10.11.17
};

while (<>) {
foreach my $org_ip (keys %change) {
s/\b\Q$org_ip\E\b/$change{$org_ip}/;
}
print;
}

might be a bit more safe :)

Enjoy, Have FUN! H.Merijn

[Coolmar]

Ok Ninad...I am almost there. My only proble now is that my flat file will look like the following:

10.l0.10.1 host
10.10.10.2 host-swc

Then the host files are like the following:

20.20.20.1 host.domain.com host
20.20.20.2 anotherhost
20.20.20.3 host-swc

So I go one step further in the script and echo $hostnm |awk -F. '{print $1}' so that I just have the name "host" .... but when the flat file is scanned for "host" it comes back with both host and host-swc and then the hosts.new gets screwed up.

[Ninad_1]

It seems that you already have pretty much mastered shell scripting, so just explaning you the logic to be changed and not the whole script :)

Just wherever you are using grep to search the hostname occurance in flatfile, use
grep " $hostnm$" flatfilename

So that it will check for a space before the hostname and check that there are no other characters after the hostname [ The $ at the end meaning ending with ]

Hope this helps.

Regards,
Ninad

[James R. Ferguson]

Hi Merijn:

Yes, you're right, of course. I was relying entirely too much on decent data and failed to be rigouous. At the least, I would have made my substitution:

s/\b$x\b/$y/;

Your variation helps me, though. It addresses the dot character's ambiguity. Thanks, Merijn!

Regards!

...JRF...

[Ninad_1]

Sorry,
One more thing - if you are using the modified script I gave - the latest script from my posts where I suggested flatfile format to be hostname and then ip address - change the $2 to $1 to get ip address from the flatfile as per your format.

Regards,
Ninad

[Coolmar]

Ninad...

I have the following (see below) and the hosts.new is good except for the entry with 10.10.10.1 host.domain.com host
That one doesn't change at all.

if [ `whoami` != "root" ]; then
echo "This script must be run with root permissions!"
exit
fi
d=$HOME/scripts/move
ips=$d/ips.list
if [ -f $d/hosts.new ]; then
rm $d/$filename.new
fi
egrep -v "^#|^$|localhost" /etc/hosts | while read ipaddr hostnm remainingline
do
if [[ $(grep -c -i "$hostnm$" $ips) != "0" ]]
then
echo `grep $hostnm$ $ips`>> $d/hosts.new
else
echo "$ipaddr $hostnm $remainingline" >> $d/hosts.new
fi
done
#cp /etc/hosts /etc/hosts.$(date +%Y%m%d%H%M)
#cp $d/hosts.new /etc/hosts

[Coolmar]

What I am thinking is that I want to somehow grep the hosts file for $hostnm and then awk out the hosts with no "." in them. So that would remove the host.domain.com strings and then it should work. Right? But I don't know how or if I can awk out the "." hosts.

[Coolmar]

Actually, what I suggested above would be no good because then I would lose the IP address...DOH!
So my next thought is possibly to force grep somehow to take the $hostnm *exactly* has it appears...so $hostnm would scan the file for "host" only and not the other variations like "host-swc". Is that possible?

[Ninad_1]

if [ `whoami` != "root" ]; then
echo "This script must be run with root permissions!"
exit
fi
d=$HOME/scripts/move
ips=$d/ips.list
if [ -f $d/hosts.new ]; then
rm $d/hosts.new
fi
egrep -v "^#|^$|localhost" /etc/hosts | while read ipaddr hostnmtmp remainingline
do
hostnm=$(echo $hostnmtmp | cut -f 1 ".")
if [[ $(grep -c -i " $hostnm$" $ips) != "0" ]]
then
echo "$(grep " $hostnm$" $ips | awk '{print $1}') $hostnm $remainingline" >> $d/hosts.new
else
echo "$ipaddr $hostnm $remainingline" >> $d/hosts.new
fi
done
#cp /etc/hosts /etc/hosts.$(date +%Y%m%d%H%M)
#cp $d/hosts.new /etc/hosts

Check this.
I think you did not put the space in grep " $hostnm$" [ Note the space before $hostnm and the double quotes - which I suggested for exactly the same reason which you are asking for ], Also I have changed a few statements and added the hostnmtmp to get the first part of the fully qualified hostname as you desired.

Please try and let me know.

Regards,
Ninad

[Coolmar]

Nadid,

Even with the space " $hostnm$", it still picks up "host" and "host-swc".

[Ninad_1]

It shouldnt - I have checked myself as well.
Also there was a d missing in my cut line
I have attached the script in notepad - its corrected with the -d option in cut. I haved checked with sample files with data you have provided and here are the results

cat 2.dat
20.20.20.1 host.domain.com host
20.20.20.2 anotherhost
20.20.20.3 host-swc

cat 3.dat
10.l0.10.1 host
10.10.10.2 host-swc

cat hosts.new
10.l0.10.1 host.domain.com host
20.20.20.2 anotherhost
10.10.10.2 host-swc

2.dat = /etc/hosts
3.dat = the flatfile

Please use the script attached with this post.

Regards,
Ninad

[Coolmar]

Got busy yesterday...I will check more into it today. Thanks Ninad.

[Coolmar]

Worked! Thanks Ninad!!! and everyone else for the help.

S.