large /etc/hosts file : performance issues?

[Robert Gamble]

Obviously maintaining a large /etc/hosts file is cumbersome, and DNS/NIS is a better way to go. However, I find this one system in a situation that requires it.
From an HP OS internals class I took years ago, I thought I remember being told that a host file over 10,000 lines would begin to affect performance. I am trying to gather advice and best practice methodologies to get the application architect to use DNS instead.

I would appreciate any links or advice you can share regarding large hosts files.

Thanks!

[Peter Godron]

Robert,
according to man the /etc/hosts should normally be used to store "a few addresses of machines on the local network".
gethostent, used for "searching" /etc/hosts has to read the file sequentially to find your entry, so the bigger the file ...

However, speed should make this a non-impacting event. Do you experience any problems with your /etc/hosts file, besides maintenenace issues?

[Robert Gamble]

Current size is about 3400 lines, we havent implemented the 14000 line file yet

[Chan 007]

Robert,

Please ensure that all 10000 lines are valid, it may be old and many of your hosts/ips would have been changed. Type to write a script with ping to find the unwanted hosts that are not pingable.

What difference it makes from application point it uses a hosts or DNS/NIS, I don't find any valid reason why you are not migrating to DNS. Entire world is using DNS for its internet/Intranet, why not one system?

Chan

[Jaime Bolanos Rojas. · Accepted solution]

Robert, I have done an extensive search and nothing related to performance and too many entries in hosts file.

I found things related to having incorrect information in the /etc/hosts, some security threads related to adding internet entries in the /etc/hosts...

But the /etc/hosts should work fine if entries are clean.

The best thing is to move to DNS having that amound of entries, it takes 3 months studying the material and implementing in test server, but after it administration is going to be so much easier.

Regards,

Jaime.

[Pete Randall]

Robert,

I went back through all my course materials from SysAdmin, to Troubleshooting, to Performance and Tuning, to Internals and could find no reference to large hosts files causing issues. That certainly doesn't prove that it won't however.

My personal preference is to keep just my few HP boxes in the hosts file and rely on DNS for the rest. I use to keep everything in the hosts file but things would end up getting out of sync. It's much easier to just let DNS handle it.


Pete

[A. Clay Stephenson]

I can't imagine a situation in which I would ever use a 10,000 entry hosts file. From a performance perspective, it's probably not a big deal because I would expect the buffer cache to be your friend if the hosts file is accessed often. On average n/2 or 5000 accesses would be required to find a given entry but that could be improved by placing the most common hosts early in the file --- but doing that already makes maintaining this file more difficult.

The most intriguing thing that you said was "I find this one system that requires it.". Why? All of the gethostbyname(), gethostent(), ... functions understand nsswitch and wouldn't care if the hosts are stored in /etc/hosts, DNS, or NIS(+) so it sounds as though you have an application that uses a "homemade" host lookup function -- and that is just plain dumb. If this is the case then you need to have a serious talk with the developer.

[Hein van den Heuvel]

Agree with the prior comments.

But if for now you have to live with large host file, and you suspect it has a performance impact then I would recommend to used 2 hosts files.
One master in which you make changes and maintain comments. The actual /etc/hosts would be the other, a slave, wich is a stripped down copy. All comments removed, whitespace collapsed to single space.

To measure performance impact, maybe you can do a network taks (ping?) 1000 times over, once with hard coded address, once by name.
One would expect the network latency to drown out /etc/hosts effects, but maybe...

fwiw,
Hein.

[Robert Gamble]

thanks for all your input.
i cannot comment on why i have to use hosts other than what i have already said due to customer requirements.

Thanks!

[John_Hancock]

For those of you where you cannot envisige a situation where you would require a large hosts file....

I worked in a complex corporate OVO environment where over half of the hosts were isolated from the OVO server where the events were managed by multiple natted firewalls. In this environment most of the hosts were not - and could not be placed in the DNS where the the top level OVO server resided. There were 5 production management servers, and over 3,500 managed nodes, not including event only hosts. This environment was rappidly expanding.

There were several solutions but none included DNS because of the environment. In this case the best (but by no means good) solution was to use a managed hosts file.

I have to say that we did have mnay problems but no serious issues with the hosts files per se.