Hi J,
This advisory only refers to connections to iLO. Leaving iLO management drivers will not cause login timeouts, and removing the drivers can cause problems with using iLO.
That being said, you can uninstall the iLO drivers on a Windows machine by removing the device from Device Manager. You could also prevent iLO connections to the server by removing the cable connected to the iLO port. You can also disable iLO during boot-up by pressing F8 when prompted to configure iLO settings.
good luck, if you appreciate the answers we appreciate the points!