We had googlebot killing our website. Performance went through the floor. Google claims to respect robots.txt in the html root dir. The content of our robots.txt is
It seems to work, though you may need to kill pre-existing bot sessions from Google.
Additional info on GoogleBot for Linux bases systems.
We used 'top' to determine greediest processes, the googlebot was steady at 90+percent. We then ran ss -p | grep 'PID' to get the ip address which we then looked up with whatsmyip.org.
Ooops, the bots seem to have come back. We blocked google inbound at the firewall (126.96.36.199/19) to our moodle server.