We had googlebot killing our website. Performance went through the floor. Google claims to respect robots.txt in the html root dir. The content of our robots.txt is
User-agent: *
Disallow: /
User-agent: Googlebot
Disallow: /
It seems to work, though you may need to kill pre-existing bot sessions from Google.
Additional info on GoogleBot for Linux bases systems.
We used 'top' to determine greediest processes, the googlebot was steady at 90+percent. We then ran ss -p | grep 'PID' to get the ip address which we then looked up with whatsmyip.org.
Ooops, the bots seem to have come back. We blocked google inbound at the firewall (66.249.64.0/19) to our moodle server.
Chris