I'm trying to track down an issue with a server that is currently getting hammered with requests to a Moodle site (huge number of requests constantly pouring in). The website has been unstable for about the last 24 hours or so. Tailing the moodle site's access logs, I'm seeing OUTBOUND requests, which is quite odd. Here's 3 examples of what is showing up in the access_log:
18.104.22.168 - - [15/May/2019:20:15:53 -0400] "CONNECT heat.qq.com:443 HTTP/1.1" 200 - "-" "-"
22.214.171.124 - - [15/May/2019:20:16:13 -0400] "CONNECT www.amazon.de:443 HTTP/1.1" 200 - "-" "Opera/9.80 (X11; U; Linux x86_64) Presto/2.9.181 Version/46.0.2597.32"
126.96.36.199 - - [15/May/2019:20:16:16 -0400] "CONNECT api.glovoapp.com:443 HTTP/1.0" 200 - "-" "-"
We have consistently over 100 tcp connections (with sometimes 300+), most of which are Apache connections. Running netstat -anp | grep 'tcp\|udp' on the server, I see a huge number of outbound requests for port 80. Running a packet capture on this server, I'm seeing a huge number of outbound DNS queries as well.
So to recap:
- Moodle is getting bombarded with requests such as the 3 example lines above
- Moodle is getting those requests, and turning around and making outbound requests to that destination, which is using up tcp sockets
- The server is forced to make an insane number of DNS lookups as well