My site has been delisted for almost 3 weeks now, and below is a photo of my stats. It is within the requirements of having an active moodle site, and the listing of my site helped my search engine results by quite a bit.
Does anybody know how to get a site back onto the stats.moodle.org list again?
Wait, so by my sheer trash luck, every single time the link checker tries to access the site, it gets a 403?
Could this be cloudflare getting in the way of the linkchecker?
I... don't know what's going on.
(Edited by Mary Cooch to change subject title - original submission Friday, 4 December 2020, 7:30 PM)
Hello David. Sorry to hear about your troubles to get the site registered. Let's work together to sort it out.
Our site checker is a simple HTTP client that perform some standard requests to your Moodle server and attempts to make sure that it is a valid Moodle installation.
Your site https://usacotutor.com/lms was last successfully checked on 2020-11-01 16:30:27 UTC. After that, whenever it was re-checked, your server responded with 403 Forbidden HTTP status and did not allow our crawler to evaluate the site. it happened four times since then on 2020-11-08 11:00:35 UTC, 2020-11-15 05:30:29 UTC, 2020-11-22 00:00:22 UTC and 2020-11-28 18:30:12 UTC.
Our checker is hosted at AWS. Chances are that you have a protection deployed that blocks access to your site for clients like this?
Dear Smart David
Haha. I'm not smart. I'm just a janitor who happens to have access to the right databases.
shouldn't the scraper already have crawled in december?
It re-checks sites every 7 days. But if the site is unreachable, it only repeats the check for the first few attempts. After the 4-th unsuccessful check (such as in your case), the delay period is prolonged to 90 days. So it would re-check your site again but after 3 months.
However, I've reset the unreachable counter for your site so it should try it again soon. I can't tell you any particular time, just that your site is back in the queue of sites to be checked.
Our script checked your site at 17:30 UTC but it still got 403 Forbidden response
Maybe you can configure your site and add an exception for our checker?
It sets the User-Agent to the value MoodleBot-LinkChecker (+https://docs.moodle.org/en/Usage) and it will have one of AWS IPs
Dear Smart David,Hmm. I don't think it will be caused solely by the User-Agent string value. When I set the user agent to the value that our linkchecker uses, I am still able to access your site:
But then, if you are to reproduce the same 403 error with some other values, is a clear sign that there is some protection in place. Maybe it is more sophisticated or so.
Are you able to ask your hosting provider for support or info? Do you have access to your server error logs? All these 403 incidents could be logged somewhere and the log might contain some hint of what/where did block the request.
Yup, MoodleBot-Link-Checker user agent string is allowed to access it. But the semrush doesn't so there definitely is some kind of protection.
I do have access to the server error logs. They're quite useless.
2020/12/09 13:04:13 [error] 4575#0: *4724 peer closed connection in SSL handshake (104: Connection reset by peer) while SSL handshaking to upstream, client: 173.245.54.83, server: usacotutor.com, request: "HEAD /wordpress HTTP/1.1", upstream: "https://74.208.250.32:7081/wordpress", host: "usacotutor.com"
That is one I found yesterday. They aren't really special, or maybe I can't understand anything.
My hosting provider is my friend. He owns the server that my site runs on, so I asked him. He doesn't know how to whitelist specific user agents on Plesk, probably because it doesn't exist. IP whitelisting also, he didn't know.
I have given up on how to make my site moodle-scrapeable.
I have given up
I haven't Yesterday in the evening at 10:30pm (UTC), your site was successfully checked by our linkchecker and is now registered as available \o/
Seriously though, I did not change anything. I only reset the unreachable counter again to re-check your site. Something somewhere decided that the linkchecker was OK to go now.
Glad to see this sorted out. Good luck with administering the site. Take care.
