We've been having some quite significant load problems with certain users making massive numbers of calls the the API function core_message_get_message from the app. We're talking over 14,000 calls from a single user in one 20 minute period.
I'm at a bit of a loss. Is this a bug or are we being hacked in some way?
We've found multiple instances of this. The same web service function being hammered - several a second by different users in different places. It's not just the messaging function but that's the usual one. Here's a short example, but even for this one there are pages and pages like this...
There's a userid in that table ... all the same in your clip. Find out what devices that account has used to access. Could be some calendar app (not in the Moodle app) but Googles? that's trying to sync?
Had something like that on a school Moodle and finally tracked down to a single user using multiple devices (iPhone, IPad, laptop) and attempting to sync Google Calendar to Moodles.
No new events are showing up in the calendar are there?
Guess.
'spirit of sharing', Ken
Ok... good point. I'll check the apache logs to see what it says it is. I assumed it was the app but, you're right, it's possible it isn't.
In this case they are calling the web service to get messages so I doubt it's the calendar but it could be something else. Nobody has every been very interested in the calendar around here so I'd be surprised if it was anything to do with that anyway.
Oddly, these are nearly always students in halls of residence (does that translate outside of the UK?). It's not always the same student either. However, that's rather my point. We're finding single users absolutely hammering the web service api for periods of time with exactly the same call.
'Halls of Residence' does translate across 'pond' - dorms.
So residents are what ... 19-22 years of 'experience'? Of generation Z? Am sure some computer sci/IT types and students with 'an interest in tech' (Wonder if any supplement their student income with electronic copies of whatever? Knew a fellow student who paid his semester fees by 'dumpster diving' at the end of every semester ... had old semester exams and finals from a variety of departments).
Web services to me has always been stealthy ... and potentially a hole if not setup properly with a user just for web services use, a token, and access controls to web service functions. Am not claiming to be an expert at mobile ... far from it ... but I'm willing to bet those 'residents' are far ahead of me. Since Moodle is open source ... anyone can inspect and possibly attempt to build their own app for their personally rooted smart phone ... could they not?
Not saying web services is in-secure either ... might be something of annoyance even though it's hammering away at the server that traffic isn't dragging performance down that much, is it?
Anyhoo! Good luck tracking back and figuring out what's up!
'spirit of sharing;, Ken
Don't think it didn't occur to me!
However, I did a few background checks and so far all "arts" subject students. This doesn't exclude them from being mad, KBG-backed hackers but I think it makes it less likely. I'm not fond of conspiracy theories. These things tend to have a more mundane explanation. I just haven't found it yet...
Hi Howard,
that WebService is used in the following cases:
So it could happen that, if the process to check unread notifications & messages are executed at the same time as viewing the list of discussions and viewing a discussion, you could have like 10 requests in a row, but not 30! Of course, the user could have the same account in different devices, but it could be strange that all requests happen at the same time (unless he has a lot of devices, but that's unlikely).
You should check the parameters of the requests, maybe they give you some clues on what is going on. IMO the most important ones are:
In Moodle 3.6, messaging is being refactored so hopefully we'll be able to reduce the number of WebService calls.
Cheers,
Dani
Hi Dani Palou,
We found out it take times to send message out with the mobile app, and most of the time the message can't send out successfully and it will keep retrying, hence duplicate message and email sent out.
I have tested by sending message to 6 users, 5 of them got this problem and 1 user no problem.
This is what i mean, the message sent so many times and if fail to send, it will have a clocking icon next to the message.
After i go out from the message sending page, and click 'Danny' again from the Message list, it prompt me this error :
If send to this user, i don't face any problem.
I'm not sure if the issue is because of the Moodle Mobile App or the Moodle Setup in our University. Appreciate if you can enlighten me. In my university, we have 6 apps servers and 3 databases. Each two apps servers connect to 1 database server
Looking anyone can reply on my post.
Thank you
Regards,
Mei Ling
Hi Mei Ling,
in the first screenshot there are a lot of messages with the text "1". Did you manually write all of them or it was written only once and then it was automatically sent several times by the app?
From what you're saying, my guess is that your server is too slow to handle the requests, so they time out. The app has a 30 seconds timeout, so if your server doesn't answer in that time then the request is cancelled.
I recommend you to follow this guide to debug the WS calls:
https://docs.moodle.org/dev/Moodle_Mobile_debugging_WS_requests
With the Chrome DevTools you should be able to see which network requests are done to the server when using messaging, and how long does the server take to answer.
Kind regards,
Dani
Hi Dani Palou,
It was written only once and then it was automatically sent several times by the app.
If my server is too slow to handle the requests, how come the problem will only occured when every time I sent message to student_A? So far, there is no problem face when i sent message to student_B.
After debug the code, the error I caught once is in below :
{exception: "invalid_parameter_exception", errorcode: "invalidparameter",…}
debuginfo: "messages => Invalid parameter value detected: Missing required key in single structure: touserid"
errorcode: "invalidparameter"
exception: "invalid_parameter_exception"
message: "Invalid parameter value detected"
Most of the time, there is no error, but just the message can't send out (I guess)
Here is the debug screen, you can see that there are few messages are pending to send, and the requests are keep going out.
May i know how to fix from here?
Looking forward to your reply.
Thank you.
Regards,
Mei Ling
Hi Mei Ling,
I think we should continue this discussion in this topic since the problem sending messages isn't related to the original message in this topic.
Kind regards,
Dani
