Managing scheduled tasks & php load on server

Re: Managing scheduled tasks & php load on server

by Howard Miller -
Number of replies: 11
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
What monitoring have you got on your servers? Do you have any idea what is going on when the server fails? High database connections? High system load and so on?

If the answer is "nothing" then "munin" is for free (you don't say what operating system you are running) and simple to set up
Average of ratings: -
In reply to Howard Miller

Re: Managing scheduled tasks & php load on server

by Eric Phetteplace -
Thanks for the response. We have zabbix monitoring internally and Uptime Robot for external pings, but neither really shows what's happening when a crash goes on. I'll look into munin. We're on Ubuntu server.

I've been on the servers while crashes are happening though and it's PHP processes running the cron script that are out of control. They monopolize CPU so badly that eventually Apache processes are blocked and the website become nonfunctional. Previously, I also noticed that ghostscript (gs) was consuming a ton of CPU too but I simply disabled that (deleted the "pathtogs" system setting and the annotate feedback feature, I forget what exactly the setting is) so that shouldn't be a problem going forward. But obviously we'd prefer to be able to use these features, it's just bad that they run out of control & crash the website.
Average of ratings: -
In reply to Eric Phetteplace

Re: Managing scheduled tasks & php load on server

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
Do your teachers and students communicate through "annotated" PDFs or other Office documents, for example in Assignment activities? The internal conversions are known performance killers. The cron tackles them in batches and can get stuck.

Average of ratings: -
In reply to Visvanath Ratnaweera

Re: Managing scheduled tasks & php load on server

by Eric Phetteplace -
Yes, they do, and as I said I disabled those features so it shouldn't be the problem anymore. But it seems like we're still seeing slowness even without the ghostscript processes.
Average of ratings: -
In reply to Eric Phetteplace

Re: Managing scheduled tasks & php load on server

by Ken Task -
Picture of Particularly helpful Moodlers

Suggest sharing info about your entire setup as an .edu is normally larger than the 'average bear' and does require some different approaches.   Example: load balanced setup with 5 nodes ... node 1 which is not used by students/teachers on a daily basis but is used as staging for updates/upgrades and is used to run cron (or specific task) as well as automated backups.

My 2 cents.

'SoS', Ken

Average of ratings: Useful (1)
In reply to Ken Task

Re: Managing scheduled tasks & php load on server

by Eric Phetteplace -
We have just one production server that does everything. It's possible to run cron on a separate server? I kind of doubt we can move to a clustered setup during the semester but this gives us something to aim for at least. We don't run automated backups with Moodle at all. We have snapshots of the server, data directory, and database made outside of the software.
Average of ratings: -
In reply to Eric Phetteplace

Re: Managing scheduled tasks & php load on server

by Ken Task -
Picture of Particularly helpful Moodlers

All in one server ... balancing act between web service and DB server. Your present issue cron and ad-hoc task/task

One hint: Skipping processing of scheduled tasks. Concurrency limit reached.

Question you asked ...

concurrency limits (we use all the defaults on this page) would make things worse. Would decreasing the concurrency limits to 2 perhaps improve things?

Would think opposite ... increasing rather than decreasing ... but that then goes back what that setting is now and .... and ... to server specs.

Suggestions by those responding are good ... and probably should be done.

You've already disabled some 'heavy hitters' (processing) ... I take it your teachers have found an acceptable alternative to each or both?


Prediction: when your sites policy of access to courses by students for 2 years after completing the course and you decide it's time for a clean up of courses (deletions)
you will be visiting the issue of cron/adhoc task/task list again ... if still using current setup ... errrr ... am reminded of a saying ... 'can't get blood out of a turnip'. smile

Symptom ...

Course cannot be deleted ... shows 'deletion in progress' ... doesn't ever seem to complete and is jamming your server up.  Cause ... a module in a course requires some heavy processing ...  solution: command line running of a cli script with 'nohup'.

By then, another question will popup ... recyclebin ... but that's another future posting.

In all in one server, am venturing an educated guess (given info so far) it's underpowered.

Since you are preparing to begin another semester, think I'd try to get things resolved ... even if it means breaking apart services ... web on one and dedicated DB server - that's easily doable and not as complicated as load balanced etc.

But ... that's my 2 cents!

Best of luck!

'SoS', Ken

Average of ratings: -
In reply to Ken Task

Re: Managing scheduled tasks & php load on server

by Andrew Lyons -
Picture of Core developers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers Picture of Testers
Hi Ken,

> One hint: Skipping processing of scheduled tasks. Concurrency limit reached.

This does not mean anything on it's own. This is a feature I built in to prevent fork-bombing the box. Sometimes you will see this and it's quite natural to do so because some tasks take longer than others.

On its own it is _not_ a sign of a problem.

> Would think opposite ... increasing rather than decreasing ... but that then goes back what that setting is now and .... and ... to server specs.

It really does depend on the nature of the load, and the location and cause of the bottleneck. It's a balance. If you increase the concurrency you make more things happen at once, which puts the server under more load. If you decrease the concurrency things take longer to run and you start to run cron during business hours when real users are doing things and you negatively affect performance that way.

> In all in one server, am venturing an educated guess (given info so far) it's underpowered.

That's pure conjecture.You just don't know. Prematurely splitting the infrastructure can have a detremental effect on performance too - it depends on the way in which it is split, the quality of the network, latency, etc. We need to _stop_ giving the advice to blindly separate things out into multiple servers before the problem is understood. Having a DB server on the same node can be extremely fast because sockets are utilised (if the server has appropriate capacity), whereas a poor quality network with high latency could lead to a performance degredation. It's all part of a bigger picture.

Andrew
Average of ratings: Useful (2)
In reply to Andrew Lyons

Re: Managing scheduled tasks & php load on server

by Ken Task -
Picture of Particularly helpful Moodlers

"pure conjecture" ... it sure is!  No doubt about it.   And, even though you've given a more in-depth technical explanation, you're really saying the same thing.

"We need to _stop_ giving the advice to blindly" - wasn't given 'blindly' ... haven't seen any specs yet, have you?  More an attempt to encourage sharing that info .... in the 'hardware and performance' forum one would expect to see such info.

Again ... my 2 cents!

'SoS', Ken


Average of ratings: -
In reply to Eric Phetteplace

Re: Managing scheduled tasks & php load on server

by Andrew Lyons -
Picture of Core developers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers Picture of Testers
Hi Eric,

> It's possible to run cron on a separate server?

Yes, it is possible and is often beneficial where you have a load characteristic which has heavy reliance upon cron. For example if you make heavy use of automated backups, PDF annotation, or have extremely active forums.

> We don't run automated backups with Moodle at all. We have snapshots of the server, data directory, and database made outside of the software.

Just as a hunch (and historical experiences), what time do your snapshotting procedures occur, and how do they run?

I've seen times in the past where system processes like backups, snapshots, scheduled software updates, virus scans, and even log rotation (because of zipping logs as they rotate) can cause massive load spikes which affect the application. Some things like database backups can even lock tables during the backup, which just about kills use of the application.

As your usage increases (Thanks C-word) you're more likely to see some of these things because there's more to backup or snapshot, the database is larger and is suddenly kicked out of memory by other processes, there are more logs so gzipping them takes longer, etc.

Just something to consider...

Andrew
Average of ratings: Useful (2)
In reply to Andrew Lyons

Re: Managing scheduled tasks & php load on server

by Eric Phetteplace -
We should look into running cron on a separate server but I don't think any of the issues brought up here (PDF annotation, automated backups) are the real problem because I've disabled them all and yet issues continue. We've never used automated backups. The crashes don't happen during the times when server snapshots are running. It definitely has to do with the cron script running out of control and not those. We do encourage a lot of forums in courses but I don't know if they'd be considered heavy use or not.

My newest theory is the one major change we made before this semester — we switched disk drives because we needed more space. Unfortunately, it seems like the new drive might be slower, thus all these problems. Perhaps the problem surfaces during cron because of various bulk file operations (cleaning up old files, the tasks that I've disabled like PDF annotation).
Average of ratings: -
In reply to Eric Phetteplace

Re: Managing scheduled tasks & php load on server

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
Running cron on a separate server effectively means that you are going down the road of a 'clustered' Moodle setup. The only difference being that the cron front-end is not publicly accessible.

This comes at the cost of a lot of extra complication and a bunch of other things that can go wrong.

However, sometimes you find yourself out of ideas and just have to do something.
Average of ratings: Useful (1)