Heavy IO problem

Heavy IO problem

by Ryan-Neal Mes -
Number of replies: 8

We are currently struggling with our web server getting overloaded with too much IO. For example every Tuesday we have handins at 10am. This could lead to 500 users uploading their assignments at once. 

Some background on our environment:

  • We have DRBD setup 
  • Our webserver syncs to a failover box web server
  • We have a seperate database server that replicates to another failover box

Some of the problems we ran into are automated backups. There would be so many writes for the backup files and these would also get sync'd across to our failover box. This would pretty much bring the site to a halt. We have mitigated this but pushing all backups to a folder that is not included in our DRBD solution, but the problem still persists with general influx up user uploads. We are also using caching and a couple of other tricks, but it is starting to feel to hacky.

Anyway I see quite a few posts with people saying moodle can go up to 10000 users and more. 

How do people manage this? What if 1000 people all uploaded an assignment that is 3 mb? This has been a huge pain for us and we would appreciate any input.

Average of ratings: -
In reply to Ryan-Neal Mes

Re: Heavy IO problem

by Daniel Neis Araujo -
Picture of Core developers Picture of Plugin developers Picture of Translators

Hello,


if you want to scale, or sometimes even run, you must not use moodle's automated backup process.

you must find another, more realistic, way of backing up things that you need.

here at universidade federal de santa catarina, brazil, we have 40+ users in various "installations" and our team are seeing nice results with filesystem snapshots for point-in-time recoveries.


kind regards,

Daniel


Average of ratings: Useful (1)
In reply to Daniel Neis Araujo

Re: Moodle in English: Re: Heavy IO problem

by Ryan-Neal Mes -
We are already doing this. And it works great. Further, We have manage to solve the heavy write problems caused by drbd by mounting a shared space and sym Linking folders in the noodledata/temp. This significantly reduced IO. We are currently running 500-1000+ users an have 50courses in excess of 1 gig each and automated backups are working like a charm(note this is only after using the mounted shared space).

My question was more around how does moodle scale. 40 users is not really the numbers I am looking at. And just to be a bit more specific with my question how can I cope with 1000 or 10000 users or more all uploading a file at the same time. The hard drive speed is a huge bottleneck for us at the moment.

Thanks for the suggestion regarding the snapshots. Just another note on the automated backups - getting this to work for us was a huge win as it allowed us to backup courses at then end of a course and automatically push them to rackspace (custom feature). If we ever need a presentation of a course for what ever reason users could just pull down a copy from rackspace onto our testing environment and restore it. Really helps for investigation or partner compliance. 

Other suggestions welcome!

Average of ratings: -
In reply to Ryan-Neal Mes

Re: Moodle in English: Re: Heavy IO problem

by Jordi Pujol-Ahulló -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

Hi!

We use a shared unit for all data (wwwroot + moodledata) and a database apart in another dedicated host.

We are having a range from 100 to 500 users online and it will always works ok. No problem at all.

I liked to hear about those good news about automated backups, because I'm looking for that being applied in my University, just with the same goal.

Thanks! I hope it helps!

Jordi

Average of ratings: -
In reply to Jordi Pujol-Ahulló

Re: Moodle in English: Re: Heavy IO problem

by Ryan-Neal Mes -

Yeah I find moodle handles many users really well. But when they all start uploading documents at the same time it starts to struggle a little. Or rather the hardware starts to struggle.

Average of ratings: -
In reply to Ryan-Neal Mes

Re: Moodle in English: Re: Heavy IO problem

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
We no idea of your hardware nor the network infrastructure!
Attachment hardware_and_performance_readme.png
Average of ratings: -
In reply to Visvanath Ratnaweera

Re: Moodle in English: Re: Heavy IO problem

by Ryan-Neal Mes -

I feel like I have given ample information and remained within the code of conduct. If you need more please feel free to ask for specifics. I am not a hardware engineer so my knowledge is limited in this area.

Your reply is not constructive and I feel hurts people in the community who are only trying to help or share knowledge. Exclamation mark is really unnecessary ... 

Average of ratings: Useful (1)
In reply to Ryan-Neal Mes

Re: Heavy IO problem

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

You are not doing yourself any favours by setting all the deadlines for 10:00am on a Tuesday.

If were possible to change, set the deadline for different courses or assignments to different days. Also, set the cut-off time to an unsociable hour, like midnight, so many people actually submit earlier than that. That will spread out the big peak in the load.

Of course, that may not be a possible change to make in your institution.

Average of ratings: Useful (1)
In reply to Tim Hunt

Re: Heavy IO problem

by Ryan-Neal Mes -

Thanks for the suggestion and we are trying to spread out the load as best possible. This example is more to highlight a bottleneck and get answers to remedy the problem with scaling and IO. There are quite a few ways to overcome them, but if you "scale" this problem would be a problem at any time of the day.


Average of ratings: -