Recently I saw a presentation of Moodle server performance for quizes in use by the OLPC project, and had the opportunity to discuss similar issues occurring at the 3rd largest Moodle installation.
It seems that simultaneous quizzes cause memory bus bandwidth issues far before memory is exhausted on modern servers. Often 2+ GHz machines will have only 150 MHz memory buses. When large modules such as are apparently used for quizzes get loaded into memory multiple times, apparently they do not share a common set of virtual memory pages.
I know from my experience with Perl that mod_perl and similar concurrency-enhancing architectures are used to enhance modules' virtual memory deduplication, so I searched mod_php and found this: http://www.majordojo.com/2007/11/is-mod-php-falling-out-of-favor-with-hosting-providers.php
Does anyone know what MPM is, and whether it can be used to reduce module memory duplication in concurrent PHP programs? Is there a PHP forum where this question is more appropriate? I couldn't find much about mod_php and memory on php.net.
php quiz module deduplication for lower memory bandwidth
Number of replies: 11Re: php quiz module deduplication for lower memory bandwidth
Aha! http://www.php.net/manual/en/faq.installation.php#faq.installation.apache2 suggests the use of FastCGI for this work.
My rusty OOP pattern training is telling me to ask: Can we move behaviors (methods and data structures) of instances to class methods using a database table instead? However, I am not sure whether the PHP object model supports class methods as a lower memory duplication alternative to instance methods.
In any case, there is obviously a huge room for improvement here. The only question is how to do it while imposing minimal concurrency constraints on Tim's ability to add new features. I propose that the current Quiz code be frozen, branched, and assigned to another team to efficiently parallelize with the goal of reducing the memory bus bandwidth requirements for large numbers of simultaneous quizzes.
My rusty OOP pattern training is telling me to ask: Can we move behaviors (methods and data structures) of instances to class methods using a database table instead? However, I am not sure whether the PHP object model supports class methods as a lower memory duplication alternative to instance methods.
In any case, there is obviously a huge room for improvement here. The only question is how to do it while imposing minimal concurrency constraints on Tim's ability to add new features. I propose that the current Quiz code be frozen, branched, and assigned to another team to efficiently parallelize with the goal of reducing the memory bus bandwidth requirements for large numbers of simultaneous quizzes.
Re: php quiz module deduplication for lower memory bandwidth
The question is, what is using that memory bandwidth? If you can run typical Moodle reports, why can't you run quizzes, which use much less data?
Can we assume that you are already running a PHP accelerator, and are already following all the advice on Performance?
Can we assume that you are already running a PHP accelerator, and are already following all the advice on Performance?
Re: php quiz module deduplication for lower memory bandwidth
Is the presentation you mention in the first line online anywhere? I would love to see it.
Re: php quiz module deduplication for lower memory bandwidth
Hi Tim and James,
The "presentation" is a preliminary run of jmeter against a OLPC XS build (XS 0.6) on six different pieces of hardware. This is part of a Masters thesis project. We'll post the results someplace online soon.
cheers,
Sameer
The "presentation" is a preliminary run of jmeter against a OLPC XS build (XS 0.6) on six different pieces of hardware. This is part of a Masters thesis project. We'll post the results someplace online soon.
cheers,
Sameer
Re: php quiz module deduplication for lower memory bandwidth
Oh right, so the "memory bus bandwidth issues" is a guess based on limited observations, not something that was directly measured.
Please to post online, and ping us here when you do. Thanks.
Please to post online, and ping us here when you do. Thanks.
Re: php quiz module deduplication for lower memory bandwidth
Have you measured memory bus bandwidth scaling behavior for multiple simultanious quizzes, Tim? What other possibilities are there?
Re: php quiz module deduplication for lower memory bandwidth
No. I have no idea to measure low-level stuff like this on a server when PHP is running. More to the point, Moodle runs on so many different OS / PHP version / database + PHP driver / hardware combinations that I am not sure that such low-level measurements are very helpful. You might try to optimise for one plaftorm, and screw things up on another.
Genearlly, when developing, the measures I keep in mind are:
- Number of DB queries required to process/display a script. This number should be as low as possible.
- Likely peak memory usage.
- Complexity/performance of any complex queries.
Those are higly corrolated with Moodle performance and scalability.
Moodle will show you 1. and 2. in the footer if you turn on the appropriate debugging option. 3. can be considered on a case-by-case basis.
Moodle does too many DB queries when handling quizzes. This is a known problem, and a fundamental design issue with prats of the quiz and question bank code. I have managed to solve half those issues in Development:Question_Engine_2 which should go into Moodle 2.1.
Solving the issue of the number of database queries it takes to load questions from the question bank (it has to do one or more queries per question, see the get_question_options function) will have to wait for later. I have some ideas about that, but they involve fairly radical changes to the DB structure.
Re: php quiz module deduplication for lower memory bandwidth
Whatever's wrong with quiz performance, I'm sure it's not - hang on - not so much 'whatever' here, we all know what's wrong with quizzes, don't we? It's the same as what's wrong with performance in all the rest of Moodle: they make lots of database queries on every page.
The only difference with quizzes is that (a) they are probably a bit heavier on the db queries than other interactive pages, and (b) many sites have a natural tendency to get all their users to run them at the exact same time.
I just did a quick test: strpos on 256MB = 0.1s on my rather slow Core2 desktop. While that's not an insignificant result, code footprint is lots smaller than 256MB, so even when you take into account that it has to both read (like my strpos) and write the memory if it's loading the code from whatever accelerator cache, it's probably not a big saving.
Bearing in mind that code size is roughly constant for all requests, if code memory performance was the main problem, you would see similar performance problems to quiz if you made a script that does nothing except require config.php (and quiz lib.php if you like) and hit that with the same frequency as quiz pages. I doubt that'd be the case; I think the additional database queries required to actually do something would make the difference, not the code memory that's occupied in all cases.
Incidentally if memory performance were the only problem with quiz performance then it would be very easy to solve: just use more than one front-end php server. It's easy to run as many front-end servers as you like. Not so easy to run multiple database servers though... (Although there are solutions. I see postgres 9 finally has clustering support so that read-only queries can go to slave servers, for instance. Think you can also do that with mysql, provided the magic pixies are co-operating.)
--sam
Re: php quiz module deduplication for lower memory bandwidth
I'm a big fan of one question per HTML page, because I like adaptive testing.
Certainly reducing database queries per page would reduce memory bus bandwidth, context changes back and forth between the database and PHP processes, and I/O bandwidth.
Can we read a whole quiz into RAM from the database and then issue its questions on subsequent pages without doing the same database query over again? I suppose the trade-off there is not having up-to-the-milisecond questions, but up-to-the-hour questions should be indistinguishable in most cases.
Re: php quiz module deduplication for lower memory bandwidth
Do you know how PHP works?
Re: php quiz module deduplication for lower memory bandwidth
Re RAM cache - Moodle is written in PHP, which unfortunately means that caching information in RAM between requests is next-to-impossible; there are some horrible hacks (hello, memcache) which can enable it, but these generally require extra software installation and maintenance.
In many parts of Moodle, possibly in quiz too, data is cached in user sessions to save repeating complex queries. Normally these are not cached in RAM exactly but on disk (still fast, probably ends up in RAM through disk caches) or database (slow but at least it's one single simple query, albeit with large data). Theoretically you can also put them in memcache if you like.
Really somebody would need to profile quiz to find out exactly which functions/queries are taking longest to process a typical request, I don't know if Tim has done that with his new question engine. If we have a profile, it might be possible to work out whether specific things could be cached to improve performance, but great care would need to be taken not to reduce reliability; after all you can usually solve performance problems as simply as getting a new server, but reliability problems (in a tool used for formal assessment) are much more critical.
Just my opinion anyhow.
--sam