Advice on optimising php/db code in moodle2+

Advice on optimising php/db code in moodle2+

by Stuart Lamour -
Number of replies: 50
Picture of Plugin developers

Moodle 2.5 & 2.6 are being really slow.

With no css or javascript (i.e. all the theme stuff i optimise) it's still really slow.

With no images it's still really slow.

With no blocks it gets a little quicker, but its still really slow.

Running on localhost on a new macbookpro mamp in chrome with no data in the db, it's still really slow.

Loading a page with 53kb of html shouldn’t take over 12 seconds to load - no matter who i’m logged in as or what settings i have - devtools shows the .php file has the latency...

Anyone know what the specific features/elements added to moodle’s core since 1.9 that have made it so slow?

I appreciate all the work being done on caching and how i can optimise my setup, but what i’m really interested in is - what is it in moodle that got so slow to render a page between 1.9 and today?

What can i do to optimise the php code to get my sites performance back?

(more than happy to hack core)

any help appreciated

cheers

stuart

Average of ratings: Useful (3)
In reply to Stuart Lamour

Re: Advice on optimising php/db code in moodle2+

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

I think there's just more stuff everywhere. If you run up the performance stats on a 1.9 site and then on 2.5 you can see the amount of memory used has almost doubled. That's potentially twice as much processing to do if all of that is code. Essentially, you need a server twice as fast (and with twice as much memory, and twice as fast disk IO) just to stand still. Well - that's a grievous over simplification but you know what I mean. 

One would hope that the work being done on cacheing would help - maybe it is but, to be honest, I don't understand it at all so I can't comment. 

There is hugely more javascript based stuff now, but, that just means more time to download the vastly more complicated pages. And so on...

I think I would give the the old advice of "if you don't use it remove it". All plugins (even core ones) burn up some processing time and memory even if not in use so delete them. It might help. 

Average of ratings: Useful (1)
In reply to Howard Miller

Re: Advice on optimising php/db code in moodle2+

by Stuart Lamour -
Picture of Plugin developers

but what exactly is all that 'stuff' doing?

I can't see any changes in functionality as a tutor or students, and like i said javascript, css, images and blocks are all turned off (some in browser, some in settings). 

i'm all for the "if you don't use it, remove it" but what exactly is all that 'stuff'?

anyone running http://newrelic.com/ on a 2.5/6 ? 

In reply to Stuart Lamour

Re: Advice on optimising php/db code in moodle2+

by Ian McNaught -

Are you using APC? That makes a huge difference to performance. I'm running 2.5 in production, and don't recognise the problems you describe. 

In reply to Ian McNaught

Re: Advice on optimising php/db code in moodle2+

by Stuart Lamour -
Picture of Plugin developers

hi Ian,

i am really trying to identify the actual underlying bottlenecks - caching can lead you to just bury that all under the carpet! 

like i say - i want the underlying code to be fast, before caching.

 

In reply to Stuart Lamour

Re: Advice on optimising php/db code in moodle2+

by Toni Soto -

Very good question Stuart! I've experienced the same you're describing. I'm not an expert but I suspect that the bottleneck for most of small and medium installations is the I/O disk. I'm using APC since I moved my school Moodle to 2.4 and did some optimization on MySQL and on removing blocks from courses as it was suggested. I should say that we don't have SCSI disks so our delays on Disk Writes can cause the slowness. 

As you pointed out there is a huge difference between 1.9 and 2.5 speed. I always thought that the first bottleneck I would find managing our School Moodle will be bandwidth because we're using the asymmetric line from our provider (Cheaper for us than hosting in a datacenter) but now I'm afraid the server itself (actually the disk) could be bottleneck. Anyway and I said before I'm not a expert smile 

 

Toni Soto 

In reply to Stuart Lamour

Re: Advice on optimising php/db code in moodle2+

by Toni Soto -

Very good question Stuart! I've experienced the same you're describing. I'm not an expert but I suspect that the bottleneck for most of small and medium installations is the I/O disk. I'm using APC since I moved my school Moodle to 2.4 and did some optimization on MySQL and on removing blocks from courses as it was suggested. I should say that we don't have SCSI disks so our delays on Disk Writes can cause the slowness. 

As you pointed out there is a huge difference between 1.9 and 2.5 speed. I always thought that the first bottleneck I would find managing our School Moodle will be bandwidth because we're using the asymmetric line from our provider (Cheaper for us than hosting in a datacenter) but now I'm afraid the server itself (actually the disk) could be bottleneck. Anyway and I said before I'm not a expert smile 

 

Toni Soto 

Average of ratings: Useful (1)
In reply to Stuart Lamour

Re: Advice on optimising php/db code in moodle2+

by Gavin Henrick -
Picture of Plugin developers

Stuart,

Yes, if you performance check Moodle (info below) there is more going on in the background than Moodle 1.9 did, and so yes is a bit slower, but I think overall that is to be expected.

But there is a deeper question: Are you trying to run a car with a pretty sophisticed engine on a dirt track?

What I mean by this is that with any enterprise system you need to consider the following questions:

1. Is the hardware you are running it on the minimum specification for what you are trying to achieve?
2. Is the webserver correctly configured for running Moodle 2?
3. Is the database correctly configured and optimised for running Moodle 2?
4. have you corrected implemented a balance of resources between the database and the webserver and other applications/services running on the server if it is just one server?
5. Is the disk and i/o fast enough for what you are trying to achieve?
6. Is there enough memory in the system to handle the webserver threads and the database and so forth...?
7. Are there any applications on the system which could impact the performance that you are looking for?

There is a difference between a correctly configured and specced system and then having it optimised, which is the next part where you look at caching and so forth.

Some of these are just basic requirements for running a system like Moodle.

Then you would need to address settings such as

- do you have theme designer mode on which redoes stuff each time
- have you any customisations added which may impact performance


If you want to do some code level optimisations, you would find the following post from Tim to be very interesting and useful in your endeavours:

http://tjhunt.blogspot.ie/2013/05/performance-testing-moodle.html

If you are interested in testing the performances of 2.4 2.5 and 2.6 then the series of posts by Justin Filip are really interesting:

http://jfilip.ca/2013/09/25/moodle-performance-testing-2-4-6-vs-2-5-2-vs-2-6dev/

http://jfilip.ca/2013/08/20/moodle-2-4-5-vs-2-5-1-performance-and-muc-apc-cache-store/

If you want some info on the different databases, have a look at

http://jfilip.ca/2013/09/24/moodle-performance-analysis-revisted-now-with-mariadb/

I would be welcome too for a faster Moodle 2, but there again on most production systems where I have used it on it is quite fast due the ensuring it is a slick race track under it with sysadmins that would make Formula1 engineers jealous.

Average of ratings: Useful (3)
In reply to Gavin Henrick

Re: Advice on optimising php/db code in moodle2+

by Stuart Lamour -
Picture of Plugin developers

Hi Gavin, 

no customisations.

running on a new (out the box last week) highest spec available macbook pro via latest version of mamp on localhost, using chrome, nothing else running on the machine - i guess that makes it a ferrari running on whatever the fastest racetrack is (until october 15thish)?

results are with theme designer off - it gets even slower when its on - and nothing in the db.

thanks for the link to Timys post - i'll try some of this and let you know the results, but incrementally changing is an utter ballache with initial performance times this poor and dev mode on.

Justin's post seems to indicate that - as Howard says - there is a lot more 'stuff' going on - files included, db queries and general I/O but never how/why/where/what this extra stuff is doing/adding. Any clues?

thanks as always,

stuart

In reply to Stuart Lamour

Re: Advice on optimising php/db code in moodle2+

by Gavin Henrick -
Picture of Plugin developers

Sounds like a good laptop, but I wouldnt call it a fit for purpose server would you?

If you are using out -of-the-box/install configurations for apache/mysql etc then probably isnt using the hardware that it has available either.


So unfortunately, any tests you do would probably not be indicative of anything except the performance of Moodle on an unconfigured laptop.

You probably would be better putting it on a dedicated and fully configured server rather than a laptop if you want to get some realistic results of performance.

G

Average of ratings: Useful (3)
In reply to Gavin Henrick

Re: Advice on optimising php/db code in moodle2+

by Stuart Lamour -
Picture of Plugin developers

like i say dude, fast machine, fast browser, latest php, sql, apache etc - not much more i can do with mamp smile

has moodle really got itself into so much bloat that it isn't possible to do local development anymore?

In reply to Stuart Lamour

Re: Advice on optimising php/db code in moodle2+

by Andrew Lyons -
Picture of Core developers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers Picture of Testers

Hi Stuart,

First off, I wouldn't touch MAMP (or LAMP, XAMP, etc) with a barge pole if it were me. It's a personal preference, but hey. I run my entire Mac-based dev stack on MacOS using the stock Apache2 server, and with php5, postgres, mysql and other components out of Homebrew. I have at least six installs at any one time, plus all of the other things I'm doing (integration + stable for master, 25, and 24). Using homebrew means I can run multiple versions of php simultaneously (against different VirtualHosts in apache for example). I'm also using postgres from the petere/postgresql tap to give me the option of different versions of postgres too.

I have none of the MUC caching enabled on my stock installs, though I do use memcached out of homebrew for testing MUC. I've not tuned APC, Opcode, or xcache at present (though I should really fix that actually). I'm not actually sure any of them are even installed come to think about it (should also fix that).

As I mentioned on Twitter, Moodle has grown a lot since version 1.9. Things like course completion, grouping features, context-specific permission overrides, capabilities, hidden activities conditional activities, etc don't come free and can require a fair amount of processing either in php, or in the database. Sometimes it's not practical to remove these entirely if you're not using them because most people are. Even if not always or they don't know it.

A lot of work goes into that processing including at both the peer review, and integration review level. Those database queries are usually optimised as best as possible, but every time you join a table, add a criterion, sort, group, etc. you increase the complexity. They'll still typically be far more efficient than getting the same result with a sub-select for example.

However, they all use memory. As others have said, you can have the world's most powerful computer, but sticking an untuned MAMP in there won't make any use of that power. I don't know what the default configuration is for MySQL on MAMP, but the stock Postgres configuration only offers something like 24MB for shared memory (this isn't just laziness, but an attempt to provide the most widely usable config for all systems - as I recall, Solaris in particular has limitations around the 24MB mark which require you to change system settings and the Postgres config tries to run out of the box without requiring reconfiguration of your system). Clearly, the defaults aren't going to cut it in many cases and you need to do some performance tuning. When there isn't enough allocated memory, most database engines will page the resultset to disk while in order to free up enough memory for the next stage. Try running one of the slow queries against an EXPLAIN ANALYZE to see what it's doing - I'm almost certain that you'll come across some which are doing exactly this. Disk is slow. Even SSD is slow compared to memory.

It's not about putting lipstick on a pig, and no matter how fast a laptop you have, without tuning it it's like having a 4-lane motorway with three of it's lanes closed, and the remaining one just a bit too narrow for that HGV in front of you and full of pot holes. Plus your tyre keeps going flat so you have to get out and pump it up long enough to get to the next junction. Not tuning your database is like buying a Formula 1 car, and putting cheap petrol in it. It may run, but it's not going to be pretty. 

I've not done a huge amount of tuning on my system (just Postgres, and I've barely done anything there), but using mdk to install a new moodle instance with 100 students, 5 teachers, 5 editing teachers, 20 courses, and to enrol each user in every course takes 59 seconds on a MacBook Air with an i7 processor. In my opinion, that really isn't too shady.  The installation performs well too.

I daresay that if I configure an opcache, I could improve that further.

Best wishes,

Andrew

Average of ratings: Useful (6)
In reply to Andrew Lyons

Re: Advice on optimising php/db code in moodle2+

by Mark Nelson -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers Picture of Testers

QFT.

Side note - world's most powerful computer? You haven't seen my over-clocked beast at home. smile

In reply to Stuart Lamour

Re: Advice on optimising php/db code in moodle2+

by Petr Skoda -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
My ancient Mac (3yo) serves pages for not-logged-in users around 0.5s with any stable PHP version, PostgreSQL and Maria DB installed via http://www.macports.org using default configuration.

I would recommend to abandon the constantly outdated and problematic mamp and instead learn to use http://brew.sh or better http://www.macports.org.

Here is a cheat sheet for my dev install http://docs.moodle.org/dev/PHPUnit_installation_in_OS_X

Edit: oh, and please make sure your are not hitting the dreaded ipv6 timeout issues
Average of ratings: Useful (1)
In reply to Petr Skoda

Re: Advice on optimising php/db code in moodle2+

by Stuart Lamour -
Picture of Plugin developers

Hi Petr,

i mostly use yeoman + grunt n bower for dev work, it is only moodle http://download.moodle.org/ that has meant i'm suddenly in retro mode with mamp, but i think your missing the point of my question maybe?

as stated above, js is disabled.

The links gavin gave above seem to suggest lots more db access, file includes and general overheads for every page. If i could understand what all these overheads are for it might help me a bit.

I'd just like to know what it is exactly in the moodle2+ core php code - through any I/O & db interaction - that is making moodle2+ so bloated compared to 1.9, and do i need any of it and if not how can i turn it off?

any help gratefully appreciated,

stuart

In reply to Stuart Lamour

Re: Advice on optimising php/db code in moodle2+

by Darko Miletić -
Picture of Core developers Picture of Plugin developers

The slowest parts are blocks and admin/module global settings. You will notice that when using moodle as non admin pages render much faster. So basically the only way to do something realustic us to remove any modules you do not plan using instead of disabling them.

After that os/lamp optimizations are a must. Ping me next week on skype and we can talk some more if you wish.

Average of ratings: Useful (2)
In reply to Darko Miletić

Re: Advice on optimising php/db code in moodle2+

by Stuart Lamour -
Picture of Plugin developers

thanks darko - happily take you up on that offer - what timezone you in?

In reply to Darko Miletić

Re: Advice on optimising php/db code in moodle2+

by Stuart Lamour -
Picture of Plugin developers

p.s. you got http://newrelic.com/ or similar running on any 2.5/6 installs?

Be very interested to see what the actual code bottlenecks are before i inevitably give up on what seems a thankless code optimisation task and brush it all under the carpet with caching and server configs.

In reply to Petr Skoda

Re: Advice on optimising php/db code in moodle2+

by Stuart Lamour -
Picture of Plugin developers

p.s. - petr, we don't have any users who don't login.

have you tried load times for logged in users? - admins (in dev mode and not), tutors or students?

In reply to Stuart Lamour

Re: Advice on optimising php/db code in moodle2+

by Petr Skoda -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
Of course, it stays under 1s for admin and other users when I access it alone locally.

Next time be a lot more specific when asking performance questions, you did not tell us anything apart from "12 seconds" in your first post - so do not expect any meaningful advice...
Average of ratings: Useful (1)
In reply to Petr Skoda

Re: Advice on optimising php/db code in moodle2+

by Stuart Lamour -
Picture of Plugin developers

Dear Petr,

firstly, i would like to thank you so much for your most polite, informative and useful comments in this discussion so far.

i will consider myself told, slapped on the wrist and generally warned from posting anything here again without consulting with timy hunt or yourself as to the proper format first. 

In this particular instance though, as you are unable to answer the question "what are the specific features/elements added to moodle’s core since 1.9 that have made it so slow?" would you most kindly be able to point myself, and any other interested commenters, in the direction of someone who could perhaps answer my enquiry?

Yours Sincerely,

Stuart Lamour

 

Average of ratings: Useful (1)
In reply to Petr Skoda

Re: Advice on optimising php/db code in moodle2+

by Stuart Lamour -
Picture of Plugin developers

Dear Petr,

your current moodle message is set to :

"Petr Škoda only accepts messages from their contacts."

i'd love to seek your advice before posting anything further to the moodle forums, but this seems to say i cannot?

Are you able to help?

Yours sincerely,

Stuart Lamour

 

In reply to Stuart Lamour

Re: Advice on optimising php/db code in moodle2+

by Howard Miller -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

Remember that you are not running in an environment optimised for server software. I use a Mac for development (as well as Ubuntu) and, although I don't use MAMP, it isn't especially quick. My Ubuntu Server (bog-standard Dell box), however, is very quick. 

I suspect that in OSX, all the Desktop stuff gets priority. 

In reply to Howard Miller

Re: Advice on optimising php/db code in moodle2+

by Stuart Lamour -
Picture of Plugin developers

howard - i don't generally use mamp either (first time in 4 years or so!), but i'm following the moodle guidelines.

however, i did a quick check with mamp versions of joomla & wordpress and they are like lightning comparatively - even with css and images on, js on and caching disabled!

 

 

In reply to Stuart Lamour

Re: Advice on optimising php/db code in moodle2+

by Russell Smith -

Hi Stuart,

When it comes to performance issues its very hard to know what is causing the issues without a lot of information.  Everybody can continue to guess suggestions about fixing it, what we've love to have is some profile information about what's actually going on.

 

Someone has already suggested newrelic and Tim Hunt's performance blog.  I use Tim stuff and http://docs.moodle.org/dev/Profiling_PHP all the time to find and remove performance bottlenecks.  It would also be important to turn on the MDL_PERF and MDL_PERFTOFOOTER to gain a perspective of how long the server says it's taking to make a page.

I cannot stress enough how important real data is in these situations.  I don't know how many times performance issue I've had could not be solved without real data like that provided about.  Also as you say you want to understand the slow parts of the code, having those profilers will allow you to delve into that properly.  A good introductory performance analysis task is to determine why admin pages load a lot slower than regular user pages.  We all mostly know the answer to that but seeing the facts is interesting and also shows why changing that is not as easy as you might think.

Good luck.

Russell

Average of ratings: Useful (5)
In reply to Stuart Lamour

Re: Advice on optimising php/db code in moodle2+

by Russell Smith -

Hi Stuart,

When it comes to performance issues its very hard to know what is causing the issues without a lot of information.  Everybody can continue to guess suggestions about fixing it, what we've love to have is some profile information about what's actually going on.

 

Someone has already suggested newrelic and Tim Hunt's performance blog.  I use Tim stuff and http://docs.moodle.org/dev/Profiling_PHP all the time to find and remove performance bottlenecks.  It would also be important to turn on the MDL_PERF and MDL_PERFTOFOOTER to gain a perspective of how long the server says it's taking to make a page.

I cannot stress enough how important real data is in these situations.  I don't know how many times performance issue I've had could not be solved without real data like that provided about.  Also as you say you want to understand the slow parts of the code, having those profilers will allow you to delve into that properly.  A good introductory performance analysis task is to determine why admin pages load a lot slower than regular user pages.  We all mostly know the answer to that but seeing the facts is interesting and also shows why changing that is not as easy as you might think.

Good luck.

Russell

In reply to Stuart Lamour

Re: Advice on optimising php/db code in moodle2+

by Ian McNaught -

I'm not sure it's fair comparing Moodle to Joomla or Wordpress. You can get a basic measure of the complexity of a web app through the number of database tables. Wordpress by default uses 11 tables, Joomla around 40. Moodle requires around 200 database tables. Moodle is in a completely different class of complexity to Joomla or Wordpress, Wordpress is a sportscar, Moodle is a juggernaut - not much point comparing their 0-60 speed!

There's also little point looking into the performance of Moodle on an unconfigured system. MySQL will by default use very little memory, regardless of how much your machine has at its disposal - so there's likely a load of paging going on in your case. I can't imagine anyone would use Moodle in production without APC (which of course is a lot more than caching, it's compiling), so why bother looking at metrics without that?

To give you a comparison, I use a single AWS large instance (2 core, 7.5GB RAM) for Moodle 2.5, and I've loadtested with 150 concurrent users thrashing it through Jmeter (on the quiz module), and it performs significantly better than you report. I've done some rudimentary tuning of MySQL and APC, but I'm no expert in those areas.

Average of ratings: Useful (2)
In reply to Ian McNaught

Re: Advice on optimising php/db code in moodle2+

by Stuart Lamour -
Picture of Plugin developers

thanks again Ian,

as stated i'm good with optimising a system to run moodle, just after some help with understanding what all that php 'stuff' and db queries are about.

cheers

stuart

In reply to Ian McNaught

Re: Advice on optimising php/db code in moodle2+

by Andrew Lyons -
Picture of Core developers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers Picture of Testers

Thanks Ian, I agree entirely,

Another thing to note is that the level, depth, and breadth of the capabilities in Moodle far exceeds Joomla, Wordpress, or any other similar systems. I would imagine that this is one source of bottlenecking in Moodle.

As Ian says, they're incomparable systems.

Andrew

In reply to Stuart Lamour

Re: Advice on optimising php/db code in moodle2+

by Danny Wahl -

Hi Stuart,

I've been doing dev locally on my MBP for years now and I've always said "forget that noise" to the Moodle "guidelines" for setting up MAMP.

I use the native Apache, native PHP, and native (now) PostgreSQL databases.  I've found the switch in OS X from MySQL to Postgres to increase speed a bit.

Now, the thing that always slowed my machine to a crawl was the db and this became especially poignant on the switch from 1.9 to 2.x, so anecdotally I blame the db.  Seriously, setting up a new install I may as well push "start" and go get a cup of coffee and read a book.

I know this doesn't address your question really, but there is some validity to other people's questions.  e.g. is a 5400 RPM HDD really a valid benchmark platform for a database intensive app?  The slowdown inherited there from 1.9 is going to be amplified  in 2.x whereas a "true" server drive (iSCSI, 10,000RMP, SSD, etc...) the difference will be quite minimal.

tl;dr: it's probably the DB.  You should get a SSD because, why not?

In reply to Danny Wahl

Re: Advice on optimising php/db code in moodle2+

by Davo Smith -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

I can certainly vouch for the fact that on my home SSD a Moodle 2.5 install can complete in about a minute, on a standard hard disk it takes 5+ min. So DB would be a clear candidate for slowness.

In reply to Davo Smith

Re: Advice on optimising php/db code in moodle2+

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
Now we are comparing the installation time of Moodle I wonder whether it should become a benchmark like what the compilation time of Linux kernel used to be?
Average of ratings: Useful (2)
In reply to Stuart Lamour

Re: Advice on optimising php/db code in moodle2+

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
Hi Stuart

A polarizing question but too late! Still thanks for the lively discussion. Here are some random thoughts related the the whole discussion.

- As others have pointed out your evidence stems from just one observation. On top of that, it is related to a product which is not well received by the FOSS community!

- We can forget the evidence and ask, has Moodle got bloated between 1.9 and 2.5? There are various benchmarks in the form of blogs and forum discussions which say, Yes.

- But then, 2.5 does a whole lot more than 1.9! So the next questions are:
-- Does 2.5 consume more per feature than 1.9 did?
-- If one needs/use only 1.9 feature in 2.5, does 2.5 consume more?
-- Is Moodle 2.5 doing the same things on a X $ machine today is slower than 1.9 did on a X $ machine 5 years ago?

Honestly, I don't know. What I know is, Moodle is mainstream. Moodle wanted it that way, those new features and version run was (and still is) the result. And there is a price to pay. For fundamental questions like "Was all that necessary?" it is too late now (short of giving 1.9 another lease, I mean.)

What surprises me is that the question comes from a Moodle Partner. For me it was logical that Moodle Partners chose mainstream.
Average of ratings: Useful (1)
In reply to Stuart Lamour

Re: Advice on optimising php/db code in moodle2+

by Stuart Lamour -
Picture of Plugin developers

Thanks to everybody for their contributions here.

Andrew has answered the question :

"Things like course completion, grouping features, context-specific permission overrides, capabilities, hidden activities conditional activities, etc don't come free and can require a fair amount of processing either in php, or in the database. Sometimes it's not practical to remove these entirely if you're not using them because most people are. Even if not always or they don't know it."

Howard - i guess this is that 'stuff' that, as you say - "if you don't use it remove it".  smile

I'll open another thread when i get a chance about how to clean up any of this functionality if you don't require it.

Thanks again

Stuart

 

In reply to Stuart Lamour

Re: Advice on optimising php/db code in moodle2+

by Anne Krijger -

Hi Stuart,

It's actually Gavin and Russell who gave the correct answer(*).

But for reasons unknown to us you do not seem keen to follow their advice.
Either that or I just don't understand your responses to them.

Point being that the only way to find out why YOUR system seems slow(er) is to do perfomance testing AKA profiling.

Anne.
(*) Not to say Andrew's answer was wrong smile

Average of ratings: Useful (3)
In reply to Anne Krijger

Re: Advice on optimising php/db code in moodle2+

by Stuart Lamour -
Picture of Plugin developers

Hi Anne,

Gavin & Russell's posts were most appreciated and i marked them both as useful.

I'm pretty oldskool and like to know what the code is doing, what the big o of the code is and what code i don't need, so i really do prefer answers about code - thus my question.

p.s. who is us? 

In reply to Stuart Lamour

Re: Advice on optimising php/db code in moodle2+

by Gareth J Barnard -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Plugin developers

Hi all,

It might be time for some investigatory analysis of program flow to reduce unnecessary repeated calls to the same method to calculate something.  This thought was sparked off by https://moodle.org/mod/forum/discuss.php?d=236254#p1025923 resulting in further discussion from https://tracker.moodle.org/browse/MDL-41516?focusedCommentId=241825&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-241825 where I raised it.  In this example, the code in question was called three times.

Now I realise that certain code is better off called many times if the result means much more memory and the calculation is quick.  But if the result is 'boolean' then far better to store it instead of repeatedly calling the method.

This could be an angle of investigation.

Cheers,

Gareth

P.S. Just read Tim's blog post with fascination and wonder - definitely the way to go.

Average of ratings: Useful (1)
In reply to Stuart Lamour

Re: Advice on optimising php/db code in moodle2+

by Andrew Lyons -
Picture of Core developers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers Picture of Testers

Hi Stuart,

Whilst I may have been one of those to provide this advice, I don't necessarily believe I was the first. Gavin mentioned that there was more going on under the hood on Sunday, and others (Russell for starters) have suggested ways to find out exactly what.

Additionally, my suggestions were not a certainty as to the cause of the slowness, merely suggestions based on my experience as to what could be a factor. It may be that some of them are already off by default (I suspect that groupmembersonly is of this nature). Sometimes having to test whether a feature is off is more often slower than just running the query, or testing as part of the same query.

In all honesty I believe that your issue is almost entirely down to the fact that you have not tuned your system in any way. I don't feel that comparing Moodle with Joomla or Wordpress are either realistic, or a fair comparison. It's pointless to say that since they work on an untuned system, Moodle should too as they offer vastly different features to Moodle and have very different sizes.

What this thread does however highlight is that we perhaps need to improve our installation and tuning documentation. Far too many people have said that they wouldn't advise MAMP (including a number of us working for Moodle) so why is it the recommendation for example. That said I'd (at least personally) never recommend running Moodle on a Mac in production and I guess the expectation is that few people would.

Andrew

Average of ratings: Useful (1)
In reply to Andrew Lyons

Re: Advice on optimising php/db code in moodle2+

by Stuart Lamour -
Picture of Plugin developers

Hi Andrew

Howard said there was more 'stuff', Gavin said there was 'going on in the background', i asked about any stats from newrelic or similar (which i'd still love to see) and i really wanted those specifics of code.

i never asked for anything about system config or setup - but replies seem to be fixated on that - with not much about the actual code.

So lets just say your answer was the most useful one for me/the original question.

on a side note updating that http://download.moodle.org/macosx/ page, which is about config and setup, makes a great deal of sense - as you know i'm much happier in brew than mamp - who looks after that page? Give them a shout smile

Thanks again

Stuart

In reply to Stuart Lamour

Re: Advice on optimising php/db code in moodle2+

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
Hi Stuart

You said:
> i never asked for anything about system config or setup - but replies seem to be fixated on that - with not much about the actual code.

Going back to OP, you started it with:
>> Moodle 2.5 & 2.6 are being really slow.
>>
>> With no css or javascript (i.e. all the theme stuff i optimise) it's still really slow.
>>
>> ...

The reader is now wondering, "What is slow"? The final items in that list being:
>> Running on localhost on a new macbookpro mamp in chrome with no data in the db, it's still really slow.
>>
>> Loading a page with 53kb of html shouldn’t take over 12 seconds to load - no matter who i’m logged in as or what settings i have - devtools shows the .php file has the latency...

It is to be expected that people first want to investigate your laptop! Obviously you did not want that. Your main question was:
>> Anyone know what the specific features/elements added to moodle’s core since 1.9 that have made it so slow?

It about the increased resource consumption of Moodle 2.5/2.6 compared to 1.9. But lot of people answered that question. My summary is that, the latest generation of Moodle has so much more features ("stuff"), it is not fair to compare them against 1.9. I though that answers the next question:
>> I appreciate all the work being done on caching and how i can optimise my setup, but what i’m really interested in is - what is it in moodle that got so slow to render a page between 1.9 and today?

You need to compare the release notes, others have counted the most prominent ones.

The last question is the hardest.
>> What can i do to optimise the php code to get my sites performance back?
>>
>> (more than happy to hack core)

There is no useful answer to that. If the developers knew the bugs [*] they would have cleared them or they are in the bug tracker. If the developers haven't recognized the bugs, they can't explain them either. Putting it another way, if one has to ask, "Show me where to clean the code", he is the wrong person for the job.

Version 1.9 was not efficient from the beginning. Somewhere around the "mid-versions" it went through major polishing phase. My belief is that, it came from some senior developers. That reminds me of a major flaw in the accelerated version run we have today. It is simply too fast for a polishing phase. Compare the 2 1/2 years between 1.9(.0) to 2.0(.0) to the 6 months between 2.3(.0) and 2.4(.0). I know this is a moot topic. It is perfectly OK for me if nobody answers.

any help appreciated

cheers

stuart
Average of ratings: Useful (5)
In reply to Visvanath Ratnaweera

Re: Advice on optimising php/db code in moodle2+

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
Oops, the last three lines in the above post were quotes from the OP which I forgot to delete!
In reply to Visvanath Ratnaweera

Re: Advice on optimising php/db code in moodle2+

by Petr Skoda -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
Thanks Visvanath, I think your summary is very accurate.
In reply to Visvanath Ratnaweera

Re: Advice on optimising php/db code in moodle2+

by Stuart Lamour -
Picture of Plugin developers

HI Visvanath,

bugs isn't what i'm after at all. I'm sure the moodle hq team do a stirling job at them - as Gareth linked to above. 

Optimising code in an open source project for me is about the code i can optimise for my users, by changing core code where necessary, and turning stuff off we don't need.

As Howard said - "if you don't use it remove it".

cheers

stuart

 

 

In reply to Stuart Lamour

Re: Advice on optimising php/db code in moodle2+

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
Hi Stuart

The term 'bug' is used today in a much wider sense than the original suicide moth has intended https://en.wikipedia.org/wiki/Software_bug#Etymology.
smile

I have planned a footnote in my OP, 'bugs [*]", then forgot. Quoting Wikipedia further:

"Most bugs arise from mistakes and errors made by people in either a program's source code or its design, or in frameworks and operating systems used by such programs, and a few are caused by compilers producing incorrect code."

So a bug even in a limited sense is not a "coding error" but could be a thinking or even planning error. (In a wider sense, a bug is an "issue". See the bug tracker for example.) Inefficient programs usually belong to the latter category.

You say:
> Optimising code in an open source project for me is about the code i can optimise for my users, ...
optimising on what? An engineer knows that you can't optimise on everything!

> by changing core code where necessary, and turning stuff off we don't need. As Howard said - "if you don't use it remove it"

You are talking of removing those parts of the code, which _your_ users do not need. Firstly, you need to fork the code, because other's users may need them. Secondly how do you know that parts of the code are missing means faster execution times? The parts which are still being used go through the same old logic. In an interpreted language like PHP it is easy to visualize, but is also true for compiled languages.

That is why I said:
>> Putting it another way, if one has to ask, "Show me where to clean the code", he is the wrong person for the job.
Average of ratings: Useful (2)
In reply to Visvanath Ratnaweera

Re: Advice on optimising php/db code in moodle2+

by Stuart Lamour -
Picture of Plugin developers

Hi Visvanath,

for most of use using moodle we use a tool called git.

It's pretty powerful and lets us a take a fork or copy of moodle we can change, and it will let us compare our forked version with other versions.

It also keeps a record of any changes we make - a revision history.

Taking a quick look there are currently over 1000 forks of moodle on moodles official git!

Lots of us all fork moodle and change the code ever day.

Lots of us all ask for help doing that on these very forums every day.

Lots of us all support each other in doing this every day smile

I'm sorry that you personally feel we are all the wrong people for the job sad

regards

stuart

In reply to Stuart Lamour

Re: Advice on optimising php/db code in moodle2+

by Andrew Lyons -
Picture of Core developers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers Picture of Testers

Hi Stuart, 

Yes, I think that Visvanath is aware of git and it's uses in Moodle, being as he is a regular in the Moodle community, a Particularly Helpful Moodler, and the moderator of the performance forum. I think that he was more trying to point out that the approach you propose is incredibly impractical in real life.

The number of forks of Moodle on GitHub isn't actually so important - I would imagine that most people who fork Moodle on GitHub (incidentally, GitHub is not the official Moodle repository) are doing so in order to push bug fixes to be used against an issue in the tracker. Similarly, I would think that most of those who are making the kinds of changes you suggest don't actually push them publicly anyway for various reasons.

You're also likely to find that the experience of some of the more experienced developers would be to keep the number of these kinds of changes to a minimum. I would strongly suspect that many of the hacks in code that the bigger installations (e.g. partners) use are kept to a minimum in quantity, and each to as few lines as possible.

You'll see where people do ask on the forums about making changes to core code, they're generally pointed in alternative directions (e.g. local plugins, events, theme renderers, etc) first. Many of those who do need to make these kinds of changes (e.g. partners) later try and hook in core to avoid modifying core directly.

The issues with this approach include that:

  • you're likely to see large numbers of merge conflicts (which can be difficult to resolve) between versions;
  • unless you have a *very* keen eye on the entire Moodle codebase, then new versions may include additional references to the code you've previously simplified and which you subsequently miss; and
  • your changes may have undesired consequences that you haven't considered - both now (many eyes vs few eyes), or later with new versions relying on features you've removed.

I'd strongly suggest not taking this approach, but of course you are welcome to try.

Andrew

Average of ratings: Useful (3)
In reply to Andrew Lyons

Re: Advice on optimising php/db code in moodle2+

by Marcus Green -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers

Writing code is easy by comparison with testing code works on a vast number of possible installations, permutations and activities that may seem wildly unlikely but essential to someone somewhere.

As Andrew suggests most Git forks are not forks in the sense of  an entirely new tributory of Moodle that will continue as a separate development strand.

In reply to Andrew Lyons

Re: Advice on optimising php/db code in moodle2+

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
Hi Andrew

Thanks, that was very helpful.

Also thanks for the inside view of Moodle "code flow"!
wink

An update to my activity on moodle.org though: True, I have co-moderated "Hardware and Performance" forum for some time. Now taking a break. I announced it in the "Facilitator's corner" somewhere last March.
In reply to Andrew Lyons

Re: Advice on optimising php/db code in moodle2+

by Stuart Lamour -
Picture of Plugin developers

for my own dev in moodle i tend to start with adding a format - normally one for a single page/section/topic at a time.

I'm not a fan of moodle's interesting version where the introduction not a page of its own, i find its better being a page in its own right.

This format includes footer next/previous stuff.

and making the title of the course a link back to the homepage of the course.

i also add a custom page footer with recent updates and other stuff as custom blocks.

 

since 2.something i then modify the default renderers - in themes if i can manage to get what i need - but otherwise the mod/component core - often to optimise html/remove crud but also to and give me access to data in json format if i pass that as a request var when doing js interactions for content which might be better display inline - in the context of the learning.

e.g. if you add a link and its to youtube/vimeo/ted etc i like it displayed on the page rather than just a link :

This often leads to modifying courselib..

then i remove the default m2 nav and add something more usable similar to https://moodle.org/plugins/view.php?plugin=block_course_menu for displaying a menu of pages/topics in a course, and a button for admin to add a new page/section/topic, and a search this course box - which are default/sticky on ever course.

this seems to be much better for user experience than the current moodle stuff.

next i update quickforms, and add the custom elements to my custom format, making users required to add a title when creating a page in their course and also adds the an inline wysiwyg for the page description/introductions.

 

 

because tutors wouldn't except an essay without titles for each section, so why should we? This also really helps with that navigation.

So yeh, that is pretty much my baseline moodle reset, and i spend a lot of time merging it with each new release but as luck would have it i have pretty efficient tools for doing exactly that, and quite a bit of experience maintaining websites which run on opensource code smile

nothing much to add here to your own experience Andrew - but thanks for sharing, and hopefully this makes sense smile

In reply to Stuart Lamour

Re: Advice on optimising php/db code in moodle2+

by Stuart Lamour -
Picture of Plugin developers

putting it all together gets you something like :