Our course is getting slow day by day. By using https://rpm.newrelic.com/, have monitored Moodle performance, where I came across a result that on rendering a page, 'core_renderer' is taking max time as compared to all other activities. And under 'core_render', 'generate_sections_and_activities' is taking most of time as shown in attachment. These are core files of, being called once and twice on every page. Still unable to find any thing.
Details of servers are as follow
Moodle version: 3.3.2
CPU: Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz
Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz
Do you guys have any suggestions about how to go and fix this as i am out of ideas?
Thanks in advance.
That monitoring tool requires an agent be installed on servers ... is that correct? Agent has to run to report back to the interface you are using to monitor ... that correct? How much memory does Agent take? How often is Agent involved in sending data to the monitoring server?
What does 'top' from terminal look like on both servers?
Either of them using SWAP space?
On the DB server, what percentage of total memory is the DB using?
Moodle has had a long history of what's called 'the scroll of death' - a 'long' course with many sections/resources takes a long time to render.
Not sure if 3.3.x has it (don't run a 3.3.x any more to see myself), but is there a setting for a course that shows only 1 section first, then there are navigational links to other sections? A course that had 52 sections and displaying all sections with their resource/other links will take longer to render.
'spirit of sharing', Ken
Yes, monitoring tool's agent is installed on servers and 'Agent has to run to report back to the interface you are using to monitor', yes.
Out of two courses, one has 14 sections and 53 activities and other has 11 sections and 49 activities (much lesser than 52 sections).
And for 'top' command screenshot is shared bellow for both servers (left one for web and right one for db server).
We also are using several grading formulas in courses. And till now, no slow query found. Above shared results was for 100 virtual users load test.
After a lot of testing , found that the user load at which the slowness occurs on our production environment is 400+ active users. Is it a load manageable by Moodle?
400+ active users generate a big load - very big if they are sitting for an on-line test (Quiz) all together. Luckily you've separate web and database servers, making it easier to understand the 'top' outputs.
- web server: Definitely over loaded. Load average of 47(!), with 50 processes running - almost all nginx - each consuming ~4.5% CPU and ~4% RAM. That machine needs more resources.
-- Xeon(R) CPU E5-2670 0 @ 2.60GHz is not a bad CPU : 8 core, 16 thread? Did you confirm with /proc/cpuinfo?
-- 2 GB RAM, are you sure? That would be a disappointment. To match the CPU, you need to be in the range of 16 GB. The low RAM could very well be your problem. (I am still puzzled that your server is not swapping.)
-- You didn't say anything about the disk system, neither the system software, PHP specially, and the caching mechanisms in use.
- DB server
Seems to be running much smoother - load average of 26. Again the same CPU but only 1 GB RAM. Way too low. For a database server a good disk system and lot of RAM for database caching are important.
You have a demanding load, need good resources to start.
Well, core_rendere is not a helpful think to focus on there. It is more interesting to focus on block_navigation and it its use of global_navigation.
Have a look at the course this performance info relates to: does it have a particualrly large number of sections and activities?
You seem to have an 11s page-load, of which 2.7 seconds is the navigation block.
For comparison, on some old profiling results I have sitting around, it was a 3.2 second page-load, of which 0.2s (6%) was block_navigation->get_contents.
So, your site seems surprisingly slow. However, there is not enough information in your screen grab to see what is really going on. (I am more used to using xhprof/Tideways - https://docs.moodle.org/dev/Profiling_PHP). As well as time, that also shows number of function calls, which is useful to see.
Ultimately, most time processing a Moodle page normally ends up being either database query or cache gets.
We tested site performance by using Moodle->Site Administration->Plugins->Caching->Test Performance.
Attached are the results. At 50,000, there was a 500 exception that is also attached.
Moodle testperformance 100
Moodle testperformance 1000
Moodle testperformance 5000
Moodle testperformance 10000
Moodle testperformance 50000
Moodle testperformance 50000 500 error description:
'[26-Oct-2018 06:33:27 America/Chicago] PHP Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to allocate 24 bytes) in D:\PROD\htdocs_live\moodle\lib\dml\mysqli_native_moodle_database.php on line 1194'
We served static content through CDN on our test server but that did not help rather response time was increased.
We installed memcache on our test server and setup all the Application Mode cache definitions in Moodle to use memcache by default. But that did not help either.
We used profiling_php, installed xDebug and *attached are the cachegrind files for one of the course/view urls
*As it exceeded max size (7MB), is shared from dropbox https://www.dropbox.com/s/hplsqrehiy3q9yf/course-view.zip?dl=0 .
Is this info helps? or anymore info would be required to help find the issue.