hi,
the new semester is closing in and our moodle installation is used more and more.
this is of course good news, but unfortunatly we have to rearrange course categories quite often (sort them alphabetical, etc.) and while this has never been a fast thing to do it has now become nearly unusable as the number of categories has grown quite a bit.
after turning on the editing mode on the "Course categories" page (/course/index.php) the page takes nearly 80 seconds to load and about 30 mb of data is transfered to the browser.
this would be bearable if it would only happen once, but after moving a category the page is loaded again....move->load->move->load.
just pushing a category up the list a few places takes ages.
while doing this the server is doing nearly nothing. all the load is happening on the client as the html page is so complex and big.
i hope someone here has an advice what to do about this!
here are some stats:
moodle 1.8.2+
767 courses in
408 course categories
perf data for loading that page:
79.108845 secs
RAM: 29.8MB
Included 69 files
DB queries 4605
ticks: 7911 user: 363 sys: 14 cuser: 0 csys: 0
Load average: 0.45
Record cache hit/miss ratio : 0/0
br,
Tobias
Hi Tobias
The high number of DB queries looks like a prime candidate as the problem. Whilst there is work being done by the developers in the background, the only thing I can suggest is that you work through the usual stages of making the best of the hardware available. Work through the performance docs, checking these:
Ken
The high number of DB queries looks like a prime candidate as the problem. Whilst there is work being done by the developers in the background, the only thing I can suggest is that you work through the usual stages of making the best of the hardware available. Work through the performance docs, checking these:
- a php accelerator is installed on the server - your load average is low, so this may not make much difference
- the Moodle record cache is enabled to act as a small buffer for the database queries (see Site Admin -> Server -> Performance)
- there are no missing indexes in the database (see Site Admin -> Miscellaneous -> XMLDB Editor -> Check Indexes)
- you have configured MySQL to use cacheing as described in the performance docs.
Ken
hi ken,
thanks for your advice!
unfortunatly i don't think that the problem is on the server side this time.
as i've mentioned in my initial post the load on the server while loading the page is minimal (and ~4000 queries are not that much).
the problem is rather on the client side. my workstation is a core 2 duo and while loading the page the webbrowser uses 100% cpu power.
the html that is transfered is pretty complex (several hundreds of dropdown lists, etc.).
i've tested it in opera, firefox and ie. they all take ages to render that page.
the only solution to this would be imho to simplify the page.
br,
Tobias
thanks for your advice!
unfortunatly i don't think that the problem is on the server side this time.
as i've mentioned in my initial post the load on the server while loading the page is minimal (and ~4000 queries are not that much).
the problem is rather on the client side. my workstation is a core 2 duo and while loading the page the webbrowser uses 100% cpu power.
the html that is transfered is pretty complex (several hundreds of dropdown lists, etc.).
i've tested it in opera, firefox and ie. they all take ages to render that page.
the only solution to this would be imho to simplify the page.
br,
Tobias
I think the problem is that each category has a popup menu which in turn has a complete category listing in it, which means that the size of that page goes up very quickly as the number of categories increases and the browser has to work harder and harder to display it.
yes, exactly.
the only way to solve this imho is to restructure the html (eg. exchange the dropdown lists with links to an extra settings page or something).
the only way to solve this imho is to restructure the html (eg. exchange the dropdown lists with links to an extra settings page or something).
This is addressed in a patchseries coming soon to 1.9/2.0. I have the categories reordering page - and related operations - down to a constant number of DB queries (~30 for that page IIRC).
Optimizing DB queries is great but it is only one side of the equation. The usability of the page as it is now with a full structure loaded in a popup menu, a separate popup menu for each category, is not optimal for larger category structures. I think that a better approach would be to use checkboxes by categories with a single popup or button for each function. I used that approach for managing courses in multiple categories and we were quite satisfied with the operation. Besides making the page faster to load, this approach also brings bulk operations, that is operating on more than a single category at a time.
As we are in hardware & performance, I tend to think that Tobias' post is about the slowness of loading the page.
If there are UI changes coming that are both better for users, and lighter for the backend... yay!
i'm of course always happy with a reduction of queries in the backend! but if those optimized queries will return the same data as before and it is presented in the same way as it is now (hundreds of dropdown lists each with dozens of elements) there will be no speed increase for the user as the bottleneck here is clearly the client (browser).
so ui changes are _very_ welcome, too
so ui changes are _very_ welcome, too
Are you sure it is moving 30MB of HTML each time? That sounds like a ton. Perhaps we can shrink the html.
Shrinking HTML is exactly what I suggested although probably not along the lines of your thinking 
I have 102 categories 3 levels deep and most category names have only one or two words. My page weighs at 1.7 MB as HTML file. In Firefox, it takes 25-30 seconds to display it from Moodle server and 15-20 seconds as a local file . Safari seems to be twice as fast. If I turn editing off, the same page displays instantly. 30 MB sounds extreme but with more categories, particularly with longer category names, and more levels it is plausible.
I have 102 categories 3 levels deep and most category names have only one or two words. My page weighs at 1.7 MB as HTML file. In Firefox, it takes 25-30 seconds to display it from Moodle server and 15-20 seconds as a local file . Safari seems to be twice as fast. If I turn editing off, the same page displays instantly. 30 MB sounds extreme but with more categories, particularly with longer category names, and more levels it is plausible.
yes i am
i saved the file with firefox.
ls -al index.php.html
-rw-r--r-- 1 marxist marxist 31259471 2007-09-05 09:16 index.php.html
this is a file with >31 million characters...
the actual memory usage of the browser displaying this file is much higher as it has to show images, etc., too.
and this file has to be loaded after _each_ operation on that page (eg. move category).
here is the filesize for the same page and editing switched off for comparison:
ls -al index.php2.html
-rw-r--r-- 1 marxist marxist 71029 2007-09-05 09:27 index.php2.html
shrinking the html in editing mode would be much appreciated!
i saved the file with firefox.
ls -al index.php.html
-rw-r--r-- 1 marxist marxist 31259471 2007-09-05 09:16 index.php.html
this is a file with >31 million characters...
the actual memory usage of the browser displaying this file is much higher as it has to show images, etc., too.
and this file has to be loaded after _each_ operation on that page (eg. move category).
here is the filesize for the same page and editing switched off for comparison:
ls -al index.php2.html
-rw-r--r-- 1 marxist marxist 71029 2007-09-05 09:27 index.php2.html
shrinking the html in editing mode would be much appreciated!
You are right that this is wrong forum, although we are still talking deficient performance (slowness of loading the page) just it is not the fault the server but page content.
ok, sorry. but which forum would then be appropriat for this kind of problem?
Hi Tobas,
Sorry this might off topic but ..may I know what software/method you used to check the load ? Is there any benchmark result I can compare?
perf data for loading that page:
79.108845 secs
RAM: 29.8MB
Included 69 files
DB queries 4605
ticks: 7911 user: 363 sys: 14 cuser: 0 csys: 0
Load average: 0.45
Record cache hit/miss ratio : 0/0
Thanks
Lee
hi lee,
i'm not sure i do understand your question.
do you want to know how i got the "perf data...: 79.108845 secs" data or do you want to know how i checked the load on the server?
to be sure, i'll answer both ;)
to get some performance data at the footer of every page in moodle go to "administration->server->debugging" in moodle an activate "Performance info perfdebug".
you can set special options in your "config.php" on the server to choose what kind of information you want to see. for some examples take a look in your config-dist.php file.
for the server load i just used the "top" command (in linux) while loading the page.
i hope i could be of any help.
br,
Tobias
i'm not sure i do understand your question.
do you want to know how i got the "perf data...: 79.108845 secs" data or do you want to know how i checked the load on the server?
to be sure, i'll answer both ;)
to get some performance data at the footer of every page in moodle go to "administration->server->debugging" in moodle an activate "Performance info perfdebug".
you can set special options in your "config.php" on the server to choose what kind of information you want to see. for some examples take a look in your config-dist.php file.
for the server load i just used the "top" command (in linux) while loading the page.
i hope i could be of any help.
br,
Tobias
Hello,
I'm just posting because I also have to face the same type of problem with a similar quantity of courses and course categories (~800 and ~100).
At first, I also thought that it was a server's issue (connection's speed or script's execution) but as I only have a small visibility of statistics i couldn't have a look to precise performance's log; likely to what Tobias Mark wrote i can give those approximative statistics:
- Moodle 1.6.4
- about 850 courses and 100 categories
- between 2 and 3 minutes to load the edition's mode
- while loading the cpu's occupation stays between 95% and 99% and firefox jumps from 70mb to 280mb
- i saved the page and this file "ExternalURLStats.htm" weights 30mb..!
If you have any informations or advice, i'll be grateful
Thanks in advance, Julien
I'm just posting because I also have to face the same type of problem with a similar quantity of courses and course categories (~800 and ~100).
At first, I also thought that it was a server's issue (connection's speed or script's execution) but as I only have a small visibility of statistics i couldn't have a look to precise performance's log; likely to what Tobias Mark wrote i can give those approximative statistics:
- Moodle 1.6.4
- about 850 courses and 100 categories
- between 2 and 3 minutes to load the edition's mode
- while loading the cpu's occupation stays between 95% and 99% and firefox jumps from 70mb to 280mb
- i saved the page and this file "ExternalURLStats.htm" weights 30mb..!
If you have any informations or advice, i'll be grateful
Thanks in advance, Julien
OTOH, we are overlooking the obvious immediate fix: enable rcache!
Hello,
Do this means that rcache needs to be enabled to improve this process..?
About the size of the file, i saved the page and yes it is 33mb exactly, it's really huge..!
Thanks in advance, Julien
Do this means that rcache needs to be enabled to improve this process..?
About the size of the file, i saved the page and yes it is 33mb exactly, it's really huge..!
Thanks in advance, Julien
hi martín!
thanks for your advice. unfortunatly rcache is not going to help at all in this case.
this is not a problem of the backend producing too many queries (this time), but having too complex and big html that can not be rendert in an acceptable time at the client (browser).
please reread my initial post. the server is idle when one loads the category list in edditing mode. but the client computer is maxed out for nearly 80 seconds.
regardless of which optmizations we make at the backend. the html output will always be the same as long as the php code generating it is not changed.
br,
Tobias
thanks for your advice. unfortunatly rcache is not going to help at all in this case.
this is not a problem of the backend producing too many queries (this time), but having too complex and big html that can not be rendert in an acceptable time at the client (browser).
please reread my initial post. the server is idle when one loads the category list in edditing mode. but the client computer is maxed out for nearly 80 seconds.
regardless of which optmizations we make at the backend. the html output will always be the same as long as the php code generating it is not changed.
br,
Tobias
Hello,
Would it be possible to imagine that the dropdown menu is sent only once by the server and repeated as many times as needed by a small javascript..? I know that the client will be sollicitated but 30mb is really big when you reach lot of courses...
Thanks in advance to let me know what you think about this possibility..
Ju
Would it be possible to imagine that the dropdown menu is sent only once by the server and repeated as many times as needed by a small javascript..? I know that the client will be sollicitated but 30mb is really big when you reach lot of courses...
Thanks in advance to let me know what you think about this possibility..
Ju
Hi,
This same problem happens to me also. And probably to every institution admin with a considerable amount of course/categories.
Hei, how do you do it?
http://docs.moodle.org/en/Installations_10000_plus
http://docs.moodle.org/en/Installations_30000_plus
i consider it a major problem...
Found an open bug on tracker... vote for it and maybe we can have it fixed:
http://tracker.moodle.org/browse/MDL-18184
thanx,
susana
This same problem happens to me also. And probably to every institution admin with a considerable amount of course/categories.
Hei, how do you do it?
http://docs.moodle.org/en/Installations_10000_plus
http://docs.moodle.org/en/Installations_30000_plus
i consider it a major problem...
Found an open bug on tracker... vote for it and maybe we can have it fixed:
http://tracker.moodle.org/browse/MDL-18184
thanx,
susana
Moving top-level categories affects the roles and capabilities, because the access control system in Moodle is heavily tied to the course categories hierarchy.
Very large sites don't move top-level categories around, because their courses are usually created / enrolled / torn down based on backend data (using an enrolment plugin). So those have pre-made rules that rarely change. If they change, the change usually is after hours or while putting Moodle in maintenance mode.
The long-term fix is to separate the categories for listing (just tag the courses) from the 'access control' categories.
That is a bit of a mission though.
Very large sites don't move top-level categories around, because their courses are usually created / enrolled / torn down based on backend data (using an enrolment plugin). So those have pre-made rules that rarely change. If they change, the change usually is after hours or while putting Moodle in maintenance mode.
The long-term fix is to separate the categories for listing (just tag the courses) from the 'access control' categories.
That is a bit of a mission though.
Actually, Martin, I don't think it is that big a mission.
As part of Navigation 2.0, my view is that the middle of pages like index.php, course/index.php and course/category.php should be changed, so that instead of specific code, that area is just filled with blocks.
At the moment we just have the strange set of 4 drop-downs on Administration -> Front Page -> Front Page settings to select and order the type of content you want on the front page. That is a bit like a really crappy blocks editing interface.
Using real blocks would give much more flexibility, and make it easy for people to make their own custom ones. (Of course, whoever converts the current code into blocks had better understand the performance implications.)
Anyway, when you combine the ability to write custom blocks to display the course list exactly how you like, with tagging of courses and whatever other metadata will be added to courses in the future, you start to have a lot of flexibility and power.
As part of Navigation 2.0, my view is that the middle of pages like index.php, course/index.php and course/category.php should be changed, so that instead of specific code, that area is just filled with blocks.
At the moment we just have the strange set of 4 drop-downs on Administration -> Front Page -> Front Page settings to select and order the type of content you want on the front page. That is a bit like a really crappy blocks editing interface.
Using real blocks would give much more flexibility, and make it easy for people to make their own custom ones. (Of course, whoever converts the current code into blocks had better understand the performance implications.)
Anyway, when you combine the ability to write custom blocks to display the course list exactly how you like, with tagging of courses and whatever other metadata will be added to courses in the future, you start to have a lot of flexibility and power.
Hi,
We also have IS integration. Courses and categories are automatically created. But we have a few courses (workshops, training, projects) that are manually created... and we need sometimes to edit those categories (show/hide, rearrange, etc)...
... and it takes so long just loading the edit course categories page...
We also have IS integration. Courses and categories are automatically created. But we have a few courses (workshops, training, projects) that are manually created... and we need sometimes to edit those categories (show/hide, rearrange, etc)...
... and it takes so long just loading the edit course categories page...
I just voted for the Tracker-Issue mentioned here:
http://tracker.moodle.org/browse/MDL-18184
See my comment there. Until we get it fixed, I proceeded as follows to delete a couple of categories in our >2500 courses and ~500 categories site:
Rosario
http://tracker.moodle.org/browse/MDL-18184
See my comment there. Until we get it fixed, I proceeded as follows to delete a couple of categories in our >2500 courses and ~500 categories site:
- open two browsers and log in as admin
- in one get displayed the Edit/Add courses list
- yo will find a sessionKey and a link to the category you wish to delete or move around if you point to the desired icon with your mouse. Leave this page open so as to wait only once (683 secs in my case!)
- go into the second browser and call the "show all courses page", i.e. the same page not in edit mode, which loads at a normal speed. Here you look up the category id numbers quickly for all those categories you want to manipulate
- in the first browser type in the urls into the address-box to delete or move around the categories, which is something similar to this one: https://your.moodle.edu/course/index.php?delete=614&sesskey=dbNyFPpwI8
- after confirmation of the deletion do not wait the whole page to be displayed again, stop it with the according browser-button
- look up the category id in the first browser and retype (copy/paste/edit) the next url, e.g. https://your.moodle.edu/course/index.php?delete=615&sesskey=dbNyFPpwI8
Rosario