It's 2:30 and I have been importing grades since 8:30AM

Re: It's 2:30 and I have been importing grades since 8:30AM

by Visvanath Ratnaweera -
Number of replies: 4
Picture of Particularly helpful Moodlers Picture of Translators
Hi

Your "sysadmin stuff" is fine. In any case, they are modern than mine!
wink

The question we are trying to answer is whether the grader report is not well scalable. Right now, in your set up, it is obviously not. But before making a case for the developers to look at, you have to be sure that a) your infrastructure is not weak b) it is not poorly set up.

About a) Your web server is a VPS with 2 CPUs and 1 GB RAM, which is definitely low-end. It is possible that the performance of the web server is not critical for this case, as the performance graphs show, there was a huge load on the database. So the database part is the bottle neck. You didn't give the resources of the database server. But I get the feeling that your IT team runs it. (Why should it be on the other end of the campus otherwise?)

My estimation is that your the infrastructure is low-end.

About b) You also did not mention how the 8 GB volume for the dataroot is attached. The response of the dataroot is critical for Moodle! Also the fact that the database server goes through the campus network (with or without proxy) is a weak point. The two servers need to be back-to-back on their own GBit segment.

Even on this point I am not too convinced.

I don't claim that Moodle handles big grader reports efficiently. There were some discussions in this forum on similar topics:
- Moodle grader report editing extremely slow
https://moodle.org/mod/forum/discuss.php?d=347370

- New Moodle Slow saving grade changes
https://moodle.org/mod/forum/discuss.php?d=325620

If you want to follow that argument, you need to collect data on the database queries the operation generated. (The DBMS have special tools for that.) And file a bug report.
In reply to Visvanath Ratnaweera

Re: It's 2:30 and I have been importing grades since 8:30AM

by Diane Soini -

It's interesting you say the database and webservers need to be back-to-back on their own whatever. That is how we used to have it set up. Both were on the same hypervisor, whatever that is. But we changed this recently and noticed a drop in performance. We changed over to a database cluster using Percona and at that time we separated the two systems. We also did not change any of the defaults to the Percona configurations, since advice out there is to accept the defaults. My tests were intended to see if we did need to change some of the defaults.

I'm sure my set-up here is pretty low end. My test was simply to try to put some load on the database and see if any tweaking to the configuration would have an effect. But there was no effect with any of the config changes. Your comments lead me to believe that the sysadmins need to look elsewhere.

My other goal was to "break" the database, which I was not able to do even with an upload of 100,000 grade items (this was a moodle-generated course so I don't think the gradebook is misconfigured.)

Thank you for your help. I'll point the systems guys to this thread and maybe it will help them.

In reply to Diane Soini

Re: It's 2:30 and I have been importing grades since 8:30AM

by Visvanath Ratnaweera -
Picture of Particularly helpful Moodlers Picture of Translators
Hi

Yours seems to be a big institution with its own sophisticated IT infrastructure. Yes, it is sensible for the "systems guys" to get acquainted with Moodle!
wink

Re-reading our discussion, it would have been more efficient, if you told us at the beginning what you want to achieve rather than what you are doing. The original clearly said you want to _test the performance_ [of your test setup] by importing a huge gradebook and came across prohibitive execution times. The cause is identified as the underpowered (and over complicated?) test rig. Obviously just "tweaking" is not going to help.

Your other goal, 'to "break" the database', in the previous post is not clear to me. Do you mean divide the work between two database servers or to crash the database server? The former is not to be achieved by easy means. The latter, your database administrators should know.
In reply to Visvanath Ratnaweera

Re: It's 2:30 and I have been importing grades since 8:30AM

by Diane Soini -

Sorry I was on vacation.

As far as "breaking" the database, we had a real-life situation where a large lecture hall class was taking the final exam and the database locked up when several of the students reached the time limit. There was row-locking on the database that resulted in numerous students receiving database errors during their quiz.

I attempted to set up a test that might repeat this, but it did not. I thought perhaps if the gradebook had to recalculate a lot of grades at the moment the quiz closed that might trigger a similar row-locking problem but it did not. (This is the reason I was trying to upload all those grades that took 6 hours to upload.) Other than that, we were hoping that there was some way to trigger some kind of database problem, but I wasn't able to get anywhere near the limits of the system using jmeter. We simply wanted to see the database reach its limits to see if any settings changes would indicate that we could avoid the problem we had with the quiz again. In the end, nothing I could get jmeter to do would cause enough stress and none of the settings changes the systems guys tried improved anything.

In the end, I think our performance problems aren't going to be figured out by jmeter and a test system so much as our old tried-and-true incremental changes out on production. (i.e. we make a small change on production and see if there's an improvement, then make another small change.)