Getting 404 errors when load testing Moodle 2.9 with only 10 concurrent users

Getting 404 errors when load testing Moodle 2.9 with only 10 concurrent users

by Montai Settapokin -
Number of replies: 10

We recently load testing our Moodle 2.9 using the LoadRunner software.


These are what we have:

Software: Moodle 2.9 on Redhat 6.4

Hardware: 4 CPU , 8GB Memory , 100GB HDD


and our database server:

Software: Windows Server 2012 with MSSQL Server 2012

Hardware: 4 CPU , 8GB Memory , 100GB HDD


First try with 250 users, and it caused our Apache to crash, and we have to restart the service.


So we tried with only 10 users (unique account for each), and this is what we get:


The test scenario is quite simple: login > entering course > entering quiz > starting a new quiz (5 multiple choice questions) > pressing next > trying to submit the quiz > logout

As you can see, there are 7 fails out of 10 when trying to submit the quiz, and the error we got is:


Error -26627:Action.c(966) Error -26627 HTTP Status-Code=404 (Not Found) for "https//www.ourserver.com/moodle/mod/quiz/processattempt.php"


404 (Not Found) error on the processattempt.php file.


We checked the CPU / Memory usage on both the web server & the database server, and the usage are still pretty low (1 - 2%)


Any help would be appreciated. Thank you.

Average of ratings: -
In reply to Montai Settapokin

Re: Getting 404 errors when load testing Moodle 2.9 with only 10 concurrent users

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

Can LoadRunner save the HTML of each page it requests? I have only used JMeter for load-testing and it can. Having the HTML of each page lets you see any error messages.

Then, to get error messages, you probably need to turn on Debugging. You probably want that off in real user, and hence when you are really load-testing, but to diagnose this problem you need to turn it on temporarily.

Average of ratings: -
In reply to Tim Hunt

Re: Getting 404 errors when load testing Moodle 2.9 with only 10 concurrent users

by Montai Settapokin -

Since the errors are 404 File not found, there were no HTML return for the requests.


Our apache error_log shows the following for the errors as well:


Database transaction aborted

automatically in /app/apache/htdocs/moodle/mod/quiz/processattempt.php, referer:

https://www.ourserver.com/moodle/mod/quiz/summary.php?attempt=478


I read the topic: https://moodle.org/mod/forum/discuss.php?d=229365 and it seems to be related to sessions issue.


Let me turn on Debugging to find out more. Thanks.


Average of ratings: -
In reply to Montai Settapokin

Re: Getting 404 errors when load testing Moodle 2.9 with only 10 concurrent users

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

"Since the errors are 404 File not found, there were no HTML return for the requests."

Wrong. E.g. go to https://moodle.org/mod/forum/discuss.php?d=-1 That is a 404 and there is a HTML respone too including the error message.

Is your load-testing system handlng logins correctly, or is every request (that is no a 404) ending up as a redirect to login/index.php?

Average of ratings: -
In reply to Tim Hunt

Re: Getting 404 errors when load testing Moodle 2.9 with only 10 concurrent users

by Montai Settapokin -

"Is your load-testing system handlng logins correctly, or is every request (that is no a 404) ending up as a redirect to login/index.php?"


I have created another Moodle site on AWS yesterday, just to test whether our loadrunner scripts are running correctly through the entire flow. Using 20 concurrent users.



The only errors we got are timeout errors:

Error -27728 Step download timeout (120 seconds) has expired when downloading non-resource(s) 


So I guess the scripts are running correctly.


"That is a 404 and there is a HTML respone too including the error message."


Let me go find the HTML response, hopefully it will show the error messages to fix our issues.


Thanks!

Average of ratings: -
In reply to Montai Settapokin

Re: Getting 404 errors when load testing Moodle 2.9 with only 10 concurrent users

by Montai Settapokin -

Got the HTML responses:


Error reading from database.


Debug info: The COMMIT TRANSACTION request has no corresponding BEGIN TRANSACTION. COMMIT TRANSACTION [NULL]

Error code: dmlreadexception


Any thoughts?

Average of ratings: -
In reply to Montai Settapokin

Re: Getting 404 errors when load testing Moodle 2.9 with only 10 concurrent users

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

Sorry, no idea. I have never seen an error like that, and given the way Moodle handles exceptions, I cannot think of any way that a COMMIT could be done if a transaction had not been started with BEGIN.

Can you reproduce the problem without LoadRunner being involved? Can you set Debugging to DEVELOPER level so we also get a stack trace for the error?

Average of ratings: -
In reply to Tim Hunt

Re: Getting 404 errors when load testing Moodle 2.9 with only 10 concurrent users

by Montai Settapokin -

Yes, I set up JMeter this morning.


This is the trace:


Debug info: The COMMIT TRANSACTION request has no corresponding BEGIN TRANSACTION. COMMIT TRANSACTION [NULL] Error code: dmlreadexception

Stack trace:
line 443 of /lib/dml/moodle_database.php: dml_read_exception thrown
line 242 of /lib/dml/mssql_native_moodle_database.php: call to moodle_database->query_end()
line 1425 of /lib/dml/mssql_native_moodle_database.php: call to mssql_native_moodle_database->query_end()
line 2408 of /lib/dml/moodle_database.php: call to mssql_native_moodle_database->commit_transaction()
line 92 of /lib/dml/moodle_transaction.php: call to moodle_database->commit_delegated_transaction()
line 192 of /mod/quiz/processattempt.php: call to moodle_transaction->allow_commit()

Average of ratings: -
In reply to Montai Settapokin

Re: Getting 404 errors when load testing Moodle 2.9 with only 10 concurrent users

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

The stack trace basically illustrates what I mean about Moodle's transaction handling. Moodle uses moodle_transaction objects to track transactions. The transaction is started in the constructor. There is code to verify that you only call moodle_transaction->allow_commit() once per transaction.

For you to get the error message you are seeing something very weird (not normal execution of Moodle code) must be going on.

Average of ratings: -
In reply to Tim Hunt

Re: Getting 404 errors when load testing Moodle 2.9 with only 10 concurrent users

by Montai Settapokin -

I have created a tracker for the issue: https://tracker.moodle.org/browse/MDL-51245


Not sure if we have to modify the Moodle codes, or we can just do some configurations on MSSQL to solve it.

Average of ratings: -
In reply to Montai Settapokin

Re: Getting 404 errors when load testing Moodle 2.9 with only 10 concurrent users

by Montai Settapokin -
After setting up several Moodle + MSSQL sites, we have finally solved the problems.

Something about MSSQL Isolation Levels: https://msdn.microsoft.com/en-us/library/ms378149%28v=sql.110%29.aspx

We run these two commands on our MSSQL:

ALTER DATABASE 'moodle' SET ALLOW_SNAPSHOT_ISOLATION ON;
ALTER DATABASE 'moodle' SET READ_COMMITTED_SNAPSHOT ON WITH ROLLBACK IMMEDIATE;

And the "Error reading from database" errors disappear. Now we are getting 503 Service Unavailable and 500 Internal Server Error errors instead, so it's just about tuning the apache to handle high load.

Thanks Tim! smile
Average of ratings: Useful (2)