Load tests for STACK / goemaxima

Load tests for STACK / goemaxima

by Oliver Smitkowski -
Number of replies: 4

Hi everyone,

We use STACK with goemaxima, with goemaxima running on a separate server. Now we would like to perform some load tests. For starters, we would like to bypass moodle and send API requests directly to the Maxima server.

I know that the command line of the Maxima server can be used to execute a statement like diff(sin(x^2),x);. However, I would much rather send requests to Maxima from outside via cURL. Is that possible?

Requests like the following are successful, but always return (%i7)\n(%i8), where "\n" stands for a line break. As far as I know, this indicates that the request was successful, but activated Maxima in an interactive mode instead of returning the result of the computation.

curl -X POST “http://<our-goemaxima-ip>:8080/goemaxima?timeout=10000” \ 
  -H “Content-Type: application/json” \
  -d '{
    "input": "diff(sin(x^2),x);"
  }'

Can I adapt this example so that Maxima computes and returns the result for input instead?

I would also be very happy about other approaches for simple load tests. The background to my question are planned tests in which 150 students (engineering sciences; first year of study) will work on a moodle quiz that consists largely of STACK-type questions.

Thanks
Oliver

Average of ratings: -
In reply to Oliver Smitkowski

Re: Load tests for STACK / goemaxima

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
I am trying to remember what we did when we moved to goemaxima.

One reasource you have, if you have been running STACK for some time, is the qtype_stack_cas_cache table in the database. That caches the command that was sent to Maxima, and the response that was recieved at the time.

Running through a few thousand rows of the table makes a good load test. I remember thinking about scripging that.

But, I don't think we ever did, because goemaxima was cearly faster than we had before (about 2x).

I think that, to prove that, all we did was to use Admin -> Plugins -> Question types -> STACK -> bulk test script on a course with a few hundred questions. Comparing the run-time of that script with our old setup and our new setup (remebering to clear the CAS cache inbetween) was sufficient testing for us.

(Note, that tests pretty-much single-threaded performance of goemaxima, not true capacity.)
Average of ratings: -
In reply to Tim Hunt

Re: Load tests for STACK / goemaxima

by Oliver Smitkowski -

Thank you, Tim!

We have 166 question tests in the most relevant course, and running all of them takes about 67 seconds — averaging 0.4 seconds per test. While that may not seem bad at first, with 150 students, waiting time could become annoying.

goemaxima can handle up to 32 parallel requests, and while our server only has two processor cores, we could use a larger server. I also know Maxima uses caching, and students sometimes receive identical questions.

Given that, would you expect processing 166 comparable student responses to be significantly faster than running the 166-question test suite via the bulk test script?

Average of ratings: -
In reply to Oliver Smitkowski

Re: Load tests for STACK / goemaxima

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
It is impossible to say with any certainty. Bear in mind that to run a question test on one question, STACK first has to do the initial randomisation and generate the question; then grade ther response; then render any maths in the feedback. And, when a student is working on the question, that will be well spread out in time as they think, it won't all happen in a fraction of a second.

You really need to build confidence in the system gradually. Don't go out and try to run a 200-student exam tomorrow, where everyone starts at the same second and submits at the same second when time expires.

Instead, can you find something like a homework assignment, where students work in there own time, so the load is spread out. If you can do that, an have monitoring of the server CPU / Ram / disc IO / networking, then you will see where you are at in reality (including the effects of things like caching) and how much room you have to grow with your current hardware - but it looks like your current hardware is pretty capable.
Average of ratings: -
In reply to Tim Hunt

Re: Load tests for STACK / goemaxima

by Oliver Smitkowski -

Thank you! I’ve since ran the bulk test script from multiple different browser tabs for the most relevant (and largest) course. Results were  promising:

  • A single test suite took 67 seconds to finish.
  • Four parallel test suites for the large course took between 95 seconds to finish.
  • Eight parallel test suites took 175 seconds.
  • 16 parallel test suites took 245 seconds.

We then upgraded our server. A single test suite now takes 31 seconds while 16 parallel test suites take 125 seconds to finish. More importantly, goemaxima seems to handle parallel requests efficiently.

Starting today, we have three weeks in which a few hundred students can work on test tasks whenever they want. Later, they will work on tests in smaller groups at the same time. Until then, we will keep an eye on the server's workload. I will post the final server configuration and a few sample questions here later and describe how the exams went.

Average of ratings: Useful (1)