We are just planning our upgrade to STACK version 4.1 (from 3.x) at the Open University, and we have a lot of questions, many of which are quite complex. Because of the major improvements in STACK, I am a bit worried that some of our old questions will stop working. Therefore, I am planning to tests things quite carefully before doing the upgrade, and I thought people might be interested in how I am going to do that. (I have not done this yet. I will report back later to tell you if it worked!)
A. Find the scope of the problem
If you do a simple query: SELECT * FROM mdl_question WHERE qtype = 'stack' on our main Moodle site, that finds just over 12,000 questions. However, the way we work at the OU, each year, for each new group of students, we make a copy of each course. Therefore, our question bank contains mulitple copies of some questions.
However, for all our complex STACK questions, almost all the Maxima code is in the question variables (which I think is very good practice). SELECT DISTINCT questionvariables FROM mdl_qtype_stack_options instead finds only 4000 'different' questions, which is more manageable.
Then, we should consider that a quesiton will only be helpful for testing if the question author wrote some question tests, and when I include that condition, that leaves 3,500 questions. (Again, think >85% of STACK questions having question tests is pretty good going by OU STACK question creators). My final query was
SELECT
MAX(questionid),
COUNT(1) number_of_similar_questions,
questionvariables,
(SELECT COUNT(1)
FROM mdl_qtype_stack_qtests iqso
WHERE iqso.questionid = MAX(qso.questionid)
) AS number_of_question_tests
FROM mdl_qtype_stack_options qso
WHERE questionvariables <> ''
GROUP BY questionvariables
HAVING (SELECT COUNT(1)
FROM mdl_qtype_stack_qtests iqso
WHERE iqso.questionid = MAX(qso.questionid)) > 0
ORDER BY COUNT(1) DESC
And
SELECT qso.questionid,
(SELECT COUNT(1)
FROM mdl_qtype_stack_qtests iqso
WHERE iqso.questionid = qso.questionid
) AS number_of_question_tests
FROM mdl_qtype_stack_options qso
WHERE questionvariables = '' AND (
SELECT COUNT(1)
FROM mdl_qtype_stack_qtests iqso
WHERE iqso.questionid = qso.questionid
) > 0
To catch the questions where the question variables are not used - we can't assume those are all the same.
B. Getting the questions off the live server
I decided that the easiest way to get the questions off the live server was to use Moodle XML export. On the question tests and deployed variants page for a STACK question there is an 'Export this question' link which leads to a URL like
https://moodle.example.com/question/type/stack/exportone.php?questionid=12345&cmid=4567&sesskey=XXXXXXXXXX. I wrote a simple script that looped over all the questionids from my queries, and downloaded those files using CURL. (I had to lot into Moodle, and put my session cookies and sesskey into that script to make it work. A bit of a hack, but.
Then I wrote another script that looped over all the XML files in a folder, and
This is not the most efficient way to transfer the bytes from one server to the other, but they were simple scripts to write, and I could do other things while the scripts ran, so this was more efficient uses of my time.
C. Running the question tests
Then, I used the standard 'run the question tests in bulk script' to run all the question tests.
Unfortunately, it was not safe to assume that the question tests all passed before the upgrade. Therefore, I also had to restore the questions into a test server running our current version of STACK. The worrying questions are the ones where the question tests pass on 3.5 but fail on 4.1.
The other details is the need to use the 'fix maths delimiters script' to upgrade questions from the 3.x to 4.1 STACK syntax.