Here in Moodle HQ, we have been looking alternatives on automated testing Moodle for a while. And thanks to David Scotson, who has mentioned Selenium.
I have written some test cases in Selenium and would like to hear some feedback from the community. What I found is the test case is not really flexible. We need some time to sit and create a test environment, and this environment should be exactly the same each time we want to run the test case (for example link in quiz questions contains numerical id that should be the same for each test) otherwise the test will fail.
Due to this reason (the sensitivity reason ), I can not give the link to the selenium test here. However, if anyone is interested just let me know and I'll give you the link. Feedback and critique is then very much appreciated
I am interested in seeing your work on Selenium. It's probably a good ideat to get in touch with Rod Norfor @ open university . IIRC, he is (was?) working on something similar, with a modified PHPUnit, and he had a reasonably good set of tests for datalib functions.
These tests run in your browsers and are more like:
Click this button, check output, select this item, check output, etc
At the Open University we have a Unit test framework built on Lastcraft's Simple Test (Martin L's post). This is similar to JUnit for Java, but written in PHP using version 4-compatible classes. The code to be tested can be class-based or procedural, and the documentation is good. There is support for grouping tests, mock objects and testing web pages at the browser level.
So far, the VLE coders have added 560 tests for new OU-Moodle code, which is very encouraging Unfortunately, we have not had the time/resource to retrospectively add many tests to the existing Moodle code base (datalib would be a prime target). This would make our code merges more straightforward, but is a difficult task
From a quick look at Selenium it is interesting - I agree with Martin D that Selenium complements, but doesn't replace unit tests.
Overall, with Selenium and/or Simple Test, I'm very keen on getting some test framework off the ground...
I suspect we won't have time to package this up for the community until May, once we have got our first ou moodle release finished.
It is probably not a lot of work, but there are some issues to sort out, like where to put the various files.
- The testcases themselves we have put in folders called 'simpletest', with filenames like text_xxx.php, so you have a /lib/simpletest/testdatalib.php
- Then there are the simpletest libraries, which we will have to move into a subfolder of lib.
- Then there is the testing infrastructure, which basically crawls the directory tree (or a subset of it) looking for all the files called **/simpletest/test*.php, and runs the tests they contain (with various options) and displays the results prettily. At the moment we put this in a '_tools/runsimpletest' top level folder, but we may want to rethink that.
Have you received my email about the Selenium test? Sent it around two weeks ago, no rush at all! just checking
I had got the original email, just not had time to check it out till now, sorry.
After only a brief poke around I do have some preliminary questions:
- could the tests clean up after themselves automatically, by deleting any activities created during the test (and either testing that they've gone or using standard setup and teardown routines to ensure a consistent environment for each test)?
- to what extent do you think the cause of difficulties in testing is split between inherent problems with GUI testing / Selenium drawbacks / fixable Moodle HTML issues?
- what other functional testing tools are in the running for selection?
- what is the security issue with Selenium? Is it just to prevent people running scripts against test.moodle.com? Would it not be possible to run these tests against local installs of Moodle (if Selenium were set up on those servers too)?
- has any thought been given to running these tests directly from the MoodleDocs wiki? Sounds crazy I know, but there's a precedent in Fit and Fitnesse.
- have you been using the Selenium IDE or Remote Control?
I think it's looking very good, a comprehensive set of such tests would be an excellent resource for many reasons.