Testing and QA

Hi Tim,

Thanks for pointing to this interesting  test software. Just installed Selenium on my Firefox and tried your Selenium Tests in my localhost 1.7 Moodle install.

Things go well at first, but stop just before importing the quiz questions, see attached screen shot.

Joseph

Its been months with no progress on this matter in the forums, but here at the OU we've been looking further at the
possibility of using the Selenium apps to help with QA starting with the Quiz and Question production fronts.

I have been extending the work started by Tim and now is a good time to add to the discussions on Moodle testing
using Selenium and the possibility of integration with the SimpleTest frame work now used in Moodle.

For those like myself, who have come to this quite recently these discussions start here Moodle Selenium Test by Christian http://moodle.org/mod/forum/discuss.php?d=43569
and here Selinium for testing Moodle by Tim Hunt http://moodle.org/mod/forum/discuss.php?d=51161#234015

What i have found is that the current state of play on both of these applications appears to have produced some very high quality results which are being used by a number of communities, and that Moodle Developments
have been heading in the same direction for some time.

I think that the main reason Selenium integration with SimpleTest hasn't occured yet, has been because of the lack of compatibility between the two tools.

Selenium provides a Java Server as a wrapper for the pages returned by the Web Server so that actions can be played in the broswer.
SimpleTest provides a Unit Testing Framework, which is currently utilised within Moodle and other developments using PHP.

Until recently Selenium only supported Java, C#.NET, Perl, Python, and Ruby.
This is changing and third parties are producing some helpful tools to assist this.

This work is now approaching maturity, to the extent that I have been able use one of the third party beta test suite offerings to create Selenium Testcases and run them within my Moodle install using the SimpleTest framework.

I began at the Selenium site and worked through the Tutorials as Tim suggested previously, the easiest way being to start via the Selenium-IDE Firefox extension: http://www.openqa.org/selenium-ide/download.action

After which I moved on to the use of the Selenium Core and Remote Control applications.

Finding no information on either of the two main sites Selenium http://www.openqa.org and the SimpleTest framework http://www.lastcraft.com/simple_test.php to assist with the integration with Moodle, I did a Google search for SimpleTest + Selenium and found some third parties who have produced development and integration work in or for PHP use.

There is on going development on a PEAR Package available here
Package Information: Testing_Selenium http://pear.php.net/package/Testing_Selenium/redirected though I did not try this out myself.

I found a post at sitepoint.com/forums which lead to the xlab6 offering which I have used for my tests.

This comes as a simple folder structure with example test case ready to use
http://www.sitepoint.com/forums/showthread.php?t=308727&goto=nextnewest
The poster does say that this suite is experimental, but I have found it extremely functional, from the testing I have completed I would estimiate
more than 80% of the migration is complete, and the suite works with few problems which ill come on to later.

However though there currently appears to be no direct support for this PHP migration at OpenQA.Org for Selenium.
(I have just read one of the readme files and find that the Pear Package will be included in Selenium Remote Control in the near future)

I downloaded the suite referenced in the forum from here http://xlab6.com/selenium/selenium_simpletest.zip and added
the extracted files to my Moodle install under a new top level folder which i called Testing.

Basically this suite provides the Selenium Server installation on the testing Host and integration with a more up to date SimpleTest library
than the one currently in use in Moodle.
It unzips to create a folder selenium_simpletest which contains the selenium and simpletest library folders and an example testcase class file.
Docs and further test files are included in the respective application folders.

I began by creating test cases using Selenium IDE to record the individual tests and build a test suite as explained previously by Tim.

This process is fine for creating tests which can be run individually in the S.IDE,
they can then be bundled into a suite to run automatically in the Selenium TestRunner application.

The TestRunner is fine for running the testsuites manually if thats all thats required,
but we'd like to see the tests incorporated into Moodle in the long run, under the control of SimpleTest so we need to take the next step.


In order to use the existing tests, currently in 'Selenese HTML', we need to change them into PHP.
For this Selenium uses a Formatter, these are installed in the Selenium IDE and allow the HTML to be output to the
different languages which are supported.

There isnt currently a PHP formatter provided in the Selenium IDE download, or anywhere on their site.

So continuing the search I found a PHP formatter referenced here
http://cakebaker.42dh.com/2007/03/11/formats-for-the-selenium-ide/ the formatter code exists in the page
which is to be copied, and inserted into FireFox Selenium IDE.
The instructions are quite clear to follow and the formatter works.


Then returning to the Testing selenium_simpleTest folder, and after looking at the examples available as 'Selenese HTML' test cases,
and the example PHP testsuite class, I began to transfer my testcases into a PHP testsuite class.

This boiled down to taking the converted html tests and turning them in to functions in the testsuite class,
in the order required to be run automatically within the TestRunner.

I was then able to point my browser at my Moodle install with the Testing folder containing the Selenium and SimpleTest files,
and run my new testsuite, as a PHP testcase, in either IE or FireFox. (with S.IDE this is only possible in FireFox).
(Strangely? the Selenium Server takes the functions in the testcase class and inserts them into the Testrunner frame,
recreating the tests as 'Selenese HTML', however the testcase class is inside the SimpleTest framework allowing for the
addition of program logic not possible in the vanilla HTML tests)


The only problems to note as mentioned earlier seem to be due to the inaccesibility of certain controls on the page,
I havent yet been able to insert text into a HTML Text area inside the HTML Editor, or the Input file type.
This could be for a number of reasons, the HTML area doesnt seem to have a valid ID for Selenium to find on the page,
and though the HTML shows that it should be, the input file type doesnt seem to be accesible.



What we intend to do next is to be able to run any tests directly from an Admin Report page under the control of the SimpleTest framework,
rather than under manual control.

This will entail community support and decision making with regard to where the Selenium and SimpleTest libraries should be inserted
and how.

Whether we should wait for the release of a stable version of the Selenium Application suite which does contain support for PHP is an issue,
however that does not stop us creating tests which will be able to be incorporated.

I would suggest that the Moodle Exchange facility is used to provide a stock of tests created,
I will upload our test cases which I have extended, as both HTML and PHP, but which currently only test the Quiz module, as both Admin and a test user.

The tests have been designed to run taking into account some of the problems related to timing issues as a complete suite, or individualy.

Having looked at the resources there already, some new tests could be created against the Moodle Test Course provided for release testing.
As well as new tests which use the interactivity provided by Selenium IDE to setup features and remove them during the testing process.