Testing and QA

I gave this some more thought and planning out a standardized course fixture is going to be key if we are distributing test writing. It would greatly simplify things to have a standard course to use for test writing/running.

Looking at the Forum and Database tests in the tracker, for example, the assumption is that a course with a forum and database would already exist (as well as the users). Distributing the test writing is going to be a lot easier if we all have the same course ID and the course has all the same objects and those objects all have the same ID.

Is there a standardized course the developers are using or something similar that we could import to ensure consistency? If not, creating one will probably be necessary. In that case maybe we want to make creating the course and objects part of the tests. It would probably be easier to get up and writing tests if we can import a course, rather than create one.

If we are using a standardized course that will be imported, that course should probably be versioned and stored in source control so we can track changes and update the tests as necessary. Does anything like this exist?

Test naming: Just use the tracker ids. MDLQA-18.html, etc.

Saving tests: We'll probably want to save them in HTML format so others can import, modify and then export if they need to be modified later.

Where to store the test? Moodle CVS repository? Attach to the tracker ticket as well? To automate running the tests, we'll definately need them in a repo somewhere. Can this be made available or should we use something external.

What language to export the tests for Selenium-RC? PHPUnit seems to make sense to me, but I'd be interested in hearing what others think.

Users and passwords: For the tests to work we'll have to have working usernames and passwords. Since we can pass in username/password variables to the tests, I think we'd want to let each person running the tests set those up so we don't have to worry about the security of shared usernames and passwords.

The steps to running a test environment would be something like this.

1. On the Moodle environment to be tested, install Moodle from CVS checkout
2. Create a teacher and student account (and/or whatever else is required)
3. Import the course. (At this point we have a "clean" ready-to-test environment. It would be useful to dump this db, and reimport it at the start of every test cycle.)
4. On your Selenium-RC environment checkout the tests from CVS. Update the suites with the AUT URL, and username/passwords of your accounts being used in testing.

Obviously the idea is to make the output visible to developers. Hudson can do a lot of things to help with this - email, RSS, IRC notification, etc. I am only familiar with Hudson, but I know there are a lot of other ways to do this as well.

The tests could run once or twice a day and communicate results to the developer community.

The steps to writing a test would be something like this.

1. Set up an environment as noted above.
2. Browse the tracker for an issue without a test. If there are any that can't be automated, make a note of it to save time for other testers.
3. Script the test and make sure to do any clean up of data created after the script is complete and saved. Be sure to remove any usernames/passwords. Export the test to PHPUnit (if that's what we use). Post both files to the tracker for review and eventual commit by someone with access.

I guess there still is the question of value. I agree, this wouldn't be a very good use of developer time. But it also doesn't make sense to do this -- even if non-developers are doing it -- if it's not of any use to the developer community. Since I'm not a developer I can't say, but seems like there is some interest. Would the time/effort be better spent just manually testing?

I'm know this kind of scripting can be done in a stable environment. I think the question is going to be can it be done by a distributed group of volunteer testers and still be of use. Do we make the AUT so fixed that it no longer represents a realistic environment? Maybe that can be addressed by having several different types of courses as fixtures?

Interested to hear if you all think something like this is worth pursuing.