Hi there, i just want give it a try to help with testing. I was looking in the tracker and for example i 'd like to start by writing a test for http://tracker.moodle.org/browse/MDLQA-16
How i'm supposed to do this functional test? Do i have to use Selenium?Is there any example on how for example login as a teacher??
I've read http://docs.moodle.org/en/Development:Unit_tests but i think this is not the way...
Thanks for any idea
Unit tests and functional tests are different things.
The aim of the functional tests is for testing Moodle 2.0 (and later releases) after we have got as far as a Moodle 2.0 beta release. We are not there yet, so for now you will have to wait. The idea is that these tests will be executed manually by volunteers.
At the moment, Helen Foster is co-ordinating the big job of making the list of all the functional tests that need to be done.
The aim of the functional tests is for testing Moodle 2.0 (and later releases) after we have got as far as a Moodle 2.0 beta release. We are not there yet, so for now you will have to wait. The idea is that these tests will be executed manually by volunteers.
At the moment, Helen Foster is co-ordinating the big job of making the list of all the functional tests that need to be done.
Question about this: "The idea is that these test will be executed manually by volunteers"
Why not script these with Selenium? Was it considered? Wouldn't be nice to have a repo of tests we could run patches against? I also think it would be a pretty easy way for people to contribute. After all, pretty much anyone can get the Selenium plugin and record a test.
I'd be willing to volunteer some time to plan out a naming schema for tests, and write up instructions if it would help.
Why not script these with Selenium? Was it considered? Wouldn't be nice to have a repo of tests we could run patches against? I also think it would be a pretty easy way for people to contribute. After all, pretty much anyone can get the Selenium plugin and record a test.
I'd be willing to volunteer some time to plan out a naming schema for tests, and write up instructions if it would help.
Hi Ezra,
The plan is to have unit tests and QA/functional tests to obtain the benefits of both types of testing.
Thanks for your offer of help. Please see the Czech Hackfest notes for details of our recent discussion regarding testing.
The plan is to have unit tests and QA/functional tests to obtain the benefits of both types of testing.
Thanks for your offer of help. Please see the Czech Hackfest notes for details of our recent discussion regarding testing.
Testing a web app as complex as Moodle with Selenium is hard. Several people, including me, have tried, but not really got anywhere. That is not to say that it is impossible, it is just that it will take a lot of persistence to get something that works reliably and we are not there yet.
I would be interested to hear what you found the main challenges to be if you have time to share. We're using it at work and I'm curious. Thanks!
Well, at one point I had a Selenium test that would Import a lot of quiz questions, add them to a quiz, then attempt the quiz getting some right and some wrong.
There were two main difficulties I had that are related.
1. The tests as recorded by Selenium were too specific. You needed to edit the recorded rules like Click on link ... actions to use a complex xPath expression to select the right link to click.
2. Even so, the tests were very fragile, and small irrelevant changes to Moodle would break the tests.
So anyway, I no longer use Selenium myself. I do use unit tests, and they help me a lot. I may be the nature of the type of code I write.
However, I still firmly believe that Selenium, or something like it, have great potential to automate a lot of Moodle testing - we just need the right person to get it working practically.
Actually, my colleague James is currently working on automating JUnit testing of Moodle. I think his stress is more on load testing. Anyway, it is work-in-progress. I keep nagging him to post about it here, but he would rather have something working to show first, and I don't want to steal his thunder.
There were two main difficulties I had that are related.
1. The tests as recorded by Selenium were too specific. You needed to edit the recorded rules like Click on link ... actions to use a complex xPath expression to select the right link to click.
2. Even so, the tests were very fragile, and small irrelevant changes to Moodle would break the tests.
So anyway, I no longer use Selenium myself. I do use unit tests, and they help me a lot. I may be the nature of the type of code I write.
However, I still firmly believe that Selenium, or something like it, have great potential to automate a lot of Moodle testing - we just need the right person to get it working practically.
Actually, my colleague James is currently working on automating JUnit testing of Moodle. I think his stress is more on load testing. Anyway, it is work-in-progress. I keep nagging him to post about it here, but he would rather have something working to show first, and I don't want to steal his thunder.
Thanks for all the responses. I'm pretty interested in contributing to testing moodle. I think both way of doing tests(unit and functional) are very important to improve and mantain moodle's quality.
Tim, you said "the tests were very fragile, and small irrelevant changes to Moodle would break the tests", but if we do "manual functional tests" it 'd be even more fragile.
We must find the way to automate functional tests.
I'm not sure about if selenium is the right tool but i 'll try to build some tests and give back to the community to get some feedback on determine which are the weak points of this aproach and discuss a bit more on how we can improve those tests.
You are doing a great work on moodle development and I'd like to start contributing to this great project.
I'm not very familiarized with the quiz module, do you have any idea about which test could be interesting to start doing some work?
Tim, you said "the tests were very fragile, and small irrelevant changes to Moodle would break the tests", but if we do "manual functional tests" it 'd be even more fragile.
We must find the way to automate functional tests.
I'm not sure about if selenium is the right tool but i 'll try to build some tests and give back to the community to get some feedback on determine which are the weak points of this aproach and discuss a bit more on how we can improve those tests.
You are doing a great work on moodle development and I'd like to start contributing to this great project.
I'm not very familiarized with the quiz module, do you have any idea about which test could be interesting to start doing some work?
I don't think you can ever eliminate humans testing. At the OU we have a testing team who test every release we make, which is great. As Martin says, with the way the Moodle community is, I am sure we will get volunteer testers. Humans have common sense, which automated tests don't.
So automated testing should compliment what the human testers do, and allow them to focus there attention where it does the most good. However, will never replace them. Still, there would be a lot of benefit from automating some of the basic stuff.
I suggest a good place to start is to simulate something like a user logging in, viewing a course, then posting to a forum. Later, we can think about adding more parts of Moodle a bit at a time.
Please keep thinking about it. It would be great to get something working.
So automated testing should compliment what the human testers do, and allow them to focus there attention where it does the most good. However, will never replace them. Still, there would be a lot of benefit from automating some of the basic stuff.
I suggest a good place to start is to simulate something like a user logging in, viewing a course, then posting to a forum. Later, we can think about adding more parts of Moodle a bit at a time.
Please keep thinking about it. It would be great to get something working.
I'm in agreement with most of what's been said here. The fragility of the test is something we're trying to address at our office. I'm keenly interested in improving this for our customized moodle app.
We're exporting the tests into PHPUnit -- this gives us the ability to control the fixures a bit more and that helps (we're still learning), but there's not a lot of support for PHPUnit and Selenium. The community seems to focus more on other languages.
We already have a number of suites that do things like create user, check roles, create quizzes and surveys and such.
I've set up our testing framework using Hudson, Selenium Grid and EC2 slaves.
What if we did something like set up a Hudson server and allowed the community to register slaves that could run tests. I'm part of the Drupal community as well, and they have a pretty sophisticated model for this (it's all custom though), but clients are allowed to connected to the master and get tests.
We could put the selenium tests in a repo and then Hudson could do an update and run the suites on the client slaves at a schedule that works for the developers.
I'd imagine we'd want to have a number of mysql dumped databases, and exported courses as fixures -- preloaded with quizzes and users and such, but we could also create them using Selenium.
Gotta run to a meeting, but I'll try and post something more formal over the holiday.
We're exporting the tests into PHPUnit -- this gives us the ability to control the fixures a bit more and that helps (we're still learning), but there's not a lot of support for PHPUnit and Selenium. The community seems to focus more on other languages.
We already have a number of suites that do things like create user, check roles, create quizzes and surveys and such.
I've set up our testing framework using Hudson, Selenium Grid and EC2 slaves.
What if we did something like set up a Hudson server and allowed the community to register slaves that could run tests. I'm part of the Drupal community as well, and they have a pretty sophisticated model for this (it's all custom though), but clients are allowed to connected to the master and get tests.
We could put the selenium tests in a repo and then Hudson could do an update and run the suites on the client slaves at a schedule that works for the developers.
I'd imagine we'd want to have a number of mysql dumped databases, and exported courses as fixures -- preloaded with quizzes and users and such, but we could also create them using Selenium.
Gotta run to a meeting, but I'll try and post something more formal over the holiday.
I gave this some more thought and planning out a standardized course fixture is going to be key if we are distributing test writing. It would greatly simplify things to have a standard course to use for test writing/running.
Looking at the Forum and Database tests in the tracker, for example, the assumption is that a course with a forum and database would already exist (as well as the users). Distributing the test writing is going to be a lot easier if we all have the same course ID and the course has all the same objects and those objects all have the same ID.
Is there a standardized course the developers are using or something similar that we could import to ensure consistency? If not, creating one will probably be necessary. In that case maybe we want to make creating the course and objects part of the tests. It would probably be easier to get up and writing tests if we can import a course, rather than create one.
If we are using a standardized course that will be imported, that course should probably be versioned and stored in source control so we can track changes and update the tests as necessary. Does anything like this exist?
Test naming: Just use the tracker ids. MDLQA-18.html, etc.
Saving tests: We'll probably want to save them in HTML format so others can import, modify and then export if they need to be modified later.
Where to store the test? Moodle CVS repository? Attach to the tracker ticket as well? To automate running the tests, we'll definately need them in a repo somewhere. Can this be made available or should we use something external.
What language to export the tests for Selenium-RC? PHPUnit seems to make sense to me, but I'd be interested in hearing what others think.
Users and passwords: For the tests to work we'll have to have working usernames and passwords. Since we can pass in username/password variables to the tests, I think we'd want to let each person running the tests set those up so we don't have to worry about the security of shared usernames and passwords.
The steps to running a test environment would be something like this.
1. On the Moodle environment to be tested, install Moodle from CVS checkout
2. Create a teacher and student account (and/or whatever else is required)
3. Import the course. (At this point we have a "clean" ready-to-test environment. It would be useful to dump this db, and reimport it at the start of every test cycle.)
4. On your Selenium-RC environment checkout the tests from CVS. Update the suites with the AUT URL, and username/passwords of your accounts being used in testing.
Obviously the idea is to make the output visible to developers. Hudson can do a lot of things to help with this - email, RSS, IRC notification, etc. I am only familiar with Hudson, but I know there are a lot of other ways to do this as well.
The tests could run once or twice a day and communicate results to the developer community.
The steps to writing a test would be something like this.
1. Set up an environment as noted above.
2. Browse the tracker for an issue without a test. If there are any that can't be automated, make a note of it to save time for other testers.
3. Script the test and make sure to do any clean up of data created after the script is complete and saved. Be sure to remove any usernames/passwords. Export the test to PHPUnit (if that's what we use). Post both files to the tracker for review and eventual commit by someone with access.
I guess there still is the question of value. I agree, this wouldn't be a very good use of developer time. But it also doesn't make sense to do this -- even if non-developers are doing it -- if it's not of any use to the developer community. Since I'm not a developer I can't say, but seems like there is some interest. Would the time/effort be better spent just manually testing?
I'm know this kind of scripting can be done in a stable environment. I think the question is going to be can it be done by a distributed group of volunteer testers and still be of use. Do we make the AUT so fixed that it no longer represents a realistic environment? Maybe that can be addressed by having several different types of courses as fixtures?
Interested to hear if you all think something like this is worth pursuing.
Looking at the Forum and Database tests in the tracker, for example, the assumption is that a course with a forum and database would already exist (as well as the users). Distributing the test writing is going to be a lot easier if we all have the same course ID and the course has all the same objects and those objects all have the same ID.
Is there a standardized course the developers are using or something similar that we could import to ensure consistency? If not, creating one will probably be necessary. In that case maybe we want to make creating the course and objects part of the tests. It would probably be easier to get up and writing tests if we can import a course, rather than create one.
If we are using a standardized course that will be imported, that course should probably be versioned and stored in source control so we can track changes and update the tests as necessary. Does anything like this exist?
Test naming: Just use the tracker ids. MDLQA-18.html, etc.
Saving tests: We'll probably want to save them in HTML format so others can import, modify and then export if they need to be modified later.
Where to store the test? Moodle CVS repository? Attach to the tracker ticket as well? To automate running the tests, we'll definately need them in a repo somewhere. Can this be made available or should we use something external.
What language to export the tests for Selenium-RC? PHPUnit seems to make sense to me, but I'd be interested in hearing what others think.
Users and passwords: For the tests to work we'll have to have working usernames and passwords. Since we can pass in username/password variables to the tests, I think we'd want to let each person running the tests set those up so we don't have to worry about the security of shared usernames and passwords.
The steps to running a test environment would be something like this.
1. On the Moodle environment to be tested, install Moodle from CVS checkout
2. Create a teacher and student account (and/or whatever else is required)
3. Import the course. (At this point we have a "clean" ready-to-test environment. It would be useful to dump this db, and reimport it at the start of every test cycle.)
4. On your Selenium-RC environment checkout the tests from CVS. Update the suites with the AUT URL, and username/passwords of your accounts being used in testing.
Obviously the idea is to make the output visible to developers. Hudson can do a lot of things to help with this - email, RSS, IRC notification, etc. I am only familiar with Hudson, but I know there are a lot of other ways to do this as well.
The tests could run once or twice a day and communicate results to the developer community.
The steps to writing a test would be something like this.
1. Set up an environment as noted above.
2. Browse the tracker for an issue without a test. If there are any that can't be automated, make a note of it to save time for other testers.
3. Script the test and make sure to do any clean up of data created after the script is complete and saved. Be sure to remove any usernames/passwords. Export the test to PHPUnit (if that's what we use). Post both files to the tracker for review and eventual commit by someone with access.
I guess there still is the question of value. I agree, this wouldn't be a very good use of developer time. But it also doesn't make sense to do this -- even if non-developers are doing it -- if it's not of any use to the developer community. Since I'm not a developer I can't say, but seems like there is some interest. Would the time/effort be better spent just manually testing?
I'm know this kind of scripting can be done in a stable environment. I think the question is going to be can it be done by a distributed group of volunteer testers and still be of use. Do we make the AUT so fixed that it no longer represents a realistic environment? Maybe that can be addressed by having several different types of courses as fixtures?
Interested to hear if you all think something like this is worth pursuing.
As Tim said, we did try Selenium ages ago and we found such tests pretty fragile in Moodle because of all the different settings we have. We'll end up spending a lot of precious developer time creating/maintaining a test framework.
This is one thing the non-developer community can easily help with, so why not let them help?
Human testers can spot things that are not directly in the actual test, they can also link to other issues in the tracker if necessary, come up with better interface ideas, communicate directly with developers about issues, and generally help us in all kinds of ways Selenium alone can not.
Of course I'd be very happy if someone is offering to build and maintain a comprehensive Selenium test suite, it would be useful, but I don't think it's a good use of core developer resources right now, that's all.
This is one thing the non-developer community can easily help with, so why not let them help?
Human testers can spot things that are not directly in the actual test, they can also link to other issues in the tracker if necessary, come up with better interface ideas, communicate directly with developers about issues, and generally help us in all kinds of ways Selenium alone can not.
Of course I'd be very happy if someone is offering to build and maintain a comprehensive Selenium test suite, it would be useful, but I don't think it's a good use of core developer resources right now, that's all.