See https://tracker.moodle.org/browse/MDL-46891. There are a few architectural changes in 3.0 which have made the transition hard for us. Quoting Raj's comments there:
It's too late for this to be in 2.9 release and Behat 3.0 has few blockers like:
https://github.com/Behat/Behat/pull/656
https://github.com/Behat/Behat/issues/653
https://github.com/Behat/Behat/issues/688
https://github.com/Behat/Behat/issues/714
It will be nice to wait for Behat 3.1 https://github.com/Behat/Behat/issues/707
In the mean while we need to decide future of supporting Step chaining, current patch do support it, but if Behat is not supporting it, then we need to seriously think about maintaining that code and compatibility.
Thanks for starting this discussion,
I worked on MDL-46891 last month and Behat 3.x is missing vital features which are needed for CI, Like re-run is not working, support for reports (Junit). Also, with re-writing of testworker in 3.1, we have to modify moodle-behat-extension. So IMHO, Behat 3.0 is not yet ready for us to be used for CI.
Behat 3.1rc1 got released and Behat 3.1 is on-it's way. Main issues because of which Moodle delayed upgrading to Behat 3.x are still not fixed but seems it high time to consider an upgrade, else it will be more harder to upgrade.
After upgrading to Behat 3.x we also should embrace behat philosophy of not supporting step chaining. So proposal is to consider getting MDL-46891 ready for Moodle 3.1 (which support step chaining) and create a follow-up issue to remove support of step chaining by Moodle 3.3
Can you explain in more detail about this step chaining issue?
Step chaining allows dev to call steps from a step. For example if you want to login as user then you have to execute following steps
- You have to go to Moodle login page
- Enter username
- Enter password
- Click on login
With step chaining you can write a single step "I login as user "student1"" and this step internally call above 4 steps.
In Behat 3.x Step chaning concept has been removed, and suggestible way call api's and not steps. That leads to the changes like this.
Current proposed Behat 3.x support chained steps via moodle-behat-extension but they are only provided for easy upgrade to 3rd party dev's. It will eventually be removed in later version of moodle-behat-extension.
I think we should just stop using step chaining in Moodle.
Most step chaining leads to 'bad' tests. E.g. tests that use steps like
When I add a 'forum' filling in the form with: [A]
Instead of
Given the following 'activities' exits: [B]
(I say 'bad' test because, when you are writing all the many tests of the forum functionality like posting and replying, then you should not be re-testing adding a forum through the UI. There should only be one test of adding a forum. Therefore, [A] should only appear in one test, and [B] (which is implemented without step chaining) should be used a lot. As an added bonus, [B] is massively faster.
Since Raj is the person who preached at me about single Given-When-Then tests until I became a total convert, I can't believe he is not behind the idea of getting rid of step chaining in Moodle. Behat 3 is making that recommendation for a reason.
I agree with Tim Hunt that Moodle should stop using step chaining as soon as possible. Step chaining increases test run time so much that I find myself completely avoiding Behat in Moodle unless I feel absolutely compelled to use it.
Since Raj is the person who preached at me about single Given-When-Then tests until I became a total convert, I can't believe he is not behind the idea of getting rid of step chaining in Moodle.
There is a large gulf between recommending best practice and breaking compatibility for our community.
If you realise this impacts on your own tests you might come round to that viewpoint
Shame. I was hoping to start using version 3 as I switched from v2 to v3 and now I will need to hack it in.. Thanks for the answers guys!
Hello Mariusz,
Which project have you switched from v2 to v3 ?
Didn't you find those issues with your project ?
MDL-46891 has a working patch, if you want to use.
Sorry Rajesh. I wasn't accurate enough. By v2 and v3 I mean Behat. I am working with different PHP frameworks and cmses and recently I am involved in some Moodle customisation. I was running through difficulties to get features running on Behat through Moodle extension. There's far too many defects on the tracker to say that I am a big fan of how Moodle behaves...
Moodle-Behat-Extension has been updated to use Behat 3.0.15 (MDL-46891)
We tried to keep BC as much as possible, but there are few cases which were not BC, please refer https://docs.moodle.org/dev/User_talk:Rajesh_Taneja#Migrating_from_Behat_2.5_to_3.x_in_Moodle for more information.