You may be aware of a point of contention about our recommendation for only a single Given/When/Then in behat feature files following this guidance:
http://docs.behat.org/en/latest/guides/1.gherkin.html
I have started a policy issue about this issue: MDL-48186
Feel free to comment here or in the issue!
I'd be really interested to hear from anyone who is using this kind of test specification in other environments - or knows examples from other open source projects with the size and complexity we have?
I have been thinking about this recently, and while in general I can see that it is good to test one thing per scenario (so then, if something fails, you know exactly what failed, and can re-run just that specific thing when you think you have fixed it) I don't believe it is acutally possible in practice.
The example I came across recently where I thought it was not reasonable to do just one thing per scenario was testing questoin preview: https://github.com/moodle/moodle/blob/234c5337f7b562465568458422a5de5a7146e8b7/question/tests/behat/preview_question.feature
When previewing a question some things can naturally only be done after other things. If you believe it is possible to break up that feature file into separate scenarios that only test one thing each, then please show me how by rewriting that file.
Note that that feature file would be much more readable if each block for lines from line 40 onwards would be easier to read if it started with another When, and switched back to Then at the appropriate point.
Just to say that Rajesh replied in the tracker issue, taking up my challenge. His re-working of my script was pretty good, and having seen it I realised that I could then do even better.
So, as part of MDL-48373 I found time to covert all the question Behat tests to the approved style. The real point of this post is to report on what I found.
What I was really doing was creating proper generators for questions, so you can say And the following "questions" exist: in the set-up for your scenario. This should make things faster than the old way of creating things through the Moodle UI.
On the other hand, splitting the existing long rambling tests into separate ones makes things slower, becuase we reset and repeat the initialisation more often.
Also, while working on these scripts I notices a few things were missed out in the testing, and added them.
So, overall, the run-time (for --tags core_question) has gone from
9 scenarios (9 passed)
288 steps (288 passed)
4m54.203s
to
28 scenarios (28 passed)
462 steps (462 passed)
7m5.508s
So, from 5 mins to 7 mins. . On the other hand, when something failes, you can just re-run that one specific test which is much shorter
. Overall, I think the new tests are better.
So, I am starting to believe that 'Test one things per scenario' is not only desirable, but also possible.
However, I still think that if you are testing more than one things per scenario, you should be allowed to use more than one When and Then because it is much easier to read.
However, I still think that if you are testing more than one things per scenario, you should be allowed to use more than one When and Then because it is much easier to read.
Indeed. But even testing one thing may require more than one Then/When. For instance, if I need to test a feature of Group submission that allows group members to work on the same entry, I need multiple logins in the same scenario. Multiple logins invite multiple "Then" (I don't use "When" at all; one opening Given and then Thens and Ands).
To make the scenario more readable I also add a substantial amount of inline comments. I even use code folding for which I define special "comments" such as #section: to open and #:section to close.
What you say shows that you have not yet got the right mind-set. What you describe is a number of different scenarios:
(To make this feasible, you need set-up steps, so you can can quickly set up the starting situation for the second and later scenarios.)
Whether what I describe is one scenario or a composite of several scenarios is merely terminological and is really not the issue. Many scenarios can be steps in bigger scenarios and at least some of our current core steps are such. You seem to suggest that for many Gherkin scenarios we need to write their php counterpart only to make complex scenarios look simple. That shouldn't be our preferred approach because it may be counter productive. And don't get me wrong. I'm already adding many "set up" steps to make the tests more efficient. But in general it is not the way. Gherkin may at some point allow re-use of scenarios within scenarios and at any rate it shouldn't be too difficult to add a pseudo step that would do that. The point remains that for some scenarios it may not be worth it to reproduce Gherkin in php only to maintain the Single Given-When-Then convention.
To be sure, I don't really have a problem with this styling convention. I can work around it and continue to comply with Moodle coding guidelines just like I worked around the weird requirement, in my opinion, to put a full stop at the end of an inline comment even when the inline comment is meant as a label rather than a descriptive statement. I enhanced the code checker with an option to fix that automatically and I don't have to think about it anymore. I can do the same with respect to the Given-When-Then in Gherkin.
Hello Itamar,
As you know behat doesn't technically distinguish between these three kind of steps, so you can use what best suite your need.
In Moodle core, it's good that we are sticking to single GIVEN-WHEN-THEN, as this try to enforce that single scenario should not be testing more then what is described in scenario outline. Also, force developer to write multiple scenarios covering specific functionality and not miss coverage.
If you think you need multiple WHEN then probably you need to re-visit your scenario and see if you are trying to test more then what you were expecting in scenario. We know that we have a drawback of maintaining more code and increase run time, but advantage is the coverage report. If you are testing enrol_user then what all scenarios are covered is clearly visible from scenario list and you don't have to read behat feature file to understand it.
FYI: Robert C. Martin has written a great post about BDD's Given-When-Then concept where he thinks of them as a finite state machine.
Also, behat 3 doesn't support step chaining and sub-context, making it hard to have a pseudo-step.
https://github.com/Behat/Behat/issues/546
https://github.com/Behat/Behat/issues/558#issuecomment-65213657
Thanks Rajesh. I think that Moodle core perspective, as you posit it here, actually misses something in the point of BDD reflecting the requirements analysis process/language. Developers see discrete functions/features as singular scenarios. Subject matter experts see wholes of complext situations. An obvious shortcoming of the the various blog posts is that they give simple examples that are easy to fit into the over-simplification of the Single Given-When-Then model. Some of the replies to the posts point that out. So, the proposed convention may serve a purpose in the context of Moodle core development, but this doesn't mean that it would be as effective in other contexts of Moodle development. But at the end of the day, all this can be merely academic. If you plan to make the code checker generate errors/warnings about this convention, please add an option to ignore this convention so that it doesn't interfere with plugins' compliance with Moodle coding guidelines.
I've been writing a number of Behat tests recently for projects I've been working on.
I've found very few situations in which a single given..when..then would make sense.
For example, I've done a lot around modifications to the facetoface module within Totara - that often involves, for example, a teacher logging in and setting up a session with dates, then a student logging in and signing up to a session, then the teacher logging in and checking that the student appears correctly, then cancelling the student sign up, then the student logging in to check they have been cancelled.
Ideally that would be 5 tests:
However, to split those into 5 tests, each of those tests would have to include all the steps from the previous tests in the 'given' section (aside from a couple of lines at the end that check the correct output is shown on the screen). To my mind that makes the file harder to read (as there are large amounts of repeated statements to wade through) and will certainly make the already slow Behat tests run even more slowly (which is an important practical consideration).
Whilst I sympathise with the ideal (maybe with expanded use of generators, or in areas with less complex workflows it might be achievable), I'd much prefer the readability of multiple When ... Then ... sections (especially if there are comments before each section to introduce what is being tested in that part).
Davo, I believe the 'correct' solution to that scenario is to extend the generators, so you for scenario 2. you can start efficiently by saying:
Given the following 'activity' exists | facetoface | ... | And the following 'face-to-face' sessions exist ...
However, that is a bit more work.
The pay-back would be when your behat test is failing at point 5.
With one long scenario, you have a long wait every time you run the test, so see if you have now fixed it.
With separate scenarios, you can just run step 5.
Hi all,
I already commented in https://tracker.moodle.org/browse/MDL-48186, just linking basic BDD documentation (http://dannorth.net/introducing-bdd/) as the Given / When / Then is not a behat-thing but a BDD base.
I was reading everzet's (Behat creator) blog and though of sharing it.
http://everzet.com/post/99045129766/introducing-modelling-by-example
