Hi,
i have question. In developing proces before new release (minor nad major) Moodle HQ test all behat in new version. For example in Moodle 3.6 all behat test in the clean instalation should be passed? In Moodle 3.6.2 alle behat test in the clean instalation should be passed?
I have installed clean 3.6 an most scenarios have been skipped (3464). Some tests have been failed (19).
Moodle HQ run all Behat tests each day, and they will now make a release unless all tests pass.
Skipped tests normally relate to features that relate to feature that require a particular thing to be set up. For example the On-screen assignment grading tests can only run if you have unoconv set up on your server. If you don't have that, those tests will be skipped. So, that is probably not a worry.
The failed tests are more interesting. You really need to investigate each failuer and see what went wrong. Our experience is that tests sometimes just fail randomly, so if you are running all the tests, you should use the re-run support (https://docs.moodle.org/dev/Running_acceptance_test#Rerun_failed_scenarios). Also, just differnet environments can cause different tests to fail. That can be worth fixing, because it basically makes Moodle more robust.