Cannot run Behat tests on Windows

Cannot run Behat tests on Windows

by Joseph Rézeau -
Number of replies: 19
Picture of Core developers Picture of Plugin developers Picture of Testers Picture of Translators

Hello to all Behat fellow users/sufferers!
This is my 3rd attempt at using Behat on my local Windows Moodle site. First time was a few years ago and I gave up. Second attempt was last year and I finally managed to get it going.
I am trying again, scrupulously following the instructions at Running acceptance tests 
but I am stumped!
Using Windows 11. PHP 8.2.13. WampServer 3.3.3. Moodle 4.3.3+ (Build: 20240215)
1.- I am getting a conflict between geckodriver and selenium-server. When I launch them one after the other, they complain that they cannot use the same port 4444. So I have launched selenium with the -port 4000 param, and it seemss to satisfy both.
It says: INFO [SeleniumServer.boot] - Selenium Server is up and running on port 4000
and geckodriver INFO Listening on 127.0.0.1:4444
2.- I have set up everything as expected in my Moodle config.php file.
3.- When I run php admin/tool/behat/cli/init.php in my local moodle test site, it does create all the behat tables in the database, etc. and it says:
Creating Behat configuration ... done in 8.79 seconds.
Building theme CSS for boost [ltr] ... done in 9.63 seconds.
Building theme CSS for classic [ltr] ... done in 9.72 seconds.
Testing environment themes built
Acceptance tests environment enabled on http://127.0.0.1, to run the tests use:
vendor\bin\behat --config C:\wamp64\www\moodledatatest_behat\behatrun\behat\behat.yml
4.- But when I try vendor\bin\behat --config C:\wamp64\www\moodledatatest_behat\behatrun\behat\behat.yml
nothing happens! no error messages, nothing, after a couple of seconds I am getting the windows prompt back.

Any help welcome...

Average of ratings: -
In reply to Joseph Rézeau

Re: Cannot run Behat tests on Windows

by AL Rachels -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers
Last year, about the time I saw you start making posts about Behat, I got Behat running on my Windows desktop computer. While it worked, it was extremely frustrating due to being SO SLOW. It was also inconsistent about being able to see the test steps in progress. I was seeing the same as you with step 4...I just ignored it for a LONG while and finally it worked.

My desktop computer is going on ten years old, and is running at 3.2GHz, but it is still slow at this due to all the things having to run to get Behat results.

I bought an Orange Pi5 and got Behat to work on it right away using Ubuntu 22.04. Then I screwed it up by trying to change something...cannot remember what. In between working on my main projects, I keep trying to get it back. I am about to the point of wiping the memory chip and starting over.
Average of ratings: -
In reply to Joseph Rézeau

Re: Cannot run Behat tests on Windows

by Renaat Debleu -
Picture of Core developers Picture of Plugin developers
Your Windows is slow, In Linux I see much shorter times:  0.68 seconds, 0.41 seconds, 0.38 seconds, ....

But if you do not see anything, you can:
  • add -vv to your command to have extra output: vendor\bin\behat -vv --config C:\wamp64\www\moodledatatest_behat\behatrun\behat\behat.yml
  • try a dry run:  vendor\bin\behat --config C:\wamp64\www\moodledatatest_behat\behatrun\behat\behat.yml --dry-run --format moodle_stepcount.
  • because your system is slow, add $CFG->behat_increasetimeout = 9 to the config.php file
  • try first running behat tests without javascript (--tags="~@javascript"). These tests do not need Selenium and are fast.
  • disconnect yourself from the world, disable your firewall and antivirus, and try again

Average of ratings: Useful (1)
In reply to Renaat Debleu

Re: Cannot run Behat tests on Windows

by Joseph Rézeau -
Picture of Core developers Picture of Plugin developers Picture of Testers Picture of Translators
@Renaat,
Thanks for trying to help. I do know that LInux would be faster than Windows but I want to stick with Windows... and I did manage to run behat tests last year, although being slow. I have tried all your suggestions but nothing works (except the dry run option).
So I'm still stuck.
Average of ratings: -
In reply to Joseph Rézeau

Re: Cannot run Behat tests on Windows

by Renaat Debleu -
Picture of Core developers Picture of Plugin developers
The output with the dry run option proves that your Behat configuration is correct.

Now it is time to dig into the log files:
  • The web server access log should have at least 1 line containing "GET /admin/tool/behat/tests/behat/fixtures/environment.php" returning 200 as status code.  This is the first file accessed by Behat to see if the configuration is correct.
  • The web server error log should contain no errors
If the logs are empty, the problem has to be related to Selenium: open Postman or something similar and make a POST request to your running server like this:

POST request http://127.0.0.1:4000/session
Body of your POST request{
  “capabilities”: {
    “browserName”: “firefox”
  }
}
Once you make the POST request, you should see Firefox open up. If nothing happens, try running Selenium with the default port 4444.
Average of ratings: -
In reply to Renaat Debleu

Re: Cannot run Behat tests on Windows

by Joseph Rézeau -
Picture of Core developers Picture of Plugin developers Picture of Testers Picture of Translators

Thanks again for your help, Renaat!

If I add $CFG->behat_dataroot_parent  = 'C:\\wamp64\www\moodledatatest_behat'; to my original Moodle test site config.php, when running vendor\bin\behat --config C:\wamp64\www\moodledatatest_behat\behatrun\behat\behat.yml

I get this error in my php_error.log:

[07-Mar-2024 17:13:04 Europe/London] Behat only can run if test mode is enabled. 

and in my access.log I get these lines:

127.0.0.1 - - [07/Mar/2024:18:17:50 +0100] "GET /moodle/admin/cron.php HTTP/1.1" 200 6462
127.0.0.1 - - [07/Mar/2024:18:17:51 +0100] "GET /moodlemaster/admin/cron.php HTTP/1.1" 200 6460
127.0.0.1 - - [07/Mar/2024:18:17:52 +0100] "GET /moodletest/admin/cron.php HTTP/1.1" 200 15702
127.0.0.1 - - [07/Mar/2024:18:17:59 +0100] "GET /admin/tool/behat/tests/behat/fixtures/environment.php HTTP/1.1" 404 316

Average of ratings: -
In reply to Joseph Rézeau

Re: Cannot run Behat tests on Windows

by Renaat Debleu -
Picture of Core developers Picture of Plugin developers
The first 3 lines return a successful 200 code, the last line a 404 error (Not found)
So the last line reveals that your $CFG->behat_wwwroot is incorrect, you should add '/moodle' or '/moodlemaster' or '/moodletests', depending on where you run behat.
Average of ratings: -
In reply to Renaat Debleu

Re: Cannot run Behat tests on Windows

by Joseph Rézeau -
Picture of Core developers Picture of Plugin developers Picture of Testers Picture of Translators

Making slow progress, but still no joy. I have re-started behat and used a (local) "moodle" test site instead of "moodletest" etc.

When running: php admin/tool/behat/cli/init.php

I get:

Testing environment themes built
Acceptance tests environment enabled on http://127.0.0.1/moodle, to run the tests use:
vendor\bin\behat --config C:\wamp64\www\moodledata_behat\behatrun\behat\behat.yml

when I run C:\wamp64\www\moodle> vendor\bin\behat --config C:\wamp64\www\moodledata_behat\behatrun\behat\behat.yml

I get:
Behat requirement not satisfied: http://127.0.0.1/moodle is not available, ensure you specified correct url and that the server is set up and started.

My php_error.log shows zero errors

and in my access.log I get these lines:

127.0.0.1 - - [09/Mar/2024:16:43:00 +0100] "GET /moodle/admin/tool/behat/tests/behat/fixtures/environment.php HTTP/1.1" 303 1487
127.0.0.1 - - [09/Mar/2024:16:43:13 +0100] "GET /moodle/admin/cron.php HTTP/1.1" 200 6456

But if I type either http://localhost/moodle/ or http://127.0.0.1/moodle in my localhost URL I do arrive on my local moodle site.

Any more things I can try before definitely giving up Behat?

Average of ratings: -
In reply to Joseph Rézeau

Re: Cannot run Behat tests on Windows

by Renaat Debleu -
Picture of Core developers Picture of Plugin developers
According to MDL-80972, you can have better info than the simple error message "Behat requirement not satisfied".
Edit the file lib/behat/classes/util.php and add ". $result . PHP_EOL" to line 229.  You should receive extra info why it does not work.

Of course, do not give up, you're almost there.

P.S. You can always vote for MDL-80972 to have it implemented faster.
Average of ratings: -
In reply to Joseph Rézeau

Re: Cannot run Behat tests on Windows

by Joseph Rézeau -
Picture of Core developers Picture of Plugin developers Picture of Testers Picture of Translators

One more attempt... Re-installed everything, now I do not make progress, in fact I regress...

Using Wampserver 3.3.3 running PHP 8.2.13 Moodle 4.3.3+ (Build: 20240308).

Wamp settings: Allow links on projects homepage.

I install Moodle in a moodletest dir. This is my config extract:

$CFG->wwwroot   = 'http://localhost/moodletest';
$CFG->dataroot  = 'C:\\wamp64\\moodledatatest';
$CFG->admin     = 'admin';

$CFG->behat_dataroot  = "C:\wamp64\www\moodledata_behat";
$CFG->behat_wwwroot = 'http://127.0.0.1/moodletest';
$CFG->behat_prefix = 'zbehat_';
require_once('C:\behat3\moodle-browser-config\init.php');

In "C:\wamp64\www\moodletest" I run php admin/tool/behat/cli/init.php

Result: Error: Composer dependencies are not installed. Ensure you set $CFG->behat_* vars in config.php and you ran admin/tool/behat/cli/init.php.

If I go to : http://127.0.0.1/moodletest/

I get: Behat config error: Behat is configured but not enabled on this test site. 

😒😒😒

Average of ratings: -
In reply to Joseph Rézeau

Re: Cannot run Behat tests on Windows

by AL Rachels -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Plugin developers Picture of Testers
Sorry I did not notice this earlier, but I think you need to escape the backslashes in your $CFG->behat_dataroot = "C:\wamp64\www\moodledata_behat"; like the regular $CFG->dataroot = 'C:\\wamp64\\moodledatatest';

I think maybe it should be:
$CFG->behat_dataroot = "C:\\wamp64\\www\\moodledata_behat";

I also do not understand why you are placing the behat data root in the www folder. Shouldn't it be like the normal data root for the moodle?
$CFG->behat_dataroot = "C:\\wamp64\\moodledata_behat";
Average of ratings: -
In reply to AL Rachels

Re: Cannot run Behat tests on Windows

by Joseph Rézeau -
Picture of Core developers Picture of Plugin developers Picture of Testers Picture of Translators
Thanks for trying to help Al. I've tried your suggestions, still no joy!
Average of ratings: -
In reply to Joseph Rézeau

Re: Cannot run Behat tests on Windows

by Renaat Debleu -
Picture of Core developers Picture of Plugin developers
The moodle-browser-config plugin is Linux only, and needs the escaped backslashes. require_once('C:\\behat3\\moodle-browser-config\\init.php');

Or you could make the plugin Windows compatible by changing all paths in the plugin to a windows version (difficult but you will be rewarded if you share your code for other Windows users), or you can replace the moodle-browser-config with a manual configuration ($CFG->behat_profiles https://moodledev.io/general/development/tools/behat/running#manually-configuring-other-browsers)
Average of ratings: -
In reply to Joseph Rézeau

Re: Cannot run Behat tests on Windows

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers
I run Behat on Windows, and it works for me, including

require_once(__DIR__ . '/../moodle-browser-config/init.php');

Strangely, I have a weird mixture of slashes in some of my paths, e.g.

$CFG->dataroot = 'C:\Users\username/data/moodledata';

I have no idea why my config file is like that, but if you use single-quoted strings, you don't need to escape \.
Average of ratings: -
In reply to Tim Hunt

Re: Cannot run Behat tests on Windows

by Joseph Rézeau -
Picture of Core developers Picture of Plugin developers Picture of Testers Picture of Translators

Hi Tim!

As stated in one of my earlier posts, when I run php admin/tool/behat/cli/init.php I get: Result: Error: Composer dependencies are not installed. 

I have traced that error message to C:\wamp64\www\moodletest\lib\behat\classes\behat_command.php

public static function are_behat_dependencies_installed() {
        if (!is_dir(__DIR__ . '/../../../vendor/behat')) {
            return false;
        }

and now I see that I have no /vendor/behat folder anywhere inside my C:\wamp64\www\moodletest/ folder.

What does __DIR__ . '/../../../vendor/behat' mean?

When I previously installed composer on my Windows box, it installed itself here: "C:\Users\All Users\ComposerSetup". Should composer reside somewhere else? Composer is in my path at C:\Users\josep\AppData\Roaming\Composer\vendor\bin.

Now time to go to bed and sleep over it.💤🛏️

Average of ratings: -
In reply to Joseph Rézeau

Re: Cannot run Behat tests on Windows

by Tim Hunt -
Picture of Core developers Picture of Documentation writers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

When I run php admin/tool/behat/cli/init.php, it runs composer for me, to ensure all the dependcies are present and up-to-date.

I think the question is: why is the code in testing_update_composer_dependencies not working for you? Perhaps add some print_object statements in there, or otherwise debug what it is doing?

Average of ratings: -
In reply to Joseph Rézeau

Re: Cannot run Behat tests on Windows

by Renaat Debleu -
Picture of Core developers Picture of Plugin developers
Did you try to run the command 'composer install' in the root directory of your Moodle install? Perhaps you will need to configure COMPOSER_ALLOW_SUPERUSER=1 somewhere.
Average of ratings: -
In reply to Joseph Rézeau

Re: Cannot run Behat tests on Windows

by Patrick Lemaire -
Picture of Particularly helpful Moodlers Picture of Testers
Is Windows a kind of challenge here?
Why not running Linux in a Windows ? https://wiki.debian.org/InstallingDebianOn/Microsoft/Windows/SubsystemForLinux
Debian WSL is as fast as a real Linux installation. I keep my Windows for other things. I still use Visual Studio Code to code from Windows to Debian.

Sorry for this probably off-topic intervention 😊
Average of ratings: -
In reply to Joseph Rézeau

Re: Cannot run Behat tests on Windows - SOLVED

by Joseph Rézeau -
Picture of Core developers Picture of Plugin developers Picture of Testers Picture of Translators

I'm pleased to report that I have finally managed to get the Moodle Behat tests to work on my local Windows machine.😊 Not sure what I did differently from previous unsuccessful attempts, but my obstination did pay. Here is a summary of my current findings.

Using Windows 11. PHP 8.2.13. WampServer 3.3.3. Moodle 4.4+ (Build: 20240425)

Behat tests work with Selenium: I run "java -jar selenium-server-4.11.0.jar standalone";

Behat tests do not work if I run the geckodriver

But after running Selenium, I can use --profile=geckodriver or --profile=headlessgeckodriver when I run vendor\bin\behat etc. commands! Does this mean that the geckodriver is "incorporated" in the Selenium driver? I still cannot run tests in browsers other than Firefox, but I don't mind.

I can now create and run Behat tests for my little plugins. It's fun watching the "marionette" at work in Firefox...

Running the tests on my github repo is another story... Not easy to find out why some tests work fine locally but fail on my github, but I'm making progress there too.

Average of ratings: Useful (2)