Problems executing Behat tests

Problems executing Behat tests

by Tim V. -
Number of replies: 5
Hi,

I'm trying to execute the Behat test on Windows. Unfortunately the tests are not working as expected.

I've a fresh Moodle installation and I adjusted my config.php (see attachment) according to https://docs.moodle.org/dev/Running_acceptance_test
Thereafter I initialized the testing environment:

cd C:\nginx\html\moodle381
php admin/tool/behat/cli/init.php

So far, so good. Everything has been executed successfully, so I started my Selenium server (standalone mode), followed by the Behat tests itself:

java -jar selenium-server-standalone-3.141.59.jar

14:03:00.058 INFO [GridLauncherV3.parse] - Selenium server version: 3.141.59, revision: e82be7d358
14:03:00.099 INFO [GridLauncherV3.lambda$buildLaunchers$3] - Launching a standalone Selenium Server on port 4444
2020-01-17 14:03:00.130:INFO::main: Logging initialized @187ms to org.seleniumhq.jetty9.util.log.StdErrLog
14:03:00.259 INFO [WebDriverServlet.<init>] - Initialising WebDriverServlet
14:03:00.401 INFO [SeleniumServer.boot] - Selenium Server is up and running on port 4444

vendor\bin\behat --config C:\nginx\behatdata\behatrun\behat\behat.yml --format=pretty

Executing these both commands, a new Firefox instance appears (see attachment), when "Scenario: Agree on behalf of another user as a manager, single policy, javascript off" is getting executed and getting redirected to "http://127.0.0.1/moodle381/login/index.php". But it never logs in and futhermore, Behat told me that some field do not exist, e.g.:

  @javascript
  Scenario: Bulk agree on behalf of another users as a manager, multiple policies, javascript on # C:\nginx\html\moodle381\admin\tool\policy\tests\behat\acceptances.feature:265
    Given I log in as "admin"                                                                    # behat_auth::i_log_in_as()
      Field matching locator "Username" not found. (Behat\Mink\Exception\ElementNotFoundException)

... or:

  Scenario: Agree on behalf of another user as an admin who is logged in as a manager                               # C:\nginx\html\moodle381\admin\tool\policy\tests\behat\acceptances.feature:233
    Given I log in as "admin"                                                                                       # behat_hooks::i_look_for_exceptions()
    And I set the following system permissions of "Manager" role:                                                   # behat_permissions::i_set_the_following_system_permissions_of_role()
      | capability               | permission |
      | tool/policy:acceptbehalf | Allow      |
      Xpath matching locator "//input[@name="tool/policy:acceptbehalf" and @value="1"]" not found. (Behat\Mink\Exception\ElementNotFoundException)

See behat.log I attached for more information.

So, I'm a bit confused whether the tests are working properly or not because of the erros I get and because Behat never logs in to.

Here are the version I'm using for my test environment:

PHP Version: 7.4.1
Moodle Version: 3.8.1
Firefox Version: 72.0.1
MariaDB Version: 10.3.13
OS: Windows 10
Selenium: 3.141.59
Geckodriver: 0.26.0

Furthermore, is there any description for the moodle-output-format? I haven't found any documentation about that... What's the meaning of the symbols 'E', 'F', 'S', '.'?

Any help is much appreciated! Thanks in advance!

Attachment firefox.png
Average of ratings: -
In reply to Tim V.

Re: Problems executing Behat tests

by Davo Smith -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

I can't help with the errors, as I've never tried to run behat under Windows.

I can help with the output:

  • E - the PHP code generated an error message
  • F - the test step failed (without a PHP error)
  • S - the test step was skipped (due to an earlier failure or the test being excluded via certain tags)
  • . - the test step completed


Average of ratings: Useful (1)
In reply to Davo Smith

Re: Problems executing Behat tests

by Tim V. -
Thank you Davo for the explanation! One more question... I recently saw something like 'F----'. What does this mean?
In reply to Tim V.

Re: Problems executing Behat tests

by Davo Smith -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers

  • F---- - the step failed, so the following steps were not executed (I believe)
  • S - (to slightly correct what I wrote earlier) step was skipped (due to settings, not due to earlier failures)

I may be wrong about any of these, but this is my understanding from writing / running Behat tests for several years.

In reply to Tim V.

Re: Problems executing Behat tests

by Renaat Debleu -
Picture of Core developers Picture of Particularly helpful Moodlers Picture of Plugin developers
Average of ratings: Useful (2)
In reply to Renaat Debleu

Re: Problems executing Behat tests

by Tim V. -

Thank you Renaat! I tried using Chrome 79 as described in MDL-58948. Now, the navigation works, Behat logs in and is doing some tests. But I'm still getting errors like "Xpath matching locator "//input[@name="tool/policy:acceptbehalf" and @value="1"]" not found. (Behat\Mink\Exception\ElementNotFoundException)".

The reason seems to be that Moodle throws the following exception when Behat clicks on "Show advanced" in the "Define roles" tab:

Exception - Unknown error type: Invalid characters passed for attempted conversion, these have been ignored in [dirroot]\lib\typo3\class.t3lib_cs.php on line 1316

Debug info:
Error code: generalexceptionmessage

Stack trace:

  • line 158 of \lib\behat\lib.php: Exception thrown
  • line ? of unknownfile: call to behat_error_handler()
  • line 1316 of \lib\typo3\class.t3lib_cs.php: call to hexdec()
  • line 2098 of \lib\typo3\class.t3lib_cs.php: call to t3lib_cs->initUnicodeData()
  • line 1727 of \lib\typo3\class.t3lib_cs.php: call to t3lib_cs->utf8_char_mapping()
  • line 433 of \lib\classes\text.php: call to t3lib_cs->specCharsToASCII()
  • line 102 of \admin\roles\classes\define_role_table_advanced.php: call to core_text::specialtoascii()
  • line 191 of \admin\roles\define.php: call to core_role_define_role_table_advanced->read_submitted_permissions()

I already re-installed Moodle 3.8.1 from the scratch but the error persists. Is there any known bug regarding this exception? I found some bug reports regarding similar exceptions but nothing that is related to the "Show advanced" button of the role definitions.