Behat on Jenkins takes forever. How can I speed it up?

Behat on Jenkins takes forever. How can I speed it up?

by Matt Gibson -
Number of replies: 3

I just set up Jenkins and running the Behat acceptance tests took 5 1/2 hrs for 179 scenarios.

On my laptop, I can run the same number of Cucumber scenarios (using Ruby and Capybara) in around 2 minutes.

Does anyone know where to look for the bottlenecks? I was trying to change it to use zombiejs, but this seems to be really awkward. Also, is it possible to run only a subset of the tests as with phpunit?

Average of ratings: -
In reply to Matt Gibson

Re: Behat on Jenkins takes forever. How can I speed it up?

by David Monllaó -

Hi Matt,

There are a few points to consider before comparing the time it takes to run 180 Moodle scenarios and other application's scenarios:

  • Most of the scenarios uses a Selenium instance, which is slower than non-JS scenarios
  • The Moodle scenarios are complex, most of them requires quite a few steps
  • A new Moodle site is restored after each scenario (it takes around 2 secs in pgsql and 7 in mysql) to run the scenarios in an isolated environment

In our Jenkins instance is taking about 3h to finish, right now, 184 steps. We are working on improving these numbers using more non-JS tests and less JS (Selenium) as well as refining the xpath queries, but 2 minutes for about 180 does not seem realistic for this kind of scenarios. PhantomJS (headless, based on webkit) will be quicker (https://github.com/detro/ghostdriver) but at the time we try it (a couple of months ago) there were Selenium commands not fully implemented, but being optimistic in the best case it will require 1 or 2 hours to finish.

To run a subset of the features: http://docs.behat.org/guides/6.cli.html#gherkin-filters, all Moodle features are tagged according to the component they belong to.

Average of ratings: Useful (1)
In reply to David Monllaó

Re: Behat on Jenkins takes forever. How can I speed it up?

by Matt Gibson -

Thanks.

I'm using postgres already, so it sounds like I should be able to get a bit more performance than currently. PHP's memory_limit is 256M, so I'm not sure what else to look at. 

Average of ratings: -