I've been working on setting up a CI server to automate the Behat + PHPUnit testing for a number of our customer's Moodle repos.
With a clean Moodle 3.0 site, I'm getting 100% pass rate for all the PHPUnit tests (great!).
However, the Behat tests are giving random failures - usually about 20-30 (out of 1168) scenarios will fail in any given run. Usually it is a different subset of tests that fail in each run (e.g. 15 failing tests will pass on the second run, 10 will still fail, 15 new tests will fail). When I look at the screenshots of the tests that fail, the problems just don't make sense - e.g. the behat step creates a new forum and fills in the forum name, type and description; but the screenshot of the failed test shows that the forum name is missing (with a 'required field' warning). The failing tests are distributed evenly through the results (not bunched together in any way I can determine) and all tests pass when run locally.
I'm running multiple tests in parallel (10) on a high-spec AWS instance, running Ubuntu (c3.4xlarge, 16 CPU + 30Gb RAM), latest selenium, latest firefox, using apache for the web server, output via xvfb.
Has anyone else come across similar issues and got any insight about what might be going wrong?