Cucumber reporting 0 scenarios

Cucumber reporting on our testing has started showing 0 scenarios, even though I can watch tests running and tail activity in the Cucmber log in the build agent working directory. The log reports that scenarios are running and passing. So, there is some kind of disconnect between the build agent and the server reporting. This has worked for us in the very recent past and does work on one of our agents (still digging into the difference between the two agents that might indicate the problem).

Looking for ideas on how to run this problem down.

Also, possibly related - Cucumber reporting has never showed a Tests tab and nice output formatting like we get from Test::Unit reporting.

thx
andy

5 comments
Comment actions Permalink

Turns out that this is because we're using the rerun functionality of cucumber. When all tests pass, there are no tests to run - an empty rerun file - and running cucumber again produces a log file containing '0 scenarios run'. Does the cucumber reporter understand cucumber's rerun functionality? As it is now, only the last log file of the reruns shows up in the build log. It'd be great if all log files from the various reruns showed up in the build log as that would be more accurate.

0
Comment actions Permalink

Hi Andy

Test rerun feature is handful for local usage, but at CI server it may make test statistics unreliable. Making your tests less dependent on the environment will give you better results in a long time.
You also may be interested in our experimental Flaky tests analyser plugin.

When you disable rerun task, are the tests reported correctly?

0
Comment actions Permalink

We are using Cucumber to run Calabash tests on mobile devices. This in itself is an unstable environment which prompts our re-rerunning of tests. We have not tried the flaky test plugin, but can look into that. Without reruns, we do get expected reporting.

0
Comment actions Permalink

Could you post a screenshot of build steps - how exactly Cucumber tests are started in your build configuration?

0
Comment actions Permalink

Cucumber is being run via a Rake task; there's nothing exceptional about the Rake build step that runs cucumber. It uses the system RVM interpreter (this is running on a Mac so that's Ruby RMI 1.8.7) and the Cucumber reporter.

0

Please sign in to leave a comment.