Rake runner support for selenium grid?

Will there be support for running selenium grid spec tests in parallel using the rake runner?

I have been launching my rspec and test unit scripts for a few months now using the rake runner.  I like the way that the rake runner reports the number of completed tests as they are executing. I also like the pass fail history per individual test in the tests run.  The problem is that I can't run tests in parallel using the team city rake runner. I am using the selenium grid for test execution.  It is very important to know that the rake task to run scripts in parallel in the ruby example with the selenium grid distribution is not a standard spec task.  The spec task is overridden by the deep test gem that allows you to run tests in parallel.   Deep test is dependant on rspec 1.1.8, if you launch your scripts using the team city rake runner, deep test cannot override the default spec task correctly because it is running on the customized rspec library used for the team city rake runner.   I finally realized that is the reason that my tests are not executing in parallel.  I changed the runner to be command line and now my tests execute in parallel. The only down side is that I don't get the per test stats that the rake runner gives.
Will there be support for running rake tasks based on deep test in the future?

20 comments
Comment actions Permalink

Hi Scott,

I've answered you in your blog - http://scottcsims.com/wordpress/?p=90. Is you project is open sourced? It will be easier for me to start playing with Selenium + TC on some real example instead of configuring all from the beginning

0
Comment actions Permalink

Hi Roman,
Thanks for the excellent response on my blog.  I was able to use the spec options in team city to pass in the parameters I needed.  I will post today how I was able to run tests using the rake runner and then using the command line runner.  In answer to your question about where to find code to test with, you can download the selenium grid source from the download page for selenium grid. http://selenium-grid.seleniumhq.org/download.html    Navigate to the examples directory for ruby selenium-grid-1.0.4/examples/ruby.  Run tests in parallel with this rake task rake tests:run_in_parallel .   It runs tests against google images.

Philippe Hanrigou is the author of the selenium grid project and the selenium-client gem.  The gem is great to use if you are codling your tests with ruby. David Vollbracht is the author of deep tests that allows you to run in parallel.

0
Comment actions Permalink

Scott,

Thanks a lot for you links. I'm going to play with it =)

0
Comment actions Permalink

This is great!.  I am looking forward to hearing about results from your research.    I finished the blog on how I configured my rake runner profile to run with the selenium grid rake task. http://scottcsims.com/wordpress/?p=97  This configuration will run the tests in sequence.   I will post tomorrow the command line configuration for running the tests in parallel.  

0
Comment actions Permalink

If you want to use selenium rspec formatter and it's html output - there is no sense in attaching TeamCity RSpec formatter. TeamCity 5.0 EAPs allows to disable our rspec formatter in Rake runner settings and thus you wont have pain with SPEC_OPTRS.

P.S: Also TC allows to attach any custom html report as a separate tab in build configuration results

0
Comment actions Permalink

Yes, I like the way I can attach my rspec html report along with the team city build results.

Have you had a chance to see how deep test overrides the Spec::Runner::ExampleGroupRunner in the deep test gem? Deep test requires rspec 1.1.8, I am guessing that the ExampleGroupRunner changes in later versions of rspec. I would like to use the rake runner in team city to run my tests to that I can make use of the excellent reporting on test execution speed and pass fail status.  Do you still need me to document how I am running these tests using the command line runner in team city?

0
Comment actions Permalink

Hi Scott,

Thank you for reminding about this. I've setup some demo project on my machine but haven't manage to play with TC rake runner plugin on selenium grid. At current moment we are working on RubyMine 2.0 coming release and I'll return to this tasks after RubyMine release.

0
Comment actions Permalink

We just upgraded to Team City Version 5.1.2.  I really like the new rake test reporting section.  Picture 13.png

We have started using Cucumber so this will be great.  I wanted to ask about running ruby tests in parallel with rspec. The gem that allows you to run tests in parallel is called deep test. It overrides the spec Runner < ::Spec::Runner::ExampleGroupRunner and depends on rspec 1.1.12. I believe that the spec runner for team city overrides my SPEC_OPTS.  Can I have more information on the internal spec ops being used so that I can start using the rake runner for my selenium spec tests instead of the command line runner.
Thanks,
Scott

0
Comment actions Permalink

Hello Scott,

Nice to hear, that you use TeamCity and like our Rake runner.

I wanted to ask about running ruby tests in parallel with rspec. The gem that allows you to run tests in parallel is called deep test. It overrides the spec Runner < ::Spec::Runner::ExampleGroupRunner and depends on rspec 1.1.12

I know about this gem and I'm planning to look closer on it in future and most likely add support for it in RubyMine/TeamCity. I suppose that if the gem merges tests events of parallel threads TC wont show results correctly

I believe that the spec runner for team city overrides my SPEC_OPTS.  Can I have more information on the internal spec ops being used

It's not a secret. TeamCity merges users SPEC_OPTS with

--require teamcity/spec/runner/formatter/teamcity/formatter --format Spec::Runner::Formatter::TeamcityFormatter:matrix


also TC patches load path using RUBYLIB env variable. It's value is agent-specific, just try to output it in any rake tasks.

so that I can start using the rake runner for my selenium spec tests instead of the command line runner.

Ok, cool! If you manage to do it please post a link on some setup instructions =)

0
Comment actions Permalink

I was able to use the rake runner to create a report that I use as an artifact.

This is what the task looks like.  The spec opts were being ignored by the rake runner.

Spec::Rake::SpecTask.new("spec:run_in_parallel_report") do |t|
  t.spec_files = FileList['./*_spec.rb']
  t.deep_test :number_of_agents => 6
  t.spec_opts << '--color'
  t.spec_opts << "--require 'rubygems,#{report_formatter_path}'"
  t.spec_opts << "--format=Selenium::RSpec::SeleniumTestReportFormatter:./tmp/test_report.html"
  t.spec_opts << "--format=progress"
  t.fail_on_error = false
end

I was able to add this into my rake runner spec options to create the html report as an artifact.
--require 'rubygems,lib/reporting/selenium_test_report_formatter' --format=Selenium::RSpec::SeleniumTestReportFormatter:./tmp/test_report.html

My Question.
Do you have any ideas on how I can get the t.deep_test option in the above task to be recognized in the rake runner?

Thanks, Scott.

0
Comment actions Permalink

I just found this information.

I need to figure out how to provide this information via the command line but the object is being marshalled.
Here is the command
@deep_test_spec_opts = [
           "--require #{deep_test_path}",
           "--runner 'DeepTest::Spec::Runner:#{deep_test_options.to_command_line}'"
         ]
Here it the function
    def to_command_line
       Base64.encode64(Marshal.dump(self)).gsub("\n","")
     end
Here is the output
--require /lib/deep-test/lib/deep_test
--runner 'DeepTest::Spec::Runner:BAhvOhZEZWVwVGVzdDo6T3B0aW9ucw86CEB1aSIaRGVlcFRlc3Q6OlVJOjpDb25zb2xlOhFAc2VydmVyX3BvcnQwOhtAZW52aXJvbm1lbnRfbG9nX2xldmVsMDoNQHBhdHRlcm4wOg5AbGlzdGVuZXIiG0RlZXBUZXN0OjpOdWxsTGlzdGVuZXI6F0BkaXN0cmlidXRlZF9ob3N0czA6EkBtZXRyaWNzX2ZpbGUwOhVAb3JpZ2luX2hvc3RuYW1lIg5hdW1zc2ltMDE6FkBudW1iZXJfb2ZfYWdlbnRzaQs6EkBzeW5jX29wdGlvbnN7AA=='


0
Comment actions Permalink

Hello Scott,

Spec::Rake::SpecTask.new("spec:run_in_parallel_report") do |t|
  t.spec_files = FileList['./*_spec.rb']
  t.deep_test :number_of_agents => 6
  t.spec_opts << '--color'
  t.spec_opts << "--require 'rubygems,#{report_formatter_path}'"
  t.spec_opts << "--format=Selenium::RSpec::SeleniumTestReportFormatter:./tmp/test_report.html"
  t.spec_opts << "--format=progress"
  t.fail_on_error = false
end

Here I see 2 different ways

1. Disable "RSpec" reporter in Rake runner options. TeamCity wont show nice tests counts/statistics in build configuration results, but all console output will be available in build.log. In this case TC won't overried your "t.spec_opts"

2. In your rake task append all necessary options to SPEC_OPTS environment variable. But please keep existing SPEC_OPTS value because it is used for attaching TC rspec reporter to rspec process.

By the way I hope the problem will be fixed in rspec2.0. We are working on corresponding changes in rspec2.0 spec_opts behaviour.

0
Comment actions Permalink

Roman,
Thanks for the information.  I made good progress on this, but I need your help.
I decided to go with option 2 of your last email and append all necessary options to SPEC_OPTS environment variable.

I was able to get the deep test options into the SPEC_OPTS variable. In my rake file I add the information below for the --runner option.

This is the output of my SPEC_OPTS environment variable.

--require 'rubygems,lib/reporting/selenium_test_report_formatter,lib/deep-test/lib/deep_test' --format=Selenium::RSpec::SeleniumTestReportFormatter:./tmp/test_report.html --require 'teamcity/spec/runner/formatter/teamcity/formatter' --format Spec::Runner::Formatter::TeamcityFormatter:matrix --runner 'DeepTest::Spec::Runner:BAhvOhZEZWVwVGVzdDo6T3B0aW9ucw86CEB1aSIaRGVlcFRlc3Q6OlVJOjpDb25zb2xlOhFAc2VydmVyX3BvcnQwOhtAZW52aXJvbm1lbnRfbG9nX2xldmVsMDoNQHBhdHRlcm4wOg5AbGlzdGVuZXIiG0RlZXBUZXN0OjpOdWxsTGlzdGVuZXI6F0BkaXN0cmlidXRlZF9ob3N0czA6EkBtZXRyaWNzX2ZpbGUwOhVAb3JpZ2luX2hvc3RuYW1lIg5hdW1zc2ltMDE6FkBudW1iZXJfb2ZfYWdlbnRzaQs6EkBzeW5jX29wdGlvbnN7AA=='

The long character string you see is the base64 encoded deep test configuration options. I was not familiar with this method, but I believe the encoding is done so that the data can contain single and double quotes with out needing to be escaped.

So I was able to run my team city job with this option.  It ran one test sucessfully and ended with this error.

[12:08:48]: [Execute run_in_parallel_multi_Firefox_3_5_on_Windows_2003_Server] /opt/buildAgent/plugins/rake-runner/lib/rb/patch/bdd/teamcity/spec/runner/formatter/teamcity/formatter.rb:417:in `stop_capture_output_and_log_it': undefined method `additional_flowid_suffix' for nil:NilClass (NoMethodError)
[12:08:48]: [Execute run_in_parallel_multi_Firefox_3_5_on_Windows_2003_Server] from /opt/buildAgent/plugins/rake-runner/lib/rb/patch/bdd/teamcity/spec/runner/formatter/teamcity/formatter.rb:154:in `example_passed'
[12:08:48]: [Execute run_in_parallel_multi_Firefox_3_5_on_Windows_2003_Server] from /home/build/.gem/ruby/1.8/gems/rspec-1.1.12/lib/spec/runner/reporter.rb:104:in `example_passed'
[12:08:48]: [Execute run_in_parallel_multi_Firefox_3_5_on_Windows_2003_Server] from /home/build/.gem/ruby/1.8/gems/rspec-1.1.12/lib/spec/runner/reporter.rb:104:in `each'
[12:08:48]: [Execute run_in_parallel_multi_Firefox_3_5_on_Windows_2003_Server] from /home/build/.gem/ruby/1.8/gems/rspec-1.1.12/lib/spec/runner/reporter.rb:104:in `example_passed'
[12:08:48]: [Execute run_in_parallel_multi_Firefox_3_5_on_Windows_2003_Server] from /home/build/.gem/ruby/1.8/gems/rspec-1.1.12/lib/spec/runner/reporter.rb:27:in `example_finished'
[12:08:48]: [Execute run_in_parallel_multi_Firefox_3_5_on_Windows_2003_Server] from ./lib/deep-test/lib/deep_test/spec/runner.rb:55:in `process_work_units'
[12:08:48]: [Execute run_in_parallel_multi_Firefox_3_5_on_Windows_2003_Server] from ./lib/deep-test/lib/deep_test/result_reader.rb:26:in `read'
[12:08:48]: [Execute run_in_parallel_multi_Firefox_3_5_on_Windows_2003_Server] from ./lib/deep-test/lib/deep_test/spec/runner.rb:54:in `process_work_units'
[12:08:48]: [Execute run_in_parallel_multi_Firefox_3_5_on_Windows_2003_Server] from ./lib/deep-test/lib/deep_test/main.rb:24:in `run'
[12:08:48]: [Execute run_in_parallel_multi_Firefox_3_5_on_Windows_2003_Server] from ./lib/deep-test/lib/deep_test/spec/runner.rb:30:in `run'
[12:08:48]: [Execute run_in_parallel_multi_Firefox_3_5_on_Windows_2003_Server] from /home/build/.gem/ruby/1.8/gems/rspec-1.1.12/lib/spec/runner/options.rb:111:in `run_examples'
[12:08:48]: [Execute run_in_parallel_multi_Firefox_3_5_on_Windows_2003_Server] from /home/build/.gem/ruby/1.8/gems/rspec-1.1.12/lib/spec/runner/command_line.rb:9:in `run'
[12:08:48]: [Execute run_in_parallel_multi_Firefox_3_5_on_Windows_2003_Server] from  /home/build/.gem/ruby/1.8/gems/rspec-1.1.12/bin/spec:4



0
Comment actions Permalink

Our rspec reporter keeps hash of running examples. According to exception it looks like somebody reported several times that the example has finished/or not started at all. RSpec reporter expects that example is "running" but cannot find associated information in the hash. I don't know whats going wrong - some bug in rspec formatter related to concurrency or changed original events protocol provided by deep-test.
We didn't tests Rake runner with Selenium and Deep tests.

I tried to setup selenium project several monthes ago, but I faced with some troubles and suspended that task. I will appreciate if your describe some test/configuration scenario for Selenium demo project with 1-2 simple tests. E.g. which version of gems should I use, which gems, how better to configure all this stuff etc. Because without live example it's too hard to fix it.

0
Comment actions Permalink

I have the information you need to get started.  I forked the grid project and updated the ruby example.It is under review by Kevin Menard (The Maintainer) of slenium grid.

Here is the path on github to get the code from my repo
http://github.com/scottcsims/selenium-grid/tree/master/examples/ruby/

This is a link to detailed information on the changes I made to make the example work. http://scottcsims.com/wordpress/?p=206  Check out the future goals section, we are working on one now!

I use the gem bundler tool to help me manage my gems, but the example does not contain the bundler.  These are the gems you need.

rspec -v 1.1.12
selenium-client -v 1.2.18

Thanks for the help!

0
Comment actions Permalink

Thanks, definitely I'll take a look on it!

0
Comment actions Permalink

Hey Roman,
I am checking to see if you were able to get rspec test to run in parallel from the rake runner.  I have been working on the project today. I created a pre release gem for deep test that supports rspec 1.1.12 http://scottcsims.com/wordpress/?p=234
I also updated the test code in the selenium grid project to use the gem instead of the code directly.  You can check out the code from http://github.com/scottcsims/selenium-grid/tree/master/examples/ruby/
Please let me know if you need any help, I am very excited about this feature being added to team city.

0
Comment actions Permalink

Hello Scott,

I appreciate your willingness to help us. I filed two issues RUBY-6438, RUBY-6437. At the moment I'm working on RCov support (in RubyMine) and coming RubyMine 2.5 version. I'll return to this task after RubyMine 2.5 release.

0
Comment actions Permalink

Hi,

I've just tried your selenium example project and I managed to launch selenium tests in sequence mode. I've checked that parallel mode really works in console, but not in RubyMine. Thank you for the example. By the way please update link to selenium hub launch demo, old link is dead, I use demo at http://selenium-grid.seleniumhq.org/run_the_demo.html.

With your help I can reproduce the problem and investigate it. At the moment we are working on RubyMine & TeamCity release last fixes, but I'm going to return to this task in future.

0
Comment actions Permalink

This is great to hear.  I am looking forward to working on this.  I have been running the beta 3.0 release of rubymine and I like the new look on OS X.

0

Please sign in to leave a comment.