TeamCity Rake execute cucumber tests: Ignoring <gem> because its extensions are not built
**deep breath** alright so here goes. Trying to support two versions of ruby for running cucumber tests because we're upgrading our Ruby support for our test framework from 2.3.3 to 2.7.0
Ruby environment configurator: Set up to use RVM with a parameterized Interpreter name and gemset.
Interpreter name: The ruby interpreter is using a parameter that is set to 2.3.3 by default but a build step determines it by 'cat'ing the .ruby-version into a teamcity parameter. The thought being the next steps will pick up that change as it goes. We set it to ruby-%rvm.ruby% which by default is set to 2.3.3.
Gemset: The gemset is set to the %teamcity.build.id%.
Build steps 1-4: Does stuff that isn't related to this
Build step 5. Sets ruby version by cat'ing the .ruby-version
Build step 6: Bundler. I install two versions of bundler to handle our parallel support (hopefully) for older test projects using our framework that are all using like 1.17.1 of bundler and newer test projects using 2.1.4. Then I run `bundle install` which should use the version defined in teh Gemfile.lock to do the bundling.One thing I'm noticing here is that gems like nokogiri install WAY too fast. I'm used to that thing taking tens of seconds and it finishes in like 8.
Build step 7-9: unrelated
Build step 10: Execute tests. Here's the meat of the problem. I'm using a rake runner with the following configuration:
rake task: our rake task that we've defined to run cucumber with a lot of custom stuff
Additional Rake command lien parameters: just more parameters into our custom rake task
Mode: RVM Interpreter
RVM Interpreter: ruby-%rvm.ruby%
RVM Gemset: %teamcity.build.id% the idea being that the bundle install step used the same RVM interpreter and gemset as defined with Ruby Environment Configurator.
Bundler: we check `bundle exec`
various other settings: track invoke/execute stages, cucumber with additional options
---------------------
It's this step that fails. Here's what I'm seeing:
deep breath alright so here goes. Trying to support two versions of ruby for running cucumber tests in TeamCity because we're upgrading our Ruby support for our test framework from 2.3.3 to 2.7.0
Ruby environment configurator: Set up to use RVM with a parameterized Interpreter name and gemset.
Interpreter name: The ruby interpreter is using a parameter that is set to 2.3.3 by default but a build step determines it by 'cat'ing the .ruby-version into a teamcity parameter. The thought being the next steps will pick up that change as it goes. We set it to ruby-%rvm.ruby% which by default is set to 2.3.3.
Gemset: The gemset is set to the %teamcity.build.id%.
Build steps 1-4: Does stuff that isn't related to this
Build step 5. Sets ruby version by cat'ing the .ruby-version file into a teamcity parameter
Build step 6: Bundler. I install two versions of bundler to handle our parallel support (hopefully) for older test projects using our framework that are all using like 1.17.1 of bundler and newer test projects using 2.1.4. Then I run bundle install which should use the version defined in teh Gemfile.lock to do the bundling.One thing I'm noticing here is that gems like nokogiri install WAY too fast. I'm used to that thing taking tens of seconds and it finishes in like 8.
Build step 7-9: unrelated
Build step 10: Execute tests. Here's the meat of the problem. I'm using a rake runner with the following configuration:
rake task: our rake task that we've defined to run cucumber with a lot of custom stuff
Additional Rake command line parameters: just more parameters into our custom rake task
Mode: RVM Interpreter
RVM Interpreter: ruby-%rvm.ruby%
RVM Gemset: %teamcity.build.id% the idea being that the bundle install step used the same RVM interpreter and gemset as defined with Ruby Environment Configurator.
Bundler: we check bundle exec
various other settings: track invoke/execute stages, cucumber with additional options
It's this step that fails. Here's what I'm seeing:
[08:54:34]Step 10/17: 8. Execute Tests (Rake) (4s)
[08:54:38][Step 10/17] Environment variable 'GEM_HOME' has predefined value '/usr/local/rvm/gems/ruby-2.7.0@11969694'. It may affect runtime build behaviour because TeamCity RVM support won't override it.
[08:54:38][Step 10/17] Environment variable 'GEM_PATH' has predefined value '/usr/local/rvm/gems/ruby-2.7.0@11969694:/usr/local/rvm/gems/ruby-2.7.0@global'. It may affect runtime build behaviour because TeamCity RVM support won't override it.
[08:54:38][Step 10/17] Environment variable 'MY_RUBY_HOME' has predefined value '/usr/local/rvm/rubies/ruby-2.7.0'. It may affect runtime build behaviour because TeamCity RVM support won't override it.
[08:54:38][Step 10/17] Environment variable 'IRBRC' has predefined value '/usr/local/rvm/rubies/ruby-2.7.0/.irbrc'. It may affect runtime build behaviour because TeamCity RVM support won't override it.
[08:54:38][Step 10/17] Cucumber Options: features --expand --format json --out ./Logs/cucumber_report.json --expand --format Teamcity::Cucumber::Formatter
[08:54:38][Step 10/17] Starting: /bin/sh /opt/buildAgent/temp/agentTmp/build4928453681402755736.sh
[08:54:38][Step 10/17] in directory: /opt/buildAgent/work/423ccd9741f6c2a4
[08:54:38][Step 10/17] Ignoring executable-hooks-1.6.0 because its extensions are not built. Try: gem pristine executable-hooks --version 1.6.0
[08:54:38][Step 10/17] Ignoring gem-wrappers-1.4.0 because its extensions are not built. Try: gem pristine gem-wrappers --version 1.4.0
[08:54:38][Step 10/17] Ignoring jaro_winkler-1.5.4 because its extensions are not built. Try: gem pristine jaro_winkler --version 1.5.4
[08:54:38][Step 10/17] Ignoring json-1.8.6 because its extensions are not built. Try: gem pristine json --version 1.8.6
[08:54:38][Step 10/17] Ignoring malloc_trim-0.1.0 because its extensions are not built. Try: gem pristine malloc_trim --version 0.1.0
[08:54:38][Step 10/17] Ignoring nokogiri-1.10.9 because its extensions are not built. Try: gem pristine nokogiri --version 1.10.9
[08:54:38][Step 10/17] Ignoring unf_ext-0.0.7.7 because its extensions are not built. Try: gem pristine unf_ext --version 0.0.7.7
[08:54:38][Step 10/17] Could not find unf_ext-0.0.7.7 in any of the sources
[08:54:38][Step 10/17] Run `bundle install` to install missing gems
[08:54:38][Step 10/17] Process exited with code 7
[08:54:38][Step 10/17] Process exited with code 7 (Step: 8. Execute Tests (Rake))
[08:54:38][Step 10/17] Step 8. Execute Tests (Rake) failedI'm at a loss with this. I added in a step that runs a new custom rake task from our framework that all it does is spit out the output of rvm gemset list and at least for THAT rake task executed in our TeamCity configuration it looks like it's using the correct/expected gemset and ruby version, so I'm assuming our rake task that executes cucumber is also using the right gemset. Our bundle install step ALSO spits out rvm gemset list and it looks correct there too.
One thing that does jump out at me are those "Environment variable 'X' has predefined value" warnings. But they all reference ruby 2.7.0 and the correct gemset so I'm not sure that's the problem.
Oh and if I set up a step to execute them via the command line it gets further, but fails trying to use the teamcity/cucumber/formatter. So there's SOMETHING about running this as a rake task that is getting it very confused about gems that need system libraries or extensions?
I'll lastly note that when I run a 2.3.3 test project, it works fine. No problems at all. it has those "predefined value" warnings as well... well... the first two. 2.3.3 does NOT spit out the MY_RUBY_HOME and IRBRC warnings.
If anyone has any bright ideas, I am ALL EARS. Thanks!!!!
[EDIT] I tried bundle exec gem pristine --all at the end of my build step that does my bundle install and got this error.
[09:27:13] [Step 6/17] Building native extensions. This could take a while...
[09:27:13] [Step 6/17] ERROR: While executing gem ... (Gem::Ext::BuildError)
[09:27:13] [Step 6/17] ERROR: Failed to build gem native extension.
[09:27:13] [Step 6/17]
[09:27:13] [Step 6/17] current directory: /usr/local/rvm/gems/ruby-2.7.0@11969897/gems/unf_ext-0.0.7.7/ext/unf_ext
[09:27:13] [Step 6/17] /usr/local/rvm/rubies/ruby-2.7.0/bin/ruby -I /usr/local/rvm/rubies/ruby-2.7.0/lib/ruby/2.7.0 -r ./siteconf20200624-747-1iuwizi.rb extconf.rb
[09:27:13] [Step 6/17] /usr/local/rvm/gems/ruby-2.7.0@11969897/gems/bundler-2.1.4/lib/bundler/spec_set.rb:86:in `block in materialize': Could not find unf_ext-0.0.7.7 in any of the sources (Bundler::GemNotFound)
[09:27:13] [Step 6/17] from /usr/local/rvm/gems/ruby-2.7.0@11969897/gems/bundler-2.1.4/lib/bundler/spec_set.rb:80:in `map!'
[09:27:13] [Step 6/17] from /usr/local/rvm/gems/ruby-2.7.0@11969897/gems/bundler-2.1.4/lib/bundler/spec_set.rb:80:in `materialize'
[09:27:13] [Step 6/17] from /usr/local/rvm/gems/ruby-2.7.0@11969897/gems/bundler-2.1.4/lib/bundler/definition.rb:170:in `specs'
[09:27:13] [Step 6/17] from /usr/local/rvm/gems/ruby-2.7.0@11969897/gems/bundler-2.1.4/lib/bundler/definition.rb:237:in `specs_for'
[09:27:13] [Step 6/17] from /usr/local/rvm/gems/ruby-2.7.0@11969897/gems/bundler-2.1.4/lib/bundler/definition.rb:226:in `requested_specs'
[09:27:13] [Step 6/17] from /usr/local/rvm/gems/ruby-2.7.0@11969897/gems/bundler-2.1.4/lib/bundler/runtime.rb:101:in `block in definition_method'
[09:27:13] [Step 6/17] from /usr/local/rvm/gems/ruby-2.7.0@11969897/gems/bundler-2.1.4/lib/bundler/runtime.rb:20:in `setup'
[09:27:13] [Step 6/17] from /usr/local/rvm/gems/ruby-2.7.0@11969897/gems/bundler-2.1.4/lib/bundler.rb:149:in `setup'
[09:27:13] [Step 6/17] from /usr/local/rvm/gems/ruby-2.7.0@11969897/gems/bundler-2.1.4/lib/bundler/setup.rb:20:in `block in <top (required)>'
[09:27:13] [Step 6/17] from /usr/local/rvm/gems/ruby-2.7.0@11969897/gems/bundler-2.1.4/lib/bundler/ui/shell.rb:136:in `with_level'
[09:27:13] [Step 6/17] from /usr/local/rvm/gems/ruby-2.7.0@11969897/gems/bundler-2.1.4/lib/bundler/ui/shell.rb:88:in `silence'
[09:27:13] [Step 6/17] from /usr/local/rvm/gems/ruby-2.7.0@11969897/gems/bundler-2.1.4/lib/bundler/setup.rb:20:in `<top (required)>'
[09:27:13] [Step 6/17] from /usr/local/rvm/rubies/ruby-2.7.0/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:92:in `require'
[09:27:13] [Step 6/17] from /usr/local/rvm/rubies/ruby-2.7.0/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:92:in `require'
[09:27:13] [Step 6/17]
[09:27:13] [Step 6/17] extconf failed, exit code 1
[09:27:13] [Step 6/17]
[09:27:13] [Step 6/17] Gem files will remain installed in /usr/local/rvm/gems/ruby-2.7.0@11969897/gems/unf_ext-0.0.7.7 for inspection.
[09:27:13] [Step 6/17] Results logged to /usr/local/rvm/gems/ruby-2.7.0@11969897/extensions/x86_64-linux/2.7.0/unf_ext-0.0.7.7/gem_make.out
[09:27:13] [Step 6/17] Could not find unf_ext-0.0.7.7 in any of the sources
[09:27:13] [Step 6/17] Could not find unf_ext-0.0.7.7 in any of the sourcesWhich is interesting. The "Gem files will remain installed in" line appears to indicate the correct gem location, but it's like the bundle install says it worked but... didn't really?
gem env results also from within my bundle install build step:
[09:26:44] [Step 6/17] RubyGems Environment:
[09:26:44] [Step 6/17] - RUBYGEMS VERSION: 3.1.2
[09:26:44] [Step 6/17] - RUBY VERSION: 2.7.0 (2019-12-25 patchlevel 0) [x86_64-linux]
[09:26:44] [Step 6/17] - INSTALLATION DIRECTORY: /usr/local/rvm/gems/ruby-2.7.0@11969897
[09:26:44] [Step 6/17] - USER INSTALLATION DIRECTORY: /home/vcsservice/.gem/ruby/2.7.0
[09:26:44] [Step 6/17] - RUBY EXECUTABLE: /usr/local/rvm/rubies/ruby-2.7.0/bin/ruby
[09:26:44] [Step 6/17] - GIT EXECUTABLE: /usr/bin/git
[09:26:44] [Step 6/17] - EXECUTABLE DIRECTORY: /usr/local/rvm/gems/ruby-2.7.0@11969897/bin
[09:26:44] [Step 6/17] - SPEC CACHE DIRECTORY: /home/vcsservice/.gem/specs
[09:26:44] [Step 6/17] - SYSTEM CONFIGURATION DIRECTORY: /etc
[09:26:44] [Step 6/17] - RUBYGEMS PLATFORMS:
[09:26:44] [Step 6/17] - ruby
[09:26:44] [Step 6/17] - x86_64-linux
[09:26:44] [Step 6/17] - GEM PATHS:
[09:26:44] [Step 6/17] - /usr/local/rvm/gems/ruby-2.7.0@11969897
[09:26:44] [Step 6/17] - /usr/local/rvm/rubies/ruby-2.7.0/lib/ruby/gems/2.7.0
[09:26:44] [Step 6/17] - GEM CONFIGURATION:
[09:26:44] [Step 6/17] - :update_sources => true
[09:26:44] [Step 6/17] - :verbose => true
[09:26:44] [Step 6/17] - :backtrace => false
[09:26:44] [Step 6/17] - :bulk_threshold => 1000
[09:26:44] [Step 6/17] - REMOTE SOURCES:
[09:26:44] [Step 6/17] - https://rubygems.org/
[09:26:44] [Step 6/17] - SHELL PATH:
[09:26:44] [Step 6/17] - /usr/local/rvm/gems/ruby-2.7.0@11969897/bin
[09:26:44] [Step 6/17] - /usr/local/rvm/gems/ruby-2.7.0@global/bin
[09:26:44] [Step 6/17] - /usr/local/rvm/rubies/ruby-2.7.0/bin
[09:26:44] [Step 6/17] - /usr/local/rvm/bin
[09:26:44] [Step 6/17] - /usr/local/bin
[09:26:44] [Step 6/17] - /usr/bin
[09:26:44] [Step 6/17] - /usr/local/sbin
[09:26:44] [Step 6/17] - /usr/sbin
[09:26:44] [Step 6/17] - /home/vcsservice/.dotnet/tools
[09:26:44] [Step 6/17] - /home/vcsservice/.local/bin
[09:26:44] [Step 6/17] - /home/vcsservice/bin
[09:26:44] [Step 6/17] - /home/vcsservice/.npm-global/bin
[09:26:44] [Step 6/17] - /usr/local/binPlease sign in to leave a comment.
Welp. A very strange solution. I deleted and re-created my TeamCity Rake build step that did test execution, changed the RVM interpreter to default (which obeys Ruby Environment Configurator settings - previously I was setting the interpeter to a specific parameter) and that did it!
I tried editing the existing Rake build step to RVM Interpreter as default, and that DIDN'T fix it.
Something about creating a whole new build step fixed it. Strange.