How to handle rake errors during the build

Hi,

I'm trying to setup TeamCity to deploy my Ruby-on-Rails application to Heroku.
I use Rake script, that deploys the code to the remote Heroku repo and restarts servers and also performs database migrations.

Actually in case of normal flow, everything works well, but if Rake-task fails with some error, the whole build doesn't fail.
It even not always shows the errors in build log (for example, when migrations could not be done because of constraints violation)
Could you please advice what I'm doing wrong and how can I achieve the expected behavior?
I'm using script like this:

#Deploy and rollback on Heroku in staging and production
task :deploy_staging => ['deploy:set_staging_app', 'deploy:push', 'deploy:restart']
task :deploy_production => ['deploy:set_production_app', 'deploy:push', 'deploy:restart', 'deploy:tag']
task :migrate_staging => ['deploy:set_staging_app', 'deploy:migrate' ]
namespace :deploy do
  PRODUCTION_APP = 'my_app_name'
  STAGING_APP = 'my_app_name'
 
  task :staging_migrations => [:set_staging_app,:off, :migrate, :restart, :on]
  task :staging_rollback => [:set_staging_app, :off, :push_previous, :restart, :on]
 
  task :production_migrations => [:set_production_app, :push, :off, :migrate, :restart, :on, :tag]
  task :production_rollback => [:set_production_app, :off, :push_previous, :restart, :on]
 
  task :set_staging_app do
    APP = STAGING_APP
  end
 
  task :set_production_app do
       APP = PRODUCTION_APP
  end
 
  task :push do
    puts 'Deploying site to Heroku ...'
    puts `git push -f git@heroku.com:#{APP}.git`
  end
  
  task :restart do
    puts 'Restarting app servers ...'
    Bundler.with_clean_env { "heroku restart --app #{APP}"}
  end
  
  task :tag do
    release_name = "#{APP}_release-#{Time.now.utc.strftime("%%Y%%m%%d%%H%%M%%S")}"
    puts "Tagging release as '#{release_name}'"
    puts `git tag -a #{release_name} -m 'Tagged release'`
    puts `git push --tags git@heroku.com:#{APP}.git`
  end
  
  task :migrate do
    puts 'Running database migrations ...'
    Bundler.with_clean_env {"heroku run rake db:migrate --app #{APP}"}
  end
  
  task :off do
    puts 'Putting the app into maintenance mode ...'
    Bundler.with_clean_env {"heroku maintenance:on --app #{APP}"}
  end
  
  task :on do
    puts 'Taking the app out of maintenance mode ...'
    Bundler.with_clean_env {"heroku maintenance:off --app #{APP}"}
  end
 
  task :push_previous do
    prefix = "#{APP}_release-"
    releases = `git tag`.split("\n").select { |t| t[0..prefix.length-1] == prefix }.sort
    current_release = releases.last
    previous_release = releases[-2] if releases.length >= 2
    if previous_release
      puts "Rolling back to '#{previous_release}' ..."
      
      puts "Checking out '#{previous_release}' in a new branch on local git repo ..."
      puts `git checkout #{previous_release}`
      puts `git checkout -b #{previous_release}`
      
      puts "Removing tagged version '#{previous_release}' (now transformed in branch) ..."
      puts `git tag -d #{previous_release}`
      puts `git push git@heroku.com:#{APP}.git :refs/tags/#{previous_release}`
      
      puts "Pushing '#{previous_release}' to Heroku master ..."
      puts `git push git@heroku.com:#{APP}.git +#{previous_release}:master --force`
      
      puts "Deleting rollbacked release '#{current_release}' ..."
      puts `git tag -d #{current_release}`
      puts `git push git@heroku.com:#{APP}.git :refs/tags/#{current_release}`
      
      puts "Retagging release '#{previous_release}' in case to repeat this process (other rollbacks)..."
      puts `git tag -a #{previous_release} -m 'Tagged release'`
      puts `git push --tags git@heroku.com:#{APP}.git`
      
      puts "Turning local repo checked out on master ..."
      puts `git checkout master`
      puts 'All done!'
    else
      puts "No release tags found - can't roll back!"
      puts releases
    end
  end
end

3 comments
Comment actions Permalink

Hi Anton,

Do you have 'the build process exit code is not zero' build failure condition enabled?
Also you can add "Fail build on specific text in build log" build failure condition.

If you still have problems please attach build log.

0
Comment actions Permalink

Well, I've double-checked build failure conditions and all neccessary items are selected.
Here is partial build log  - there is a problem with bundler's ruby version, which causes build step failure, but for TeamCity and even further Rake tasks everything looks like OK. So I don't understand how to say to TeamCity "Stop, if any error occurs"
Of course, I can add particular strings in build failure conditions, but I even don't know all possible errors, that could heroku  send during deployment, or for example, during database migration.
So, the only solution for me is to stop in case of any error.

[15:17:36]Step 3/3: Migrate DB (Rake) (1s)

[15:17:36][Step 3/3] Starting: /usr/local/bin/ruby /usr/local/TeamCity/buildAgent/plugins/rake-runner/rb/runner/rakerunner.rb --rakefile /usr/local/TeamCity/buildAgent/work/e76d00de5ef499ec/build6013725525059049148.tmp deploy:staging_migrations

[15:17:36][Step 3/3] in directory: /usr/local/TeamCity/buildAgent/work/e76d00de5ef499ec

[15:17:37][Step 3/3]

[15:17:37][Step 3/3] Invoke deploy:staging_migrations

[15:17:37][Invoke deploy:staging_migrations]

[15:17:37][Invoke deploy:staging_migrations] (first_time)

[15:17:37][Invoke deploy:staging_migrations]

[15:17:37][Invoke deploy:staging_migrations] Execute deploy:set_staging_app

[15:17:37][Execute deploy:set_staging_app]

[15:17:37][Invoke deploy:staging_migrations]

[15:17:37][Invoke deploy:staging_migrations] Execute deploy:off

[15:17:37][Execute deploy:off] Putting the app into maintenance mode ...

[15:17:38][Execute deploy:off] /usr/local/lib/ruby/gems/2.0.0/gems/bundler-1.3.5/lib/bundler/definition.rb:361:in `validate_ruby!': Your Ruby version is 1.9.3, but your Gemfile specified 2.0.0 (Bundler::RubyVersionMismatch)

[15:17:38][Execute deploy:off] from /usr/local/lib/ruby/gems/2.0.0/gems/bundler-1.3.5/lib/bundler.rb:116:in `setup'

[15:17:38][Execute deploy:off] from /usr/local/lib/ruby/gems/2.0.0/gems/bundler-1.3.5/lib/bundler/setup.rb:17:in `<top (required)>'

[15:17:38][Execute deploy:off] from /usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require'

[15:17:38][Execute deploy:off] from /usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require'

[15:17:38][Execute deploy:off]

[15:17:38][Execute deploy:off]

[15:17:38][Invoke deploy:staging_migrations]

[15:17:38][Invoke deploy:staging_migrations] Execute deploy:migrate

[15:17:38][Execute deploy:migrate] Running database migrations ...

[15:17:38][Execute deploy:migrate]

[15:17:38][Invoke deploy:staging_migrations]

[15:17:38][Invoke deploy:staging_migrations] Execute deploy:restart

[15:17:38][Execute deploy:restart] Restarting app servers ...

[15:17:38][Execute deploy:restart]

[15:17:38][Invoke deploy:staging_migrations]

[15:17:38][Invoke deploy:staging_migrations] Execute deploy:on

[15:17:38][Execute deploy:on] Taking the app out of maintenance mode ...

[15:17:38][Execute deploy:on]

[15:17:38][Invoke deploy:staging_migrations]

[15:17:38][Invoke deploy:staging_migrations] Execute deploy:staging_migrations

[15:17:38][Execute deploy:staging_migrations]

[15:17:38][Invoke deploy:staging_migrations]

[15:17:38][Step 3/3] Process exited with code 0

[15:17:38]Publishing internal artifacts

[15:17:38][Publishing internal artifacts] Sending build.finish.properties.gz file

[15:17:38]Build finished
0
Comment actions Permalink

Now i see problem.
Looks like task 'off' calling 'Bundler.with_clean_env' but exception inside that method printed into console and not thrown upper, so rake thinks that all is ok.
So the exit code is also zero.
So TeamCity dows not see any problem (because there no problem reported).

I suggest to modify rake script to not ignore exceptions.

Also. TC does not support ruby version setting via Gemfile.  You can use Ruby Environment Configurator build feature to set specific version of ruby. Also if you use rvm, you can add agent requirement on 'rvm.rubies.list' property.

0

Please sign in to leave a comment.