Best approach to running additional tests during nightly build?

I have a build configuration for a .NET project.  It has two triggers - one VCS Trigger that runs the build after each commit, and another Schedule Trigger which runs a nightly build.  For this particular project, the integration tests (NUnit, if it matters) are painfully long-running (over an hour).  It is very disruptive to have all of these tests run after each commit, because it uses up a build agent for a long time while devs are actively committing to different projects.  So, I'd like to have those tests run only during the nightly build.  What is the best approach to setting this up?

Obviously, I could duplicate the build configuration, separating the two triggers into the different configurations, and add an NUnit step to the nightly configuration.  I'd like to avoid this for a couple reasons.  All of our other projects have only one configuration, so this would add some clutter.  And I don't want to have to duplicate the configuration as it will add to maintenance complexity.

Any advice is welcome.  Thanks.

4 comments
Comment actions Permalink

Justin,

I think your best option will be splitting it into two configurations, but in a slightly different manner than how you were thinking. What I'd suggest is creating a second configuration that has a dependency on your existing configuration, and this new configuration only contains the integration testing build steps. I know that you didn't want to clutter things up with an extra configuration, but I think this will actually be easier to maintain and make it easier to intepret where failures are happening.

On a high-level, here is how I would go about it:

1. On your existing configuration, under "General Settings", use the "Artifact Paths" to grab all of the assemblies and assets that you need for integration testing. This happens after all the builds steps have completed, so you can point this to your bin folders with the freshly built assemblies.This will package them up making them available for later use.
Here is an example of me grabbing the integration test assemblies and the functional test assemblies and placing them in the respective folders in a zipfile called tests.zip:

  • Source/Tests/IntegrationTests/bin/Release/ => tests.zip!/Integration/
  • Source/Tests/FunctionalTests/bin/Release/ => tests.zip!/Functional/

2. Create a new build configuration for your integration tests.
3. On the new configuration, under Dependencies, set up an Artifact Dependency. Set it to depend on your first configuration. Setting it to use the last successful build probably makes the most sense at this point. The "Artifact Rules" is where you tell it to get the artifacts you configured in step 1 and where to put them to be available for your tests. Here is an example Artifact Rule to extract the tests.zip artifact I created in step 1:

  • tests.zip!**=>BuildArtifacts/Tests/ (This takes my zipfile above and extracts the contents to BuildArtifacts/Tests/. It maintains the folder structure, so it contains the Integration and Functional folders that I set up earlier.

4. Move your integration nunit build step from your old configuration and into the new one. It should only exist in the new configuration.
5. On the old configuration, configure the Build Triggering to build after each commit.
6. On the new configuration, configure the Build Triggering to build on your nightly schedule.


What happens is that when the new configuration triggers, it'll look for the latest successful build that happened during the day, download the assets from it, and then run the integration tests against it. Some of the benefits of doing it this way are:

  • You can easily kick of the integration tests during the day manually if you ever need to.
  • You get visualization on the dashboard of whether it was your integration tests or your build / unit tests that are broken. If your configurations are combined, your builds that happen across the day will hide the integration testing results. (e.g. If your integration tests fail, in the morning when a developer commits a change that results in a succesful build, your dashboard will show green, even though the integration tests are still broken)
  • The histories and metrics of your integration tests will be separate from your build/unit tests.


Let me know if you need an more details or if you have any questions.

0
Comment actions Permalink

Thanks Joel that seems like a good idea.  I will pursue that solution.

0
Comment actions Permalink

Thank you for so detailed answer, Joel

0
Comment actions Permalink

Hi Justin

Technicaly, there is a way to determine how the build was triggered - Groovy plugin provides build.triggeredBy property.

But in your scenario we do not recommend to use it. As Joel said:

If your integration tests fail, in the morning when a developer commits a change that results in a succesful build, your dashboard will show green, even though the integration tests are still broken

0

Please sign in to leave a comment.