Best Practices for Platform Matrix?

The report that I want to generate for my management team is one that will give them an accurate picture of the current state of our software in each of our supported platforms ( OS x Database x Application Server x Ant Target ... )

What are the best practices that I should know about in setting up a project for this?

From my initial review, it looks like my best bet is going to be to have a configuration for each target in the matrix. I should be able to save some headache by creating a dummy/smoke configuration that the other targets depend on. To minimize the duplication, I should parameterize as much of the matrix as I can, where each build configuration defines the ant properties specific to its place in the matrix.

Is there a good solution for managing the duplication among these build configurations?

Is there a good solution for assembling the results of multiple build configurations into a single report?

What keywords should I have searched for to find previous discussions of this problem?

What question would I know to ask if I knew more about the problem than I do now?

5 comments

I'd like to know too.  Ideally, we could re-use the same build configuration and just vary the parameters (requires linux versus requires windows, run quick tests for every run versus run full tests only nightly, etc...)  Is this possible with TeamCity?  How does JetBrains set up their build and test configurations?

- Kevin

0

So here is how we do it:


I'm looking for pointers as well

Matrix Creation:

For us, our matrix is based on platform + jdk + appserver + database+architecture

I have around 35 build agents.  They consist of windows, solaris, aix, rhel 5, and suse.   There are many windows flavors mixed in 2000, 2003, 2008....  There are at least one 32 bit and one 64 bit agent of each os flavor.

On each build agent I install sql server and oracle.  This covers the appserver portion.
On each build agent I install a 32 bit jdk and 64 bit jdk if applicable.
On each build agent I install jboss and other app servers if applicable.

The net here is that 1 build agent may =  > 1 platforms.

e.g  the same build agent run  Windows 2003 x64 + SQL Server + jdk 1.6 32 bit + JBoss and
Windows 2003 x64 + SQL Server + jdk 1.6 32 64 bit + JBoss and
Windows 2003 x64  + Oracle + jdk 1.6 32 64 bit + JBoss and

On each worker I setup environment variables which describe it's OS, architecture, and so on.

Next I create a plethora of build configs.  Each config is named for the platform it represents.   This part really sucks.  We have around 30 configs to manage.
For each config, I have to limit which agents it runs on based on the agent environment variables i set up above.   I also must set a bunch of ant properties which tell that particular
config which database and appserver to run on.   The build system is then responsible for running it.

I set up an installer build configuration.   This will build our installable project.
We have around 30 build configurations which spawn are triggered based on installer creation.   Each of these build configurations will run our installer, test it, and produce test results.

Teamcity doesn't have a good way of aggregating test results into a top level portal.   We've written a homegrown app to handle this.     Each platform build config makes a call to our homegrown app after it runs and tells what build it tested and what the results were.    The homegrown app then can display the platform matrix and its current results.

The only ci system i know of that has support for this sort of thing is hudson, and its a relatively new feature.

http://ericlefevre.net/wordpress/2007/07/29/matrix-project-building-with-hudson/

0

Thanks for the excellent feedback!

ruelloehr wrote:

...
Next I create a plethora of build configs.  Each config is named for the platform it represents.   This part really sucks.  We have around 30 configs to manage.


Darn, I was hoping to avoid creating an maintaining a build configuration for each platform and test configuration.

Teamcity doesn't have a good way of aggregating test results into a top level portal.   We've written a homegrown app to handle this.     Each platform build config makes a call to our homegrown app after it runs and tells what build it tested and what the results were.    The homegrown app then can display the platform matrix and its current results.


The only ci system i know of that has support for this sort of thing is hudson, and its a relatively new feature.


http://ericlefevre.net/wordpress/2007/07/29/matrix-project-building-with-hudson/


Interesting.  Is your homegrown app a TeamCity plugin or just a separate app on the side?

Ah!  It just occured to me that we could queue up downstream builds using the HTTP GET plus extra parameters feature of TeamCity.  Then you only need one build configuration that knows how to deploy and run against all of the various platforms, databases, and test suites.  It seems that TeamCity should provide a way to queue downstream builds with parameters directly from the 'Build Triggering' tab.

Thanks again for your feedback.

0

Neat.  I didn't know about the url trick.     I'll have to file that away.

Our app isn't a plugin.  It's a self contained web app.

That being said, the plugin system teamcity offers is pretty nice.   Concievably, you could implement your own tab that displayed the information. 

The reason I chose to go a seperate app is as follows.

We wished to store data about a build (e.g. jdk used, os used and so on) in a database so that we could later retrieve and view it.   Since a build config isn't really aware of what os, jdk, db ... it is using, i rely on custom ant task to send me that info and store it in an external database.

0

Guys, I think you should vote for some of these issues (probably you've already done this before ):
http://www.jetbrains.net/tracker/issue2/TW-3661
http://www.jetbrains.net/tracker/issue2/TW-4940
http://www.jetbrains.net/tracker/issue2/TW-5033
http://www.jetbrains.net/tracker/issue2/TW-3350

We hope to address build configurations maintenance in the next major release (TeamCity 5.0).

As for our own configuration, we need to test TeamCity on several platforms (Windows, Linux, Solaris) and against different databases (MySQL, HSQLDB, PSQL, MSSQL, Oracle). We decided that we do not need to test each combination. Now we have separate build configurations for all DB engines, and two platform specific configurations for Linux and Solaris. This is enough to catch most of the bugs.

We also considered a possibility to run builds with different platforms under the same build configuration. However this has some disadvantages: if something is broken on one platform only you will need more time to find which change is the cause because your history contains builds from different platforms. Another problem: it is not easy in TeamCity to start build on several platforms at once. This can be done with help of separate build configuration or with help of plugin, which is not convenient.

Anyway, we understand that improvements are needed, and hope to fix some critical issues in the next release.

0

Please sign in to leave a comment.