Automated benchmark system for game development (custom charts).


my name is Pawel Lekki from EXOR Studios - an independent game development studio. We use Team City for compiling our code (C++) for platforms such as Windows, Xbox 360, PS3 and Android. We also use it to build our game content packages and run automated tests on all of these platforms. We have actually used Team City 6.5 Professional for about a year when we worked on Zombie Driver HD, after that we were testing Jenkins for about 6 months and we are currently getting back to Team City 8 Enterprise.

I am currently working on improving our automated testing system and have a few bug reports, feature requests and questions how to construct some mechanisms. The best way to start is by showing you how our setup looks like - you can download .pdfs with print outs of our benchmark custom charts and sample corresponding teamcity-info.xml files from these links:

I have included a bit of old benchmark data from Zombie Driver HD to show you how we are planning to have multiple benchmarks from a single project and a print out from our current project that includes a lot more data in a single test (it will also have as many or even more separate benchmarks than Zombie Driver HD). We are running these benchmarks automatically on a daily basis or even more often depending on the daily changes.

My current issues with custom chart setup in Team City are:
1. Since migrating to Team City 8.02 and a MySQL database the system is truncating our floating point data results to integer values. This is a serious problem because we are running our performance tests at a 0.1ms error level. The integer values kill this precision and we are losing a lot of valuable data. We can always manually check the results in the .xml files, but this is not the point with such a robust system as Team City. We didn't have this problem earlier on TC 6.5 with a test HSQLDB (we actually used it for a year without problems).

2. The width of a chart should be constant or customizable - if you look at our Shooter benchmark results you will notice that the Performance chart is actually longer than the Memory chart. As an effect it is difficult to correlate specific build results from the Performance chart and the Memory chart. The problem is caused by the width of the Y axis labels.

3. The custom charts setup process is quite cumbersome. Adding each chart and data series in the main-config.xml file takes a lot of time and it would be great if it actually was not a manual process. As a comparison this works a lot better in Jenkins (using the Plot plugin) where I was able to specify an XML data file that was automatically parsed based on node names and new data series were added automatically. This way our programmers could simply add new benchmark metrics without consulting a system admin to have them displayed. I've tried searching for a similar solution for TC but couldn't find any. Can you recommend an automated solution for such an approach or implement this in future versions of Team City?

4. It would be great if it would be possible to customize the height of each chart. As you can see in our examples (Shooter-Synthetic1-Performance), some of our tests have broad data ranges and viewing them on such small charts is far from convenient. I know that it is possible to manually set the max/min Y values, but this is not an ideal solution because we want to compare the whole data range.

5. At the moment there is a nice option to dynamically zoom in on a chart in the X axis, it would be great if this would also be possible in the Y scale (by click-dragging a selection box around the area).

6. It would be great if it would be possible to change the shape of the data points for a specific data series e.g. diamonds for memory leak values. It would improve readability for complex charts.

I am looking forward to hearing your feedback on these points and possible solutions. It would be great if you could let me know if you have any recommendations on how we could improve our benchmarking setup.


Comment actions Permalink

Hello Pawel,

Thank you for feedback

  1. It's a known issue caused by HSQLDB bug. Actually we don't support values with floating point at the moment. The only workaround now is to multiply data by 10 before publishing. There is an issue for this:
  2. I understood your usecase. Right now it's impossible to configure chart width. I'll see how we can implement this.
  3. Yeap. This is tricky. We have several requests tracked and planned to handle:
  4. Didn't you try to use logarithmic scale? Regarding height adjusting I think I will implement it among with second issue.
  5. As I remember it's implemented for coming 8.1 version and will be integrated to 8.0.x version soon. Zooming on both axes will be possible holding Ctrl key.
  6. We do have this functionality in our code but there is no way to configure such things at the moment. You can manually set different colors for your series if you didn't yet.

Please file requests for problems 2 and 6 to our issue tracker so I could track them.

Thanks, Andrey

Comment actions Permalink

Hello Andrey,

Thank you for the answers, I'll report the issues as you suggested.

4. Unfortunately the logarythmic scale doesn't solve all problems for us. In some of our tests we have a lot of data series that have very similar results e.g. (15 sybsystems with just a milisecond time difference) and it would be much more convenient if we could simply expand some of the charts to be much bigger. This would make analyzing data much more convenient.

There is also a small issue with the data series display list for each chart - when the list is long checking/unchecking each box scrolls the list back to the top making it difficult to quickly show/hide a single data series at the bottom of the list.

Looking forward to TC 8.1 :)


Comment actions Permalink

There is also a small issue with the data series display list for each chart - when the list is long checking/unchecking each box scrolls the list back to the top making it difficult to quickly show/hide a single data series at the bottom of the list.

This annoying issue is tracked here:

Comment actions Permalink

Thanks Pawel!

I just found that actually we do have possibility to configure charts width and height through main-config.xml. This feature is not documented so I can't recommend to use it but it seems quite finished and shouldn't change. Just add "width" and "height" parameters to chart definition like this:

<graph title="The title">
    <property name="width" value="500" />
    <property name="height" value="1000" />


  <valueType key="ArtifactsSize" title="Artifacts Size" buildTypeId="BuildType1Id" />

Though this will not solve the problem with charts alignment it will be useful for charts containing many data.
Comment actions Permalink

I can confirm that these parameters are working and it's a big improvement in making our charts more readable. Thanks Andrey!


Please sign in to leave a comment.