Automated benchmark system for game development (custom charts).


my name is Pawel Lekki from EXOR Studios - an independent game development studio. We use Team City for compiling our code (C++) for platforms such as Windows, Xbox 360, PS3 and Android. We also use it to build our game content packages and run automated tests on all of these platforms. We have actually used Team City 6.5 Professional for about a year when we worked on Zombie Driver HD, after that we were testing Jenkins for about 6 months and we are currently getting back to Team City 8 Enterprise.

I am currently working on improving our automated testing system and have a few bug reports, feature requests and questions how to construct some mechanisms. The best way to start is by showing you how our setup looks like - you can download .pdfs with print outs of our benchmark custom charts and sample corresponding teamcity-info.xml files from these links:

I have included a bit of old benchmark data from Zombie Driver HD to show you how we are planning to have multiple benchmarks from a single project and a print out from our current project that includes a lot more data in a single test (it will also have as many or even more separate benchmarks than Zombie Driver HD). We are running these benchmarks automatically on a daily basis or even more often depending on the daily changes.

My current issues with custom chart setup in Team City are:
1. Since migrating to Team City 8.02 and a MySQL database the system is truncating our floating point data results to integer values. This is a serious problem because we are running our performance tests at a 0.1ms error level. The integer values kill this precision and we are losing a lot of valuable data. We can always manually check the results in the .xml files, but this is not the point with such a robust system as Team City. We didn't have this problem earlier on TC 6.5 with a test HSQLDB (we actually used it for a year without problems).

2. The width of a chart should be constant or customizable - if you look at our Shooter benchmark results you will notice that the Performance chart is actually longer than the Memory chart. As an effect it is difficult to correlate specific build results from the Performance chart and the Memory chart. The problem is caused by the width of the Y axis labels.

3. The custom charts setup process is quite cumbersome. Adding each chart and data series in the main-config.xml file takes a lot of time and it would be great if it actually was not a manual process. As a comparison this works a lot better in Jenkins (using the Plot plugin) where I was able to specify an XML data file that was automatically parsed based on node names and new data series were added automatically. This way our programmers could simply add new benchmark metrics without consulting a system admin to have them displayed. I've tried searching for a similar solution for TC but couldn't find any. Can you recommend an automated solution for such an approach or implement this in future versions of Team City?

4. It would be great if it would be possible to customize the height of each chart. As you can see in our examples (Shooter-Synthetic1-Performance), some of our tests have broad data ranges and viewing them on such small charts is far from convenient. I know that it is possible to manually set the max/min Y values, but this is not an ideal solution because we want to compare the whole data range.

5. At the moment there is a nice option to dynamically zoom in on a chart in the X axis, it would be great if this would also be possible in the Y scale (by click-dragging a selection box around the area).

6. It would be great if it would be possible to change the shape of the data points for a specific data series e.g. diamonds for memory leak values. It would improve readability for complex charts.

I am looking forward to hearing your feedback on these points and possible solutions. It would be great if you could let me know if you have any recommendations on how we could improve our benchmarking setup.



Please sign in to leave a comment.