TeamCity Benchmark Plugin

Hi all,

After reading about the benchmark plugin on the jetbrains blog, I immediatly had to test it on my testing/staging instance. The test results where of course rather ... optimistic ... but nontheless interesting. Now my question is, if it would be possible to take this test further:

1.) Produce log and test results on a different machine, this way I can also test if my network interface is holding up, not just cpu, memory and disk. If I could, for example start two instances of TC, both using the same plugin, but one contains the build configurtion, while the other one hosts the virtual agents, traffic would have to flow via my actual network. Alternatively, if the agent part could be run in its own JVM, without the help of a separate TC instance it would be even better.

2.) Run continuesly - Currently the plugin creates all build configurations, all agents, starts the job everywhere and then starts sending logs and deletes everything afterwards. If instead agents and build configurations would stick around a bit, it would be possible to also see if the Queue can handle that many jobs and agents without issue. As in, keep creating many jobs, getting consumed by the virtual agents in parallel to handling the load.

Is there any ongoing support going to happen, or is the source available for extra experiments?

3 comments
Comment actions Permalink

Hello,

Thank you for trying this plugin and providing feedback, much appreciated.
Answering to your questions:
1) although in our experience network is rarely a bottleneck, we considered this possibility and theoretically it is possible to start the load generator on separate machine; it should not be hard to allow that
2) so you're speaking about a test which involves the whole workflow of the build, maybe it makes sense

Plugin is closed source, but we have plans to improve it, so it makes sense to post requests for this plugin in our tracker.

0
Comment actions Permalink

1) I guess this is not something I can do on my own? Should I (or could you) create a support ticket for this?
2) Yes. While already very useful for Benchmarking purposes (keeping the variables to a known set), I think it could be enhanced to be useable for System testing of the server. This is assuming those jobs and agents follow a "normal" lifecycle and fire all events. This way it would be possible to see if the server is still behaving properly under full load, and the impact custom plugins have on the overal performance of the server. Though to be more "realistic" not all jobs should create test failures ;)

0
Comment actions Permalink

1. Please watch for this request: https://youtrack.jetbrains.com/issue/TW-42592
2. Currently I am not convinced that this plugin should perform complete load testing because the main purpose of the plugin is to provide an answer on a number of agents suported by a specific server; there are some aspects that benchmark plugin does not measure: publishing of artifacts, publishing of statistic values, but start of the build and procesing of the queue seems out of scope.
Usually not many builds are starting at the same time because when server decides what build can be started, in real life the choice is limited because there are only 2, 3 idle compatible agents. From our experience processing of the queue does not produce significant load on the server. I am not saying there were no performance problems with it when server started builds too slowly, but these are just bugs which should be fixed.

0

Please sign in to leave a comment.