Need help to get TeamCity to report the results of my JMeter tests correctly

Answered

I am attempting to run JMeter tests from TeamCity and am not getting it to report correctly. 

I have reviews these two articles:
1: https://devblog.xero.com/run-jmeter-performance-tests-on-teamcity-8315f7ccffc1#.vk6k8n6xm
2: https://devblog.xero.com/run-jmeter-performance-tests-on-teamcity-8315f7ccffc1#.vk6k8n6xm

As well as reviewing the JMeter plugin for TeamCity located here: https://github.com/jtorgan/jmeter_plugin/blob/master/readme.txt#L36

Currently my tests are running from TeamCity and are creating a logfile (.jtl), however the results are not being reported correctly in TC.  I have a POST that is sending back a 500 and in JMeter fails.  However, in TeamCity it reports as a pass.  Let me also say that this test plan has 7 tests.  It only runs 4 then stops.  The 4th one fails in JMeter, TC reports it as a pass, but does not continue to run the remaining 3 tests.  I believe this is due to the 4th test actually failing and it stopping on the fail.

If I go into TeamCity > Build Features where I have Performance Metrics Calculation and place a check in Assertions in Format Settings, all my tests fail.  If I remove the check in Assertions, then all my tests pass.  Placing and removing Response Assertion in JMeter does not seem to affect this.  I also have placed and removed Generate Summary Results and Assertion Results in order to see if I can get it to report correctly.   So far none of these changes have been able to show the correct results from TC which should be 5 passes and 2 fails.  (my POST and GET)

Does anyone have any insight as to what I can do in order to get TeamCity to report the results of my JMeter tests correctly?

Thanks in advance
Todd

4 comments

Hello, Todd,

I’d like to notice that you mentioned the same article twice, so I cannot give you the full answer.

The problem with tests stopping right after one of them had failed should be somewhere in test runner, not in the plugin. For example, in nosetests there is a parameter -x (--stop). For jmeter as the test runner I don’t know the answer. Probably you may try different values for «Action to be taken after a Sampler error (Thread Group)» or use Maven parameter ignoreResultFailures (https://github.com/jmeter-maven-plugin/jmeter-maven-plugin/wiki/Test-Results-File-Format) if you’re using jmeter-maven-plugin.
Also if your tests start in different build steps on TeamCity (TC/Build Configuration Settings/Build steps), you should set value «Even if some of the previous steps failed» to parameter «Execute step» in build configuration settings for these steps. If there are several command line commands in one build step, check that all of them return zero exit code.

As for TC reporting, this problem should be investigated separately. Please check the following:
1) is there a «Performance Statistics» tab on build page in TeamCity and are there any plots? Please attach a screenshot with TeamCity Build Features settings
2) is there a «Performance tests statistiс» step in the end of build log?
3) is your log file (.jtl) properly formatted (based on https://github.com/jtorgan/jmeter_plugin/blob/master/readme.txt#L78)? Please give an example of this log.
4) check agent log file buildAgent/logs/teamcity-agent.log for errors/exceptions/additional info

Yours respectfully,
Kesarev Sergey.

0

Hi! I find your plugin and trying to use it, but have some problems. It doesn't parse report *.jtl as it should.

 

In your documentation you give example of file https://github.com/jtorgan/jmeter_plugin/blob/master/readme.txt#L106

And we have jmeter CSV Log Format here http://jmeter.apache.org/usermanual/listeners.html#csvlogformat

 

1. Why tabs separators, while the default is comma?

2. Why did you changed the order of columns in report? You wait "success" after "responseCode", while at jmeter doc its clearly said that the order of appearance of columns is fixed and "success" is after "dataType". Now it lloks like I need to use some awk-magic to fix this.. :(

1

Hello, Max,

You can use these jmeter properties in jmeter.properties (or user.properties) file, according to the http://jmeter.apache.org/usermanual/listeners.html#defaults:

jmeter.save.saveservice.output_format=
jmeter.save.saveservice.default_delimiter=

Yours respectfully,
Kesarev Sergey.

-1

Sergey, thanks! This is workaround for csv separator, but why reorder?))

I have to reorder columns with this:
awk -F "\t" 'BEGIN {OFS = FS} {print $1,$2,$3,$4,$8,$5,$6,$7,$9,$10,$11,$12}' log.jtl > report.jtl

Only this magic allows me to enable "used TeamCity tests format" checkbox to see real count of my tests

1

Please sign in to leave a comment.