Why are Intern Functional Tests marked "ignored" by TeamCity when they run and pass

We are using Intern.io testing framework to run functional tests on our Ember web application. I currently have 11 tests running, and each one of them passes. Yet when the build finishes, TeamCity reports: Tests passed: 0, ignored: 11

What is causing TeamCity to think the tests are ignored?  Here is an excerpt from the build log:

Step 6/9] main (running for 35s)
[01:23:02][Step 6/9] main - Login (running for 35s)
[01:23:02][Step 6/9] main - Login - valid login should navigate to dashboard where name is displayed (running for 35s)
[01:23:02][Step 6/9] this.timeout 60000
[01:23:06][Step 6/9] PASS: main - Login - valid login should navigate to dashboard where name is displayed (3774ms)
[01:23:06][Step 6/9] main - Login - login with too short password should display validation error (running for 31s)
[01:23:14][Step 6/9] PASS: main - Login - login with too short password should display validation error (7673ms)
[01:23:14][Step 6/9] main - Login - login with invalid username should display validation error (running for 23s)
[01:23:22][Step 6/9] PASS: main - Login - login with invalid username should display validation error (7866ms)
[01:23:22][Step 6/9] 0/3 tests failed
[01:23:22][Step 6/9] main - Basic Clock In (running for 15s)
[01:23:22][Step 6/9] main - Basic Clock In - valid login should navigate to dashboard where name is displayed (running for 15s)
[01:23:29][Step 6/9] PASS: main - Basic Clock In - valid login should navigate to dashboard where name is displayed (7336ms)
[01:23:29][Step 6/9] main - Basic Clock In - clicking clock in button should display clock in dialog (running for 8s)
[01:23:30][Step 6/9] selector text: Project
[01:23:30][Step 6/9] PASS: main - Basic Clock In - clicking clock in button should display clock in dialog (376ms)
[01:23:30][Step 6/9] main - Basic Clock In - click save button should dismiss dialog and start time clock running (running for 8s)
[01:23:30][Step 6/9] PASS: main - Basic Clock In - click save button should dismiss dialog and start time clock running (458ms)
[01:23:30][Step 6/9] 0/3 tests failed
[01:23:30][Step 6/9] main - Clock In To Project (running for 7s)
[01:23:30][Step 6/9] main - Clock In To Project - valid login should navigate to dashboard where name is displayed (running for 7s)
[01:23:33][Step 6/9] PASS: main - Clock In To Project - valid login should navigate to dashboard where name is displayed (3033ms)
[01:23:33][Step 6/9] main - Clock In To Project - clicking clock in button should display clock in dialog (running for 4s)
[01:23:34][Step 6/9] selector text: Project
[01:23:34][Step 6/9] PASS: main - Clock In To Project - clicking clock in button should display clock in dialog (424ms)
[01:23:34][Step 6/9] main - Clock In To Project - clicking project selector should display list of projects (running for 4s)
[01:23:34][Step 6/9] PASS: main - Clock In To Project - clicking project selector should display list of projects (441ms)
[01:23:34][Step 6/9] main - Clock In To Project - selecting project should display project title on selector (running for 3s)
[01:23:34][Step 6/9] PASS: main - Clock In To Project - selecting project should display project title on selector (295ms)
[01:23:34][Step 6/9] main - Clock In To Project - click save button should dismiss dialog and start time clock running (running for 3s)
[01:23:35][Step 6/9] stop button text Clock Out
[01:23:35][Step 6/9] PASS: main - Clock In To Project - click save button should dismiss dialog and start time clock running (478ms)
[01:23:35][Step 6/9] 0/5 tests failed
[01:23:35][Step 6/9] 0/11 tests failed
[01:23:35][Step 6/9] Process exited with code 0
3 comments
Comment actions Permalink

This is an example of the service message block that Intern is producing:


##teamcity[testSuiteStarted name='main' startDate='Mon Jun 29 2015 10:06:40 GMT-0600 (MDT)' flowId='2' timestamp='2015-06-29T16:06:40.199']
##teamcity[testSuiteStarted name='main - Login' startDate='Mon Jun 29 2015 10:06:40 GMT-0600 (MDT)' flowId='3' timestamp='2015-06-29T16:06:40.201']
##teamcity[testStarted name='main - Login - clicking create account link should display form' flowId='4' timestamp='2015-06-29T16:06:40.202']
PASS: main - Login - clicking create account link should display form (8319ms)
##teamcity[testFinished name='main - Login - clicking create account link should display form' duration='8319' flowId='5' timestamp='2015-06-29T16:06:48.521']
##teamcity[testStarted name='main - Login - submitting empty form should prompt for username' flowId='6' timestamp='2015-06-29T16:06:48.522']
... [more tests reported in like manner]
##teamcity[testSuiteFinished name='main - Login' duration='10100' flowId='17' timestamp='2015-06-29T16:06:50.301']
1/6 tests failed
##teamcity[testSuiteFinished name='main' duration='10103' flowId='18' timestamp='2015-06-29T16:06:50.302']

According to the documentation, (https://confluence.jetbrains.com/display/TCD9/Build+Script+Interaction+with+TeamCity#BuildScriptInteractionwithTeamCity-ReportingTests), intern is following the correct procedure for reporting tests.

Tests should only be 'ignored' if an explicit ignore message is sent:

##teamcity[testIgnored name='testName'
 
message='ignore comment']


The question still remains: What is causing TeamCity to think the tests are ignored?

0
Comment actions Permalink

Hi Blaine,

It seems that the issue is caused by different flowIds. The messages in one block should have the same FlowId.

0
Comment actions Permalink

Thank you. So, just to be clear, the corrected flow id would be as follows:

##teamcity[testSuiteStarted name='main' startDate='Mon Jun 29 2015 10:06:40 GMT-0600 (MDT)' flowId='2' timestamp='2015-06-29T16:06:40.199']
##teamcity[testSuiteStarted name='main - Login' startDate='Mon Jun 29 2015 10:06:40 GMT-0600 (MDT)' flowId='3' timestamp='2015-06-29T16:06:40.201']
##teamcity[testStarted name='main - Login - clicking create account link should display form' flowId='4' timestamp='2015-06-29T16:06:40.202']
PASS: main - Login - clicking create account link should display form (8319ms)
##teamcity[testFinished name='main - Login - clicking create account link should display form' duration='8319' flowId='4' timestamp='2015-06-29T16:06:48.521']
##teamcity[testStarted name='main - Login - submitting empty form should prompt for username' flowId='5' timestamp='2015-06-29T16:06:48.522']
... [more tests reported in like manner]
##teamcity[testSuiteFinished name='main - Login' duration='10100' flowId='3' timestamp='2015-06-29T16:06:50.301']
1/6 tests failed
##teamcity[testSuiteFinished name='main' duration='10103' flowId='2' timestamp='2015-06-29T16:06:50.302']

I've brought the matter to the attention of the developers. When the issue has been resolved, I will mark the question answered.

0

Please sign in to leave a comment.