Why are Intern Functional Tests marked "ignored" by TeamCity when they run and pass
We are using Intern.io testing framework to run functional tests on our Ember web application. I currently have 11 tests running, and each one of them passes. Yet when the build finishes, TeamCity reports: Tests passed: 0, ignored: 11
What is causing TeamCity to think the tests are ignored? Here is an excerpt from the build log:
Step 6/9] main (running for 35s)
[01:23:02][Step 6/9] main - Login (running for 35s)
[01:23:02][Step 6/9] main - Login - valid login should navigate to dashboard where name is displayed (running for 35s)
[01:23:02][Step 6/9] this.timeout 60000
[01:23:06][Step 6/9] PASS: main - Login - valid login should navigate to dashboard where name is displayed (3774ms)
[01:23:06][Step 6/9] main - Login - login with too short password should display validation error (running for 31s)
[01:23:14][Step 6/9] PASS: main - Login - login with too short password should display validation error (7673ms)
[01:23:14][Step 6/9] main - Login - login with invalid username should display validation error (running for 23s)
[01:23:22][Step 6/9] PASS: main - Login - login with invalid username should display validation error (7866ms)
[01:23:22][Step 6/9] 0/3 tests failed
[01:23:22][Step 6/9] main - Basic Clock In (running for 15s)
[01:23:22][Step 6/9] main - Basic Clock In - valid login should navigate to dashboard where name is displayed (running for 15s)
[01:23:29][Step 6/9] PASS: main - Basic Clock In - valid login should navigate to dashboard where name is displayed (7336ms)
[01:23:29][Step 6/9] main - Basic Clock In - clicking clock in button should display clock in dialog (running for 8s)
[01:23:30][Step 6/9] selector text: Project
[01:23:30][Step 6/9] PASS: main - Basic Clock In - clicking clock in button should display clock in dialog (376ms)
[01:23:30][Step 6/9] main - Basic Clock In - click save button should dismiss dialog and start time clock running (running for 8s)
[01:23:30][Step 6/9] PASS: main - Basic Clock In - click save button should dismiss dialog and start time clock running (458ms)
[01:23:30][Step 6/9] 0/3 tests failed
[01:23:30][Step 6/9] main - Clock In To Project (running for 7s)
[01:23:30][Step 6/9] main - Clock In To Project - valid login should navigate to dashboard where name is displayed (running for 7s)
[01:23:33][Step 6/9] PASS: main - Clock In To Project - valid login should navigate to dashboard where name is displayed (3033ms)
[01:23:33][Step 6/9] main - Clock In To Project - clicking clock in button should display clock in dialog (running for 4s)
[01:23:34][Step 6/9] selector text: Project
[01:23:34][Step 6/9] PASS: main - Clock In To Project - clicking clock in button should display clock in dialog (424ms)
[01:23:34][Step 6/9] main - Clock In To Project - clicking project selector should display list of projects (running for 4s)
[01:23:34][Step 6/9] PASS: main - Clock In To Project - clicking project selector should display list of projects (441ms)
[01:23:34][Step 6/9] main - Clock In To Project - selecting project should display project title on selector (running for 3s)
[01:23:34][Step 6/9] PASS: main - Clock In To Project - selecting project should display project title on selector (295ms)
[01:23:34][Step 6/9] main - Clock In To Project - click save button should dismiss dialog and start time clock running (running for 3s)
[01:23:35][Step 6/9] stop button text Clock Out
[01:23:35][Step 6/9] PASS: main - Clock In To Project - click save button should dismiss dialog and start time clock running (478ms)
[01:23:35][Step 6/9] 0/5 tests failed
[01:23:35][Step 6/9] 0/11 tests failed
[01:23:35][Step 6/9] Process exited with code 0
Please sign in to leave a comment.
This is an example of the service message block that Intern is producing:
##teamcity[testSuiteStarted name='main' startDate='Mon Jun 29 2015 10:06:40 GMT-0600 (MDT)' flowId='2' timestamp='2015-06-29T16:06:40.199']
##teamcity[testSuiteStarted name='main - Login' startDate='Mon Jun 29 2015 10:06:40 GMT-0600 (MDT)' flowId='3' timestamp='2015-06-29T16:06:40.201']
##teamcity[testStarted name='main - Login - clicking create account link should display form' flowId='4' timestamp='2015-06-29T16:06:40.202']
PASS: main - Login - clicking create account link should display form (8319ms)
##teamcity[testFinished name='main - Login - clicking create account link should display form' duration='8319' flowId='5' timestamp='2015-06-29T16:06:48.521']
##teamcity[testStarted name='main - Login - submitting empty form should prompt for username' flowId='6' timestamp='2015-06-29T16:06:48.522']
... [more tests reported in like manner]
##teamcity[testSuiteFinished name='main - Login' duration='10100' flowId='17' timestamp='2015-06-29T16:06:50.301']
1/6 tests failed
##teamcity[testSuiteFinished name='main' duration='10103' flowId='18' timestamp='2015-06-29T16:06:50.302']
According to the documentation, (https://confluence.jetbrains.com/display/TCD9/Build+Script+Interaction+with+TeamCity#BuildScriptInteractionwithTeamCity-ReportingTests), intern is following the correct procedure for reporting tests.
Tests should only be 'ignored' if an explicit ignore message is sent:
The question still remains: What is causing TeamCity to think the tests are ignored?
Hi Blaine,
It seems that the issue is caused by different flowIds. The messages in one block should have the same FlowId.
Thank you. So, just to be clear, the corrected flow id would be as follows:
##teamcity[testSuiteStarted name='main' startDate='Mon Jun 29 2015 10:06:40 GMT-0600 (MDT)' flowId='2' timestamp='2015-06-29T16:06:40.199']
##teamcity[testSuiteStarted name='main - Login' startDate='Mon Jun 29 2015 10:06:40 GMT-0600 (MDT)' flowId='3' timestamp='2015-06-29T16:06:40.201']
##teamcity[testStarted name='main - Login - clicking create account link should display form' flowId='4' timestamp='2015-06-29T16:06:40.202']
PASS: main - Login - clicking create account link should display form (8319ms)
##teamcity[testFinished name='main - Login - clicking create account link should display form' duration='8319' flowId='4' timestamp='2015-06-29T16:06:48.521']
##teamcity[testStarted name='main - Login - submitting empty form should prompt for username' flowId='5' timestamp='2015-06-29T16:06:48.522']
... [more tests reported in like manner]
##teamcity[testSuiteFinished name='main - Login' duration='10100' flowId='3' timestamp='2015-06-29T16:06:50.301']
1/6 tests failed
##teamcity[testSuiteFinished name='main' duration='10103' flowId='2' timestamp='2015-06-29T16:06:50.302']
I've brought the matter to the attention of the developers. When the issue has been resolved, I will mark the question answered.