xcode test reporting problems after upgrade to 9.0.1

Hi!

Recently I've upgraded my TeamCity from 8.x to 9.0.1, as soon as I did that TeamCity started showing problems in reporting the execution of xctests of my xcode project.
While there are 620 executed tests TeamCity only reports 30, under the "tests" tab I see that indeed there are only 30 tests, but that the last two tests has multiple runs (I attached a screenshot, uploading images doesn't seem to work).

I've tried digging into the xcode output but everything seems fine, I get test execution lines like the following -

Test Case '-[BillGuardUtilsTest testFormatNumberWithoutFractionalPart]' started. Test Case '-[BillGuardUtilsTest testFormatNumberWithoutFractionalPart]' passed (0.000 seconds). [11:11:36] :[BillGuardUtilsTest] testFormatThousandsNumberWithNegativeNumber [11:11:36] :[testFormatThousandsNumberWithNegativeNumber] [Test Output] Test Case '-[BillGuardUtilsTest testFormatThousandsNumberWithNegativeNumber]' started. Test Case '-[BillGuardUtilsTest testFormatThousandsNumberWithNegativeNumber]' passed (0.000 seconds). [11:11:36] :[BillGuardUtilsTest] testFormatThousandsNumberWithNegativeNumberWithoutFractionalPart [11:11:36] :[testFormatThousandsNumberWithNegativeNumberWithoutFractionalPart] [Test Output]


And in the end I get the summary -

[11:11:36] :[BillGuardUtilsTest]       Executed 9 tests, with 0 failures (0 unexpected) in 0.003 (0.005) seconds [11:11:36] :[BillGuardUtilsTest]       Executed 620 tests, with 0 failures (0 unexpected) in 8.619 (8.848) seconds [11:11:36] :[BillGuardUtilsTest]       Executed 620 tests, with 0 failures (0 unexpected) in 8.619 (8.849) seconds [11:11:36] :[BillGuardUtilsTest] Process exited with code 0 [11:12:00]i:[BillGuardUtilsTest] ##teamcity[buildStatisticValue key='buildStageDuration:buildStepRUNNER_6' value='146007.0']


Small note - if I look at old test executions that happened on TeamCity 8.x version I had a similar issue but instead of aggregating the missing tests into an existing test TeamCity reported the tests as "unknown test name".

Would appreciate help in understanding this issue as everything seems to execute fine.

Thanks,
Nimrod Gutman



Attachment(s):
Screen Shot 2015-01-01 at 11.39.15 AM.png
15 comments
Comment actions Permalink

I think this case is described in the Upgrade Notes: https://confluence.jetbrains.com/display/TCD9/Upgrade+Notes

0
Comment actions Permalink

Thanks,

So apparently the real issue I'm having is that most of the tests doesn't get a name.
Any reason for that behaviour?

One thing I did notice is that when the xcode plugin is running the tests it looks like each test is ran inside the "context" of the previous test - it's visible in the indentation of the output and in the scope of the test (also visible from the screenshot)

0
Comment actions Permalink

Hey guys,

I'm bumping this as it's still an issue for us - test names are not being recognized using the "native" xcode builder in TeamCity.
Is anyone else experiencing problems with that?

Thanks,
Nimrod

0
Comment actions Permalink

We get names for our test, but the test count is reduced from >1000 to just 34. This is mainly because most of the tests runing as child of another test for TeamCity. Teamcity is stacking tests as a testsuite even if the don't have the same name. they only resides in the same folder.
The problem with tests having no name we'd only have with TeamCity 8.1.x

0
Comment actions Permalink

We have experienced similar problems to the ones above.

We were running 500 plus tests sucessfully using Kiwi 2.2.4 and SenTestingKit framework in TeamCity 8.x and when the tests failed, it gave us the correct number of tests passed and failed and enabled us  to drill down to  specific tests on the overview tab.

However due to Kiwi 2.2.4 not building in iOS 8.1, we are required to use Kiwi 2.3.1 which only uses the XCTest framework and I have built and deployed a version with this in to TeamCity

On TeamCity 8.x, the tests run and it gives us the correct number of tests passed and failed but the failed tests themselves are <unknown test name> and don't allow a drill down to actually see them on the overview tab.

On TeamCity 9.0.1 the tests run and it gives us the incorrect number of tests passed and failed (seems to be aggregating them in some way) and the failed tests are incorrectly named but you can drill down to actually see them on the overview tab.

Is there a fix in the pipeline or is there a work around?

0
Comment actions Permalink

So it seems that TeamCity always nested xcode tests and it didn't have any affect in 8.0.1 because of the 'null test name' but in 9.0.1 it lowered the total number of tests...

I'm repeating the last question - anyone in JetBrains aware of this problem? any fixes planned? are we doing something wrong?

Thanks,
Nimrod

0
Comment actions Permalink

Hi Nimrod,

I apologize for the delay in reply caused by holidays. Could you please provide us with full build log and give examples of tests names that were reported by TeamCity and expected tests names?

0
Comment actions Permalink

Hi Alina,

Thanks for the reply - do you have an email I can send the logs and the tests names to?
I'd prefer not to upload the build log to the forum, not sure what kind of information we're printing there.

Best regards,
Nimrod

0
Comment actions Permalink

Just emailed everything you asked, thanks!

0
Comment actions Permalink

I have also uploaded our problematic logs with XCTest for both Team City 8 and TeamCity 9 to your ftp server under my name.

0
Comment actions Permalink

Nimrod and Robert, thank you, we received the logs. The investigation will require some time.

0
Comment actions Permalink

Alina, any progress on this issue yet?

0
Comment actions Permalink

Hi Robert,

Sorry for delay. We've found and fixed the issue https://youtrack.jetbrains.com/issue/TW-39793. The cause is "Test Suite finished" message was changed to "Test Suite passed" in Xcode 6. The fix will be available in TeamCity 9.0.2 which is planning to be released the next week.

0
Comment actions Permalink

Thank you Alina!

Looking forward for 9.0.2.

0

Please sign in to leave a comment.