EMMA - Unsteady coverage results

Hi!

We are using EMMA as code coverage tool in our build.

We noticed that the coverage results are really unsteady. Sometimes, classes are covered, but in the next build, they aren't covered (without changing anything in the code).

Is there anything we can do in order to get stable results from EMMA?

Regards,
Markus Bischof

12 comments

Coverage can be unstable because some code is not always working the same way. For example, there can be code which tries to allocate some resource and if allocation fails, tries again. In some build you may have successful resource allocation from the first attempt, while in other build - it is not. In these cases coverage can differ.

Also coverage can differ if there is some platform specific code.

0

Hi Pavel,

thanks for your reply.

I understand that code coverage differs in some cases. But that's not really my problem.

I can give you an example: TeamCity tells me, that a testcase XY was successful. This testcase uses an instance of class A. But the EMMA coverage report tells me, that class A is not covered (class-coverage). And some buils later, the EMMA coverage report tells me, that the class is covered...that's really confusing...

Markus

0

Probably this is a bug in EMMA. In fact we are moving to use IDEA coverage engine, because EMMA is not supported anymore by its authors. IDEA coverage engine is available starting with TeamCity 5.0. So if possible, I would recommend to upgrade to TC 5.0.

0

Right now I'm running into OutOfMemoryErros when using EMMA...let's switch to IDEA coverage!


Actually, I tried to use IDEA code coverage, but I get the following log error:

Publishing artifacts

[16:42:48]: [Publishing artifacts] Paths to publish: [....MYPATH....\coverage.ic]
[16:42:48]: [Publishing artifacts] Publishing artifacts 'coverage.ic' to root artifacts directory
[16:42:48]: [Publishing artifacts] Artifacts path coverage.ic not found
[16:42:48]: Failed to generate coverage report, error: java.io.FileNotFoundException: coverage.ic file does not exist


Do I have to install additional packages/plugins to get that to work? Or do I miss some configuration parameters? Perhaps I have to put coverage.ic into my artifacs?

(I'm using TeamCity 5.0.2)

Regards,
Markus

0

No, you don't need to install anything. Could you please check is there a coverage-error.log file in the working directory on the agent? Also please provide your build log. If it contains sensitive information you can send it to teamcity-feedback@jetbrains.com

0

Hi Pavel,

hm, I would prefer not to send my build log (it really contains sensitive data). But I can provide the part of my ANT-Buildfile which is used in the run-configuration (I've attachted it).

In my build log, I see the following line right when the runTests-target is called:

[runTests] IDEA Coverage: Java agent added to JUnit task

But after the tests did run, I still get the error:

...
[07:29:26]: [antcall] clean

[07:29:26]: [clean] delete
[07:29:26]: [delete] Deleting directory ...BIN_DIRECTORY... // <<- this ist not the workdir...
[07:29:26]: Process exit code: 0
[07:29:26]: Publishing artifacts
[07:29:26]: [Publishing artifacts] Paths to publish: [...WORKDIR...\coverage.ic]
[07:29:26]: [Publishing artifacts] Publishing artifacts 'coverage.ic' to root artifacts directory
[07:29:26]: [Publishing artifacts] Artifacts path coverage.ic not found
[07:29:26]: Failed to generate coverage report, error: java.io.FileNotFoundException: coverage.ic file does not exist
[07:29:26]: Publishing artifacts
...

I do not find any coverage-error.log file in the WORKDIR folder, nor in the buildAgent folder or the .BuildServer folder. Furthermore, I do not see any more IDEA log outputs in the build log.

Can you see any problems with my ant-script?

Sorry that I'm not able to sent my build log...

Kind regards,
Markus


P.S.: In the buildAgent/log/teamcity-agent.log file, I see the following error:

[2010-03-18 07:56:48,121]   INFO -    jetbrains.buildServer.AGENT - Publishing artifacts 'coverage.ic' to root artifacts directory
[2010-03-18 07:56:48,121]   WARN -    jetbrains.buildServer.AGENT - Artifacts path coverage.ic not found
[2010-03-18 07:56:48,121]   INFO -    jetbrains.buildServer.AGENT - Done publishing artifacts ...WORKDIR...\coverage.ic, total files published: 0
[2010-03-18 07:56:48,121]   WARN -    jetbrains.buildServer.AGENT - Failed to generate coverage report
java.io.FileNotFoundException: coverage.ic file does not exist
at jetbrains.buildServer.coverage.idea.IDEACoverageArtifactsProvider.createCoverageData(IDEACoverageArtifactsProvider.java:118)
at jetbrains.buildServer.coverage.idea.IDEACoverageArtifactsProvider.beforeBuildFinish(IDEACoverageArtifactsProvider.java:76)
at sun.reflect.GeneratedMethodAccessor21.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at jetbrains.buildServer.util.EventDispatcher.dispatch(EventDispatcher.java:110)
at jetbrains.buildServer.util.EventDispatcher$2.invoke(EventDispatcher.java:61)
at $Proxy8.beforeBuildFinish(Unknown Source)
at jetbrains.buildServer.agent.impl.FinishBuildProcessor.finishBuild(FinishBuildProcessor.java:40)
at jetbrains.buildServer.agent.impl.BuildRunAction$1$1.run(BuildRunAction.java:62)
at jetbrains.buildServer.agent.impl.GuardedAction.runGuarded(GuardedAction.java:29)
at jetbrains.buildServer.agent.impl.BuildRunAction$1.run(BuildRunAction.java:60)
at java.lang.Thread.run(Thread.java:595)


Attachment(s):
antsnapshop.txt.zip
0

What include / exclude patterns do you use?

0

First, I used the default "*" as include pattern and none as exclude pattern.

Then I tried to use one of my package names as include pattern, which really contains classes: "company.project.*"
As exclude pattern, I used the package name which contained the tests: "company.project.unitTests.*"

Unfortunately, I had the same error result.

0

Do you run your tests in separate JVM (is fork enabled)? If not please try to enable fork.

0

Great!

Using junit with fork="yes" solved the issue!

Thank you very much for your help.

0

coverage-stats.png

Finally, we have steady coverage results!

0

Congratulations! I think we need to add some note on the web UI about fork requirement.

0

Please sign in to leave a comment.