Ant Junit report import ignores errors

Answered

When importing an Ant JUnit test result, testsuites where errors have occurred are ignored.  I would like those tests to be reported as having encountered an error and the build to fail.  I'm not sure if this is due to the xml format being incorrect, I noticed that when errors occur the xml indicates 0 tests:

<testsuite errors="7" failures="0" hostname="127.0.0.1" name="foo"package="bar" skipped="0" tests="0" time="0" timestamp="2016-02-18T16:37:05">

Has anybody encountered this sort of problem?  Not sure it's helpful but this is testing a legacy ColdFusion system, using their ant runner.

8 comments

Hello Andrew,

What TeamCity version is used? How did you generate the report? Could you please attach full xml report?

0

Hi Alina,

Thanks for getting back to me.  I used TeamCity version: TeamCity Enterprise 8.1 (build 29879)

The report is generated using an antRunner written in ColdFusion, this is supplied by testbox, I'm no ColdFusion developer so don't really understand it.

I've redacted some info out of the report, but this contains an example of passing tests and ones that have encountered errors.  You will notice the lack of test elements for the suite that had the error.  TeamCity is equivalently reporting this as a success with 3 tests passed.

<testsuites>
<testsuite tests="0" failures="0" errors="7" skipped="0" time="1.187" timestamp="2016-02-19T15:53:28" hostname="xxx" package="_tests.suites.framework.routing" name="_tests.suites.framework.routing">
<properties>
<property name="supportedlocales" value=",Chinese (China),Chinese (Hong Kong),Chinese (Taiwan),Dutch (Belgian),Dutch (Standard),English (Australian),English (Canadian),English (New Zealand),English (UK),English (US),French (Belgian),French (Canadian),French (Standard),French (Swiss),German (Austrian),German (Standard),German (Swiss),Italian (Standard),Italian (Swiss),Japanese,Korean,Norwegian (Bokmal),Norwegian (Nynorsk),Portuguese (Brazilian),Portuguese (Standard),Spanish (Mexican),Spanish (Modern),Spanish (Standard),Swedish,ar,ar_AE,ar_BH,ar_DZ,ar_EG,ar_IQ,ar_JO,ar_KW,ar_LB,ar_LY,ar_MA,ar_OM,ar_QA,ar_SA,ar_SD,ar_SY,ar_TN,ar_YE,be,be_BY,bg,bg_BG,ca,ca_ES,cs,cs_CZ,da,da_DK,de,de_AT,de_CH,de_DE,de_GR,de_LU,el,el_CY,el_GR,en,en_AU,en_CA,en_GB,en_IE,en_IN,en_MT,en_NZ,en_PH,en_SG,en_US,en_ZA,es,es_AR,es_BO,es_CL,es_CO,es_CR,es_CU,es_DO,es_EC,es_ES,es_GT,es_HN,es_MX,es_NI,es_PA,es_PE,es_PR,es_PY,es_SV,es_US,es_UY,es_VE,et,et_EE,fi,fi_FI,fr,fr_BE,fr_CA,fr_CH,fr_FR,fr_LU,ga,ga_IE,hi,hi_IN,hr,hr_HR,hu,hu_HU,in,in_ID,is,is_IS,it,it_CH,it_IT,iw,iw_IL,ja,ja_JP,ja_JP_JP_#u-ca-japanese,ko,ko_KR,lt,lt_LT,lv,lv_LV,mk,mk_MK,ms,ms_MY,mt,mt_MT,nl,nl_BE,nl_NL,no,no_NO,no_NO_NY,pl,pl_PL,pt,pt_BR,pt_PT,ro,ro_RO,ru,ru_RU,sk,sk_SK,sl,sl_SI,sq,sq_AL,sr,sr_BA,sr_BA_#Latn,sr_CS,sr_ME,sr_ME_#Latn,sr_RS,sr_RS_#Latn,sr__#Latn,sv,sv_SE,th,th_TH,th_TH_TH_#u-nu-thai,tr,tr_TR,uk,uk_UA,vi,vi_VN,zh,zh_CN,zh_HK,zh_SG,zh_TW"/>
<property name="appserver" value="Tomcat"/>
<property name="productlevel" value="Developer"/>
<property name="productversion" value="11,0,03,292480"/>
<property name="updatelevel" value="03"/>
<property name="productname" value="ColdFusion Server"/>
<property name="servicetagattributes" value="{cfcollection=action,categories,collection,engine,language,name,path, cfimap=accept,action,attachmentpath,connection,debug,delimiter,folder,generateuniquefilenames,maxrows,messagenumber,name,newFolder,password,port,recurse,secure,server,startrow,stopOnError,timeout,uID,username, cffeed=action,columnMap,enclosureDir,escapeChars,ignoreEnclosureError,name,outputFile,overwrite,overwriteEnclosure,properties,proxyPassword,proxyport,proxyserver,proxyUser,query,source,timeout,userAgent,xmlVar, cfstoredproc=blockfactor,cachedafter,cachedwithin,cacheId,cacheRegion,clientInfo,datasource,dbname,dbtype,debug,fetchClientInfo,maxrows,name,ormOptions,password,procedure,result,returncode,sql,timeout,username, cfldap=action,attributes,clientCert,clientCertPassword,delimiter,dn,filter,maxrows,modifytype,name,password,port,rebind,referral,returnAsBinary,scope,secure,separator,server,sort,sortControl,start,startrow,timeout,userName,useTLS, cfdbinfo=blockfactor,cachedafter,cachedwithin,cacheId,cacheRegion,clientInfo,datasource,dbname,dbtype,debug,fetchClientInfo,maxrows,name,ormOptions,password,pattern,result,sql,table,timeout,type,username, cfindex=action,attributecollection,autocommit,body,category,categorytree,collection,collectionnames,criteria,custom1,custom2,custom3,custom4,dihconfigpath,docboost,extensions,external,fieldboost,indexMode,key,language,prefix,query,recurse,status,title,type,urlpath, cfhttp=authType,charset,clientCert,clientCertPassword,columns,compression,delimiter,domain,file,firstRowAsHeaders,getAsBinary,method,multipart,multipartType,name,password,path,port,proxyPassword,proxyport,proxyserver,proxyUser,redirect,resolveurl,result,textqualifier,throwonerror,timeout,url,useragent,username,workstation, cfquery=blockfactor,cachedafter,cachedwithin,cacheId,cacheRegion,clientInfo,datasource,dbname,dbtype,debug,fetchClientInfo,maxrows,name,ormOptions,password,result,sql,timeout,username, cfftp=action,actionParam,alloSize,asciiextensionlist,bufferSize,connection,debug,directory,existing,failifexists,fingerPrint,item,key,localfile,name,new,passive,passphrase,password,port,proxybypass,proxyserver,remotefile,result,retrycount,secure,server,stoponerror,systemType,timeout,transfermode,username, cfmail=bcc,cc,charset,debug,encrypt,encryptionAlgorithm,failTo,from,group,groupcasesensitive,keyalias,keyPassword,keystore,keystorePassword,mailerid,maxrows,mimeattach,password,port,priority,query,recipientCert,remove,replyTo,server,sign,spoolenable,startrow,subject,timeout,to,type,username,useSSL,useTLS,wrapText, cfSearch=category,categorytree,collection,contextBytes,contextHighlightBegin,contextHighlightEnd,contextPassages,criteria,external,language,maxrows,name,orderby,previousCriteria,searchcolumns,searchtime,startrow,status,suggestions,type, cfpdf=action,addQuads,algo,align,ascending,author,bottomMargin,charset,compressTiffs,copyfrom,ddxFile,destination,directory,encodeAll,encrypt,flatten,foreground,format,height,hiRes,honourSpaces,hScale,image,imageprefix,info,inputFiles,isBase64,jpgDpi,keepBookmark,keyalias,keyPassword,keystore,keystorePassword,leftMargin,maxBreadth,maxLength,maxScale,name,newOwnerPassword,newUserPassword,noAttachments,noBookmarks,noComments,noFonts,noJavascripts,noLinks,noMetadata,noThumbnails,numberFormat,opacity,order,outputFiles,overridePage,overwrite,package,pages,password,permissions,position,resolution,rightMargin,rotation,saveOption,scale,showOnPrint,signaturefieldname,source,stopOnError,text,topMargin,transparent,type,unsignall,useStructure,version,vScale,width, cfpop=accept,action,attachmentpath,debug,delimiter,generateuniquefilenames,maxrows,messagenumber,name,password,port,secure,server,startrow,timeout,uID,username}"/>
<property name="rootdir" value="D:\ColdFusion11\API1"/>
<property name="expiration" value="{ts '2015-02-18 12:10:33'}"/>
<property name="installkit" value="Native Windows"/>
<property name="arch" value="amd64"/>
<property name="additionalinformation" value=""/>
<property name="version" value="6.2"/>
<property name="buildnumber" value=""/>
<property name="name" value="Windows Server 2012"/>
<property name="https_secretkeysize" value=""/>
<property name="remote_host" value="xxx"/>
<property name="server_protocol" value="HTTP/1.1"/>
<property name="cert_server_subject" value=""/>
<property name="remote_addr" value="xxx"/>
<property name="cert_server_issuer" value=""/>
<property name="server_software" value="Microsoft-IIS/8.5"/>
<property name="path_translated" value="xxx"/>
<property name="https_server_subject" value=""/>
<property name="cert_keysize" value=""/>
<property name="cf_template_path" value="xxx"/>
<property name="cert_serialnumber" value=""/>
<property name="cert_subject" value=""/>
<property name="http_referer" value=""/>
<property name="auth_password" value=""/>
<property name="https" value="off"/>
<property name="content_type" value=""/>
<property name="request_method" value="GET"/>
<property name="script_name" value="/_tests/antRunner.cfm"/>
<property name="cert_issuer" value=""/>
<property name="server_name" value="xxx"/>
<property name="path_info" value=""/>
<property name="auth_type" value=""/>
<property name="gateway_interface" value="CGI/1.1"/>
<property name="server_port" value="80"/>
<property name="https_server_issuer" value=""/>
<property name="http_accept_language" value=""/>
<property name="context_path" value=""/>
<property name="server_port_secure" value="0"/>
<property name="cert_cookie" value=""/>
<property name="web_server_api" value=""/>
<property name="https_keysize" value=""/>
<property name="auth_user" value=""/>
<property name="remote_user" value=""/>
<property name="http_host" value="xxx"/>
<property name="content_length" value="0"/>
<property name="query_string" value="directory=_tests.suites&recurse=true&reporter=ANTJunit&labels=&propertiesSummary=true&propertiesFilename=TEST.properties&reportPath=D:\buildagent\work\c9e86a456371163e\API\testroot/results"/>
<property name="http_accept" value="text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2"/>
<property name="cert_secretkeysize" value=""/>
<property name="http_user_agent" value="Java/1.8.0_60"/>
<property name="http_accept_encoding" value=""/>
<property name="http_cookie" value=""/>
<property name="cert_flags" value=""/>
<property name="http_connection" value="keep-alive"/>
</properties>
</testsuite>
<testsuite tests="3" failures="0" errors="0" skipped="0" time="27.953" timestamp="2016-02-19T15:53:28" hostname="xxx" package="_tests.suites.v1.addresses.create" name="_tests.suites.v1.addresses.create">
<properties>
<property name="supportedlocales" value=",Chinese (China),Chinese (Hong Kong),Chinese (Taiwan),Dutch (Belgian),Dutch (Standard),English (Australian),English (Canadian),English (New Zealand),English (UK),English (US),French (Belgian),French (Canadian),French (Standard),French (Swiss),German (Austrian),German (Standard),German (Swiss),Italian (Standard),Italian (Swiss),Japanese,Korean,Norwegian (Bokmal),Norwegian (Nynorsk),Portuguese (Brazilian),Portuguese (Standard),Spanish (Mexican),Spanish (Modern),Spanish (Standard),Swedish,ar,ar_AE,ar_BH,ar_DZ,ar_EG,ar_IQ,ar_JO,ar_KW,ar_LB,ar_LY,ar_MA,ar_OM,ar_QA,ar_SA,ar_SD,ar_SY,ar_TN,ar_YE,be,be_BY,bg,bg_BG,ca,ca_ES,cs,cs_CZ,da,da_DK,de,de_AT,de_CH,de_DE,de_GR,de_LU,el,el_CY,el_GR,en,en_AU,en_CA,en_GB,en_IE,en_IN,en_MT,en_NZ,en_PH,en_SG,en_US,en_ZA,es,es_AR,es_BO,es_CL,es_CO,es_CR,es_CU,es_DO,es_EC,es_ES,es_GT,es_HN,es_MX,es_NI,es_PA,es_PE,es_PR,es_PY,es_SV,es_US,es_UY,es_VE,et,et_EE,fi,fi_FI,fr,fr_BE,fr_CA,fr_CH,fr_FR,fr_LU,ga,ga_IE,hi,hi_IN,hr,hr_HR,hu,hu_HU,in,in_ID,is,is_IS,it,it_CH,it_IT,iw,iw_IL,ja,ja_JP,ja_JP_JP_#u-ca-japanese,ko,ko_KR,lt,lt_LT,lv,lv_LV,mk,mk_MK,ms,ms_MY,mt,mt_MT,nl,nl_BE,nl_NL,no,no_NO,no_NO_NY,pl,pl_PL,pt,pt_BR,pt_PT,ro,ro_RO,ru,ru_RU,sk,sk_SK,sl,sl_SI,sq,sq_AL,sr,sr_BA,sr_BA_#Latn,sr_CS,sr_ME,sr_ME_#Latn,sr_RS,sr_RS_#Latn,sr__#Latn,sv,sv_SE,th,th_TH,th_TH_TH_#u-nu-thai,tr,tr_TR,uk,uk_UA,vi,vi_VN,zh,zh_CN,zh_HK,zh_SG,zh_TW"/>
<property name="appserver" value="Tomcat"/>
<property name="productlevel" value="Developer"/>
<property name="productversion" value="11,0,03,292480"/>
<property name="updatelevel" value="03"/>
<property name="productname" value="ColdFusion Server"/>
<property name="servicetagattributes" value="{cfcollection=action,categories,collection,engine,language,name,path, cfimap=accept,action,attachmentpath,connection,debug,delimiter,folder,generateuniquefilenames,maxrows,messagenumber,name,newFolder,password,port,recurse,secure,server,startrow,stopOnError,timeout,uID,username, cffeed=action,columnMap,enclosureDir,escapeChars,ignoreEnclosureError,name,outputFile,overwrite,overwriteEnclosure,properties,proxyPassword,proxyport,proxyserver,proxyUser,query,source,timeout,userAgent,xmlVar, cfstoredproc=blockfactor,cachedafter,cachedwithin,cacheId,cacheRegion,clientInfo,datasource,dbname,dbtype,debug,fetchClientInfo,maxrows,name,ormOptions,password,procedure,result,returncode,sql,timeout,username, cfldap=action,attributes,clientCert,clientCertPassword,delimiter,dn,filter,maxrows,modifytype,name,password,port,rebind,referral,returnAsBinary,scope,secure,separator,server,sort,sortControl,start,startrow,timeout,userName,useTLS, cfdbinfo=blockfactor,cachedafter,cachedwithin,cacheId,cacheRegion,clientInfo,datasource,dbname,dbtype,debug,fetchClientInfo,maxrows,name,ormOptions,password,pattern,result,sql,table,timeout,type,username, cfindex=action,attributecollection,autocommit,body,category,categorytree,collection,collectionnames,criteria,custom1,custom2,custom3,custom4,dihconfigpath,docboost,extensions,external,fieldboost,indexMode,key,language,prefix,query,recurse,status,title,type,urlpath, cfhttp=authType,charset,clientCert,clientCertPassword,columns,compression,delimiter,domain,file,firstRowAsHeaders,getAsBinary,method,multipart,multipartType,name,password,path,port,proxyPassword,proxyport,proxyserver,proxyUser,redirect,resolveurl,result,textqualifier,throwonerror,timeout,url,useragent,username,workstation, cfquery=blockfactor,cachedafter,cachedwithin,cacheId,cacheRegion,clientInfo,datasource,dbname,dbtype,debug,fetchClientInfo,maxrows,name,ormOptions,password,result,sql,timeout,username, cfftp=action,actionParam,alloSize,asciiextensionlist,bufferSize,connection,debug,directory,existing,failifexists,fingerPrint,item,key,localfile,name,new,passive,passphrase,password,port,proxybypass,proxyserver,remotefile,result,retrycount,secure,server,stoponerror,systemType,timeout,transfermode,username, cfmail=bcc,cc,charset,debug,encrypt,encryptionAlgorithm,failTo,from,group,groupcasesensitive,keyalias,keyPassword,keystore,keystorePassword,mailerid,maxrows,mimeattach,password,port,priority,query,recipientCert,remove,replyTo,server,sign,spoolenable,startrow,subject,timeout,to,type,username,useSSL,useTLS,wrapText, cfSearch=category,categorytree,collection,contextBytes,contextHighlightBegin,contextHighlightEnd,contextPassages,criteria,external,language,maxrows,name,orderby,previousCriteria,searchcolumns,searchtime,startrow,status,suggestions,type, cfpdf=action,addQuads,algo,align,ascending,author,bottomMargin,charset,compressTiffs,copyfrom,ddxFile,destination,directory,encodeAll,encrypt,flatten,foreground,format,height,hiRes,honourSpaces,hScale,image,imageprefix,info,inputFiles,isBase64,jpgDpi,keepBookmark,keyalias,keyPassword,keystore,keystorePassword,leftMargin,maxBreadth,maxLength,maxScale,name,newOwnerPassword,newUserPassword,noAttachments,noBookmarks,noComments,noFonts,noJavascripts,noLinks,noMetadata,noThumbnails,numberFormat,opacity,order,outputFiles,overridePage,overwrite,package,pages,password,permissions,position,resolution,rightMargin,rotation,saveOption,scale,showOnPrint,signaturefieldname,source,stopOnError,text,topMargin,transparent,type,unsignall,useStructure,version,vScale,width, cfpop=accept,action,attachmentpath,debug,delimiter,generateuniquefilenames,maxrows,messagenumber,name,password,port,secure,server,startrow,timeout,uID,username}"/>
<property name="rootdir" value="D:\ColdFusion11\API1"/>
<property name="expiration" value="{ts '2015-02-18 12:10:33'}"/>
<property name="installkit" value="Native Windows"/>
<property name="arch" value="amd64"/>
<property name="additionalinformation" value=""/>
<property name="version" value="6.2"/>
<property name="buildnumber" value=""/>
<property name="name" value="Windows Server 2012"/>
<property name="https_secretkeysize" value=""/>
<property name="remote_host" value="xxx"/>
<property name="server_protocol" value="HTTP/1.1"/>
<property name="cert_server_subject" value=""/>
<property name="remote_addr" value="xxx"/>
<property name="cert_server_issuer" value=""/>
<property name="server_software" value="Microsoft-IIS/8.5"/>
<property name="path_translated" value="xxx"/>
<property name="https_server_subject" value=""/>
<property name="cert_keysize" value=""/>
<property name="cf_template_path" value="xxx"/>
<property name="cert_serialnumber" value=""/>
<property name="cert_subject" value=""/>
<property name="http_referer" value=""/>
<property name="auth_password" value=""/>
<property name="https" value="off"/>
<property name="content_type" value=""/>
<property name="request_method" value="GET"/>
<property name="script_name" value="/_tests/antRunner.cfm"/>
<property name="cert_issuer" value=""/>
<property name="server_name" value="xxx"/>
<property name="path_info" value=""/>
<property name="auth_type" value=""/>
<property name="gateway_interface" value="CGI/1.1"/>
<property name="server_port" value="80"/>
<property name="https_server_issuer" value=""/>
<property name="http_accept_language" value=""/>
<property name="context_path" value=""/>
<property name="server_port_secure" value="0"/>
<property name="cert_cookie" value=""/>
<property name="web_server_api" value=""/>
<property name="https_keysize" value=""/>
<property name="auth_user" value=""/>
<property name="remote_user" value=""/>
<property name="http_host" value="xxx"/>
<property name="content_length" value="0"/>
<property name="query_string" value="directory=_tests.suites&recurse=true&reporter=ANTJunit&labels=&propertiesSummary=true&propertiesFilename=TEST.properties&reportPath=D:\buildagent\work\c9e86a456371163e\API\testroot/results"/>
<property name="http_accept" value="text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2"/>
<property name="cert_secretkeysize" value=""/>
<property name="http_user_agent" value="Java/1.8.0_60"/>
<property name="http_accept_encoding" value=""/>
<property name="http_cookie" value=""/>
<property name="cert_flags" value=""/>
<property name="http_connection" value="keep-alive"/>
</properties>
<testcase name="Create address... ...POST Address (/addresses) with minimum payload" time="26.532" classname="_tests.suites.v1.addresses.create"/>
<testcase name="Create address... ...POST Address (/addresses) with no payload" time="0.141" classname="_tests.suites.v1.addresses.create"/>
<testcase name="Create address... ...POST Address (/addresses) with User but no other payload" time="1.265" classname="_tests.suites.v1.addresses.create"/>
</testsuite>
</testsuites>
0

Alina, any update on this?  Does the TeamCity parser ignore test suites with only errors in them?

Kind Regards,
Andrew Fraser

0

Hi Andrew,

TeamCity does not ignore testsuits with errors only. For example the following .xml file was parsed correctly:

<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
<testsuite errors="1" failures="0" name="TestSuite1" tests="0" timestamp="2016-02-26T00:00:00">
<testcase classname="ModuleIAmTesting" name="testCreatePlugin" time="2">
<error message="Error while executing test testCreatePlugin()!" type="error">
</error>
</testcase>
</testsuite>
</testsuites>

It failed the build and reported one test failure.

It seems that format of your report is incorrect. Please check it.

0

Hi Alina,

Thanks for getting back to me.  It seems like the reporter is leaving the testcase element out, I think this may be because the error occurs in the suite setup.  We're following this up with the coldbox guys.

I still think that if TeamCity imports a testsuite with 0 testcases but a whole lot of errors it should reflect that error somewhere.  Could you verify what TeamCity does with this:

<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
<testsuite errors="1" failures="0" name="TestSuite1" tests="0" timestamp="2016-02-26T00:00:00">
</testsuite>
</testsuites>

Kind Regards,
Andrew Fraser

0

Hi Andrew,

The original Ant JUnit format is supported in TeamCity. We do not report failed testsuites, only tests are reported. The structure should be the following:

<!ELEMENT testsuites (testsuite*)>
<!ELEMENT testsuite (properties, testcase*, failure?, error?,
                  system-out?, system-err?)>
<!ELEMENT properties (property*)>
<!ELEMENT testcase (failure?, error?, system-out?, system-err?)>

According to this dtd if the error is reported correctly (using the embedded /error or /failure tag), we will at least print it to the build log as error message. 

It seems the correct approach is to fix it in ColdBox, not in TeamCity. What is your opinion?

0

That makes sense, I agree that ColdBox needs a fix.  But I also think that if TeamCity consider the xml not formed corrected (no test cases in a suite that has errors) that it should fail the build.  I would imagine it's rare enough that it shouldn't be a priority for you guys.

Thank you for your help, much appreciated.

0

Actually the xml report structure is correct, just no tests are reported. I think that we should not fail build in this case. However you are always welcome to create a feature request in our tracker: https://youtrack.jetbrains.com/issues/TW.
Thank you for your feedback and happy building.

0

Please sign in to leave a comment.