Performance issues running inspection builds.

We are currently running TeamCity 1.2 for our changeset, and nightly builds. We had previously used cruisecontrol but it had significant stability issues. We really like the integration with Intellij that TeamCity provides.

For our current project we inherited a fairly large, older java system that did not have very strict coding standards or quality checks. We have a pretty severe inspection configuration we run on our newer code. We want to maintain that high level of inspection checking on new code and changes made to this project. Unfortunately we are running into some very serious performance issues doing this. Our current inspection build takes just under four hours! This does not include the hour it takes to render the inspection report.

I was hoping there were some configuration changes we could make to reduce the length of this build. Here are some questions:

1. It currently appears that there is no way to run inspections on new code only. Is this the case?

2. I would like to exclude our test directory from being inspected. This directory includes unit tests, fixtures and supporting code, that is not part of the deployed system. In Intellij I have the option of unchecking "Include test sources" from an inspection. I can not seem to find an equivalent option in TeamCity. Is this possible?

3. Are there known inspections that are very time intensive? Perhaps there is a subset of inspections I could disable to give us a significant performance boost.


I appreciate the assistance in advance. We find a lot of value in TeamCity but the performance issues here are getting overwhelming.

Thank you,

Adam Butler
John Deere Agri Sevices

15 comments
Comment actions Permalink

Adam Butler wrote:

I was hoping there were some configuration changes we could make to
reduce the length of this build. Here are some questions:

1. It currently appears that there is no way to run inspections on
new code only. Is this the case?

This is not quite possible due to nature of "global" inspections - in
contrast to local inspections that IDEA runs almost in real time in
source editor, "global" (see Analyze/Inspect Code) require
dependency-analysis . All inter-class inspections are "global". So
change in the one file may produce/remove any number of inspection
messages addressing any other file.

2. I would like to exclude our test directory from being inspected.
This directory includes unit tests, fixtures and supporting code,
that is not part of the deployed system. In Intellij I have the
option of unchecking "Include test sources" from an inspection. I
can not seem to find an equivalent option in TeamCity. Is this
possible?

Define several scopes in IDEA project settings and attach them to
different inspection profiles. We use this with IDEA and TeamCity projects.

3. Are there known inspections that are very time intensive? Perhaps
there is a subset of inspections I could disable to give us a
significant performance boost.

All "global" inspections. But they are actually most useful. You may
distinguish them by "(available for Analyze/Inspect Code)" comment in
inspection profile settings.

Global inspection is very CPU and RAM hungry process. You may both
fine-tune your profiles/scopes and ensure that inspections configuration
is to be run on best hardware available. Here at JetBrains we run
Inspections on IDEA project (which is really huge:) overnight.

--
Alexey Gopachenko
JetBrains Inc.
http://www.intellij.com
"Develop with pleasure!"

0
Comment actions Permalink

Thank you for the response. Sorry for the delayed update, but I wished to experiment with your suggestions first.

This is not quite possible due to nature of "global"
inspections - in
contrast to local inspections that IDEA runs almost
in real time in
source editor, "global" (see Analyze/Inspect Code)
require
dependency-analysis . All inter-class inspections are
"global". So
change in the one file may produce/remove any number
of inspection
messages addressing any other file.


1. I understand the limitations here. One change that might help. The inspections take quite a bit of time to run (four hours), but rendering the inspection report can take well over an hour and severely tax the TeamCity machine. Could the inspection report be generated as the inspections are generated? OR, if this isn't feasible, could we at least get the top level inspection metrics, without needing the inspection report generated? We would know that we wouldn't need to view the report if there were no new inspection errors or warnings.

Define several scopes in IDEA project settings and
attach them to
different inspection profiles. We use this with IDEA
and TeamCity projects.


2. I hate to ask such a newbish question, but how would I do this? I spent a good deal of time trying to figure it out. I was able to create a custom inspection profile in Intellij, make it the project default, and then define a custom scope. I can export the inspection profile, but the scope does not appear to be in the export. I can leave the inspections in the project and define the custom scope, but TeamCity doesn't seem to respect this. Any insight, or pointers to documentation (I have read through the intellij docs) would be greatly appreciated.

All "global" inspections. But they are actually most
useful. You may
distinguish them by "(available for Analyze/Inspect
Code)" comment in
inspection profile settings.


3. Thank you for the qualifier, that helped a great deal.

Thank you again for the response and assistance.

0
Comment actions Permalink

I'm also wondering how to specify the scope for server side inspections

0
Comment actions Permalink

Hello,

Thank you for the response. Sorry for the delayed update, but I
wished to experiment with your suggestions first.

>> This is not quite possible due to nature of "global"
>> inspections - in
>> contrast to local inspections that IDEA runs almost
>> in real time in
>> source editor, "global" (see Analyze/Inspect Code)
>> require
>> dependency-analysis . All inter-class inspections are
>> "global". So
>> change in the one file may produce/remove any number
>> of inspection
>> messages addressing any other file.

1. I understand the limitations here. One change that might help.
The inspections take quite a bit of time to run (four hours), but
rendering the inspection report can take well over an hour and
severely tax the TeamCity machine. Could the inspection report be
generated as the inspections are generated? OR, if this isn't
feasible, could we at least get the top level inspection metrics,
without needing the inspection report generated? We would know that
we wouldn't need to view the report if there were no new inspection
errors or warnings.


What do you mean under rendering them? The page is loading for an hour?

>> Define several scopes in IDEA project settings and
>> attach them to
>> different inspection profiles. We use this with IDEA
>> and TeamCity projects.

2. I hate to ask such a newbish question, but how would I do this? I
spent a good deal of time trying to figure it out. I was able to
create a custom inspection profile in Intellij, make it the project
default, and then define a custom scope. I can export the inspection
profile, but the scope does not appear to be in the export. I can
leave the inspections in the project and define the custom scope, but
TeamCity doesn't seem to respect this. Any insight, or pointers to
documentation (I have read through the intellij docs) would be greatly
appreciated.


When you set up project default profile this profile will be used by TeamCity
to inspect your project (without any additional settings - just left the
"inspection profile" field empty). If you want to have different parts of
your project to be inspected by different profiles then you have to define
scope-profile relation. This relation will be loaded from your .ipr file
automatically. Again you do not need to specify anything in TeamCity

>> All "global" inspections. But they are actually most
>> useful. You may
>> distinguish them by "(available for Analyze/Inspect
>> Code)" comment in
>> inspection profile settings.

3. Thank you for the qualifier, that helped a great deal.

Thank you again for the response and assistance.


Thank you


0
Comment actions Permalink

What do you mean under rendering them? The page is loading for an hour?


Yes when we get to work in the morning and click on the inspection build that ran during the previous evening I would see this:

Code inspection
Loading...

It can take at least an hour before the "Loading..." becomes a link to the inspection report. And during that time, TeamCity is essentially unusable. I am running 1.2 Release, windows installation.

When you set up project default profile this profile will be used by TeamCity
to inspect your project (without any additional settings - just left the
"inspection profile" field empty). If you want to have different parts of
your project to be inspected by different profiles then you have to define
scope-profile relation. This relation will be loaded from your .ipr file
automatically. Again you do not need to specify anything in TeamCity


Hmm...then this does not appear to be working. Let me run through my configuration:

I have created an inspection profile, called BMS. In Intellij I go to File -> Settings -> Errors. In the dialog I see IDE Profiles, and Project Profiles. My inspection profile, BMS, is located under Project Profiles. There are no other profiles under that node. The icon for the BMS profile has a "D" on it. If I select the Project Profile Node, I see that BMS is selected in the drop down as the Default Project Profile. In the Project Inspection Profile Assignments, I have one row. It has a custom scope I have defined, and the BMS inspection profile.

Is this all correct? If I open my projects ipr file I see this:

snip... and: ]]>
<scope name="TeamCityInspector" pattern="src:..&&!src:com.agris.apps.bode.messaging..*" />
</component>

If I look at my configuration in TeamCity I specify the "Path to .ipr file:", but do not specify the "Inspections profile path:".

When the inspections are run, my src tree (including the messaging package, which I explicity excluded) and my test tree are fully analyzed. So it appears TeamCity is ignoring the scope defined in my project file.

I appreciate you taking the time to help us out with this issue.

Thank you,

Adam Butler

0
Comment actions Permalink

Hello Adam,

Sorry I was not quite clear. All project will be processed with project default
profile (in your case its BMS).

If you do not want to inspect smth just set up empty profile:

and ]]>

Sorry for inconvenience.

We understand that it is not simple to configure If you have any complains
please feel free voice them :)

Thank you

>> What do you mean under rendering them? The page is loading for an
>> hour?
>>

Yes when we get to work in the morning and click on the inspection
build that ran during the previous evening I would see this:

Code inspection
Loading...
It can take at least an hour before the "Loading..." becomes a link to
the inspection report. And during that time, TeamCity is essentially
unusable. I am running 1.2 Release, windows installation.

>> When you set up project default profile this profile will be used by
>> TeamCity to inspect your project (without any additional settings -
>> just left the "inspection profile" field empty). If you want to have
>> different parts of your project to be inspected by different profiles
>> then you have to define scope-profile relation. This relation will be
>> loaded from your .ipr file automatically. Again you do not need to
>> specify anything in TeamCity
>>

Hmm...then this does not appear to be working. Let me run through my
configuration:

I have created an inspection profile, called BMS. In Intellij I go to
File -> Settings -> Errors. In the dialog I see IDE Profiles, and
Project Profiles. My inspection profile, BMS, is located under
Project Profiles. There are no other profiles under that node. The
icon for the BMS profile has a "D" on it. If I select the Project
Profile Node, I see that BMS is selected in the drop down as the
Default Project Profile. In the Project Inspection Profile
Assignments, I have one row. It has a custom scope I have defined,
and the BMS inspection profile.

Is this all correct? If I open my projects ipr file I see this:

<component name="InspectionProjectProfileManager">
<option name="PROJECT_PROFILE" value="BMS" />
<option name="USE_PROJECT_LEVEL_SETTINGS" value="true" />
<scopes>
<scope profile="BMS" name="TeamCityInspector" />
</scopes>
snip...
</component>
and:

<component name="DependencyValidationManager">
<scope name="TeamCityInspector"
pattern="src:..&&!src:com.agris.apps.bode.messaging..*" />
</component>
If I look at my configuration in TeamCity I specify the "Path to .ipr
file:", but do not specify the "Inspections profile path:".

When the inspections are run, my src tree (including the messaging
package, which I explicity excluded) and my test tree are fully
analyzed. So it appears TeamCity is ignoring the scope defined in my
project file.

I appreciate you taking the time to help us out with this issue.

Thank you,

Adam Butler



0
Comment actions Permalink

Adam Butler wrote:
>> What do you mean under rendering them? The page is loading for an hour?


Yes when we get to work in the morning and click on the inspection build that ran during the previous evening I would see this:

Code inspection
Loading...

It can take at least an hour before the "Loading..." becomes a link to the inspection report. And during that time, TeamCity is essentially unusable. I am running 1.2 Release, windows installation.

Inspection storage and reporting was completely rewritten in TC 2.0
which is available in EAP and will be released in the March'07. Please
note that previous inspection information will be lost on upgrade.


--
Alexey Gopachenko
JetBrains Inc.
http://www.intellij.com
"Develop with pleasure!"

0
Comment actions Permalink

I have encountered the completely same issue - opening an inspection result took several minutes and all other clicks into TC web UI were meantime frozen (also, TC process took all processor time).

Interestingly, this concerns only reports for which exists a previous report (in other words, the first report is fine, all others suffer from this bug).

I tried to install eap version (build 3654) but it is very unstable I could not setup even CVS connection.

Is there any way to setup the clean-up policy that only one report is always kept? I tried to setup this policy: 'keep history for 1 days but not less than 1 builds' and ran it but still TC kept both reports.

Any idea?

0
Comment actions Permalink

I have, so far, been completely unsuccessful getting TeamCity to respect my custom scopes. I have followed all the instructions provided here in this forum. I am likely doing something wrong, but it seems very convoluted. It would be nice if the TeamCity inspector would simply use the default project inspection profile, and let me specify the name of the custom scope. If I get a chance, I will put together a sample intellij project that demonstrates my issue.

In the meantime, I am going to give the EAP version a shot, based on Alexey's comment above.

Thanks again for the assistance, I will continue to update with my progress.

0
Comment actions Permalink

PePan wrote:

I tried to install eap version (build 3654) but it is very unstable I could not setup even CVS connection.


Could you please provide details on this?


Is there any way to setup the clean-up policy that only one report is always kept? I tried to setup this policy: 'keep history for 1 days but not less than 1 builds' and ran it but still TC kept both reports.


TeamCity perform cleanup once a day. So this will work only if you have one Inspections build per
day.

Regards,
KIR


--
Kirill Maximov
JetBrains, Inc.
http://www.jetbrains.com
"Develop with pleasure!"

0
Comment actions Permalink

I tried to install eap version (build 3654) but it

is very unstable I could not setup even CVS
connection.

Could you please provide details on this?


When setuping the CVS connection I pressed the button 'Test connection'. A window saying: 'No object' (or something like that) appeared and the UI got frozen (the Save button was not enabled).

Is there any way to setup the clean-up policy that

only one report is always kept? I tried to setup this
policy: 'keep history for 1 days but not less than 1
builds' and ran it but still TC kept both reports.

TeamCity perform cleanup once a day. So this will
work only if you have one Inspections build per
ay.


Manual way to delete reports would be very helpful.


Regards,
KIR


--
Kirill Maximov
JetBrains, Inc.
http://www.jetbrains.com
"Develop with pleasure!"PePan wrote:

0
Comment actions Permalink

PePan wrote:

>>> Is there any way to setup the clean-up policy that
>> only one report is always kept? I tried to setup this
>> policy: 'keep history for 1 days but not less than 1
>> builds' and ran it but still TC kept both reports.
>>
>> TeamCity perform cleanup once a day. So this will
>> work only if you have one Inspections build per
>> ay.


Manual way to delete reports would be very helpful.


TC 2.0 inspection build data storage and browsing was completely
rewritten and does not have any performance issues. You can try it in
EAP. (If you decide to try it please backup your DB and .buildserver in
case you decide to downgrade to 1.x)

--
Alexey Gopachenko
JetBrains Inc.
http://www.intellij.com
"Develop with pleasure!"

0
Comment actions Permalink

Hello,

>> Could you please provide details on this?
>

When setuping the CVS connection I pressed the button 'Test connection'. A
window saying: 'No object' (or something like that) appeared and the UI
got frozen (the Save button was not enabled).


Try to refresh browser caches by clicking on Refresh button in browser
toolbar while pressing Shift button.

--
Pavel Sher



0
Comment actions Permalink

I tried to install eap version (build 3654) but it

is very unstable I could not setup even CVS
connection.

Could you please provide details on this?

Is there any way to setup the clean-up policy that

only one report is always kept? I tried to setup this
policy: 'keep history for 1 days but not less than 1
builds' and ran it but still TC kept both reports.

TeamCity perform cleanup once a day. So this will
work only if you have one Inspections build per
ay.


I did run the inspections twice - yesterday and this morning. There are two inspection results now. I scheduled the Clean-up policy this way: Inspection keep history for 1 days.
I pressed the button 'Start clean-up now', however, no inspection result was deleted - both inspection results are still shown.
Kirill, is there any way to delete inspections from DB? I know you use hsqldb, can I use some DB tool to delete something from these DB files.
If there is no such way, I'm afraid, Team City is useless. It can't be used for regular inspections.


Regards,
KIR


--
Kirill Maximov
JetBrains, Inc.
http://www.jetbrains.com
"Develop with pleasure!"PePan wrote:

0
Comment actions Permalink

PePan wrote:
>> PePan wrote:
>>> I tried to install eap version (build 3654) but it
>> is very unstable I could not setup even CVS
>> connection.
>>
>> Could you please provide details on this?
>>
>>> Is there any way to setup the clean-up policy that
>> only one report is always kept? I tried to setup this
>> policy: 'keep history for 1 days but not less than 1
>> builds' and ran it but still TC kept both reports.
>>
>> TeamCity perform cleanup once a day. So this will
>> work only if you have one Inspections build per
>> ay.


I did run the inspections twice - yesterday and this morning. There are two inspection results now. I scheduled the Clean-up policy this way: Inspection keep history for 1 days.
I pressed the button 'Start clean-up now', however, no inspection result was deleted - both inspection results are still shown.
Kirill, is there any way to delete inspections from DB? I know you use hsqldb, can I use some DB tool to delete something from these DB files.
If there is no such way, I'm afraid, Team City is useless. It can't be used for regular inspections.


Are there chances that there are less than 24h since the first inspection build?

>> Regards,
>> KIR
>>
>>
>> --
>> Kirill Maximov
>> JetBrains, Inc.
>> http://www.jetbrains.com
>> "Develop with pleasure!"


--
Kirill Maximov
JetBrains, Inc.
http://www.jetbrains.com
"Develop with pleasure!"

0

Please sign in to leave a comment.