Python script execution extremely slow

Hello,

I am experiencing a performance issue when running the python script that builds all the source for a project.  I'm new to using TeamCity and did some searches of the Forum but have not found a solution for my issue.

We have a project setup that has multiple hardware targets so we use a python script that performs all the compiles on the source files with the proper parameters for each of the targets.  When running the script on the build server from a command prompt the script executes in about 2 minutes.

What I am attempting to configure is to call the script from TeamCity to automate the build.  I have created a new project in TeamCity and configured it in the same manner as other project.  The first step in the project (after retrieving from source control) is to call the python script.  I initially did this via the command line (runner type).  I have since downloaded and configured the script to be called from the Python Runner plugin, both resulting in extremely slow performance.  Based on the timestamps in the Build Log it takes about 30 seconds per source file for the compile.  The entire script execution takes 2 hours 30 minutes to complete via TeamCity.

    [08:21:39][Step 1/4] [0m[ 37%]  [32mBuilding C object Source/Application/CMakeFiles/Application.dir/App/DebugConsole.c.o

    [08:22:13][Step 1/4] [0m[ 37%]  [32mBuilding C object Source/Application/CMakeFiles/Application.dir/App/Device.c.o

When running the python script interactively from a command prompt it processes about 3 source files per second.


While the script was executing I began to investigate the overall system health.  I noticed that the CPU utilization was almost non-existent and the memory usage was also within usable bounds.  When I started to look into the network usage I noticed high levels.  Using the Windows Resource Monitor, the Network graph was at the top of the scale (100 kbps) and the TCP Connections would hover around 50 connections (this is with only the TeamCity build running and me logged on locally).  I did find a discussion in the forum about Ant scripts running slow due to Anti-virus software, so I uninstalled the aint-virus software and tried the build again.  This produced the same results, over 2.5 hours to execute the build.  Out of curiosity, while the script was executing I disabled the ethernet connection.  After that the script began to compile multiple files per second, just as it does interactively.  So just to prove what I was seeing was a result of the network connection, I re-enabled the ethernet connection and the script again slowed down to about 30 seconds per source file.  When I run the Project with the ethernet connection disabled (after the source retrieval) it completes in about 2 minutes 30 seconds.

So my question is...  How do I configure TeamCity to not go thru the external network connection when executing either command line runner or python runner steps?  I would have expected these executions to be performed from the Build Agent Service directly on the system or at the most via the loopback if a network connection must be used.

Thanks!


1 comment
Comment actions Permalink

Hello,

After considerable searching along with trial & error, I have found a solution.

In the Window -> Services, display the properties for the 'TeamCity Build Agent', under the 'Log On' tab.  The default setting is to log on as 'Local System account'.  Change this property:  select the 'This account:" radio button and using the 'Browse' button find and select the 'NETWORK SERVICE" account.  After clicking on OK, restart the TeamCity Build Agent service and the Python script execution will perfom the same as running from a command prompt on the system.

Hope this helps others that experience performance issues running python scripts.

This web link also has some good information on investigating performance issues in TeamCity:
http://confluence.jetbrains.com/display/TCD8/Common+Problems#CommonProblems-BuildfailsorbehavesdifferentlyinTeamCitybutnotlocally

0

Please sign in to leave a comment.