How to use EC2 build agents with local TeamCity Server

I have a local TeamCity server (5.0.1), which is accessible on the Internet on a public domain name. It has local build agents that work just fine. I have now been playing with having Amazon EC2 based build agents. But it never gets past the "Agent has unregistered (will upgrade)" phase. I have then tried running a TeamCity server on the cloud, and here it works just fine.

I have attached two screenshots:

"TeamCity Server running on EC2.png" shows the build agent screen for the EC2 build agent running just fine (had actually build a project), when the TeamCity server is running on EC2 as well.

"TeamCity Server running on local server.png" shows the build agent screen for the EC2 build agent, when the TeamCity is my local installation.

Something I have noticed is the IP address for the build agent when running a local TeamCity server is a local IP address (127.0.0.1).



Attachment(s):
TeamCity Server running on local server.png
TeamCity Server running on EC2.png
15 comments
Comment actions Permalink

Please check server's own address is accessible from the build agent that was started in EC2.
Open server/Administration/Server Configuration and check Server URL field.

TeamCity server requires two side connection with agent. Please check build agent
has it's port opened in the firewall of EC2. By default agent uses 9090 port.
You may change this setting in the buildAgent.properties file.

0
Comment actions Permalink

Hi Eugene,

Server URL is filled out with a public DNS name pointing to a public IP address accessible from everywhere.

The build agent EC2 instance is using a security group with port 9090 open (only TCP).

Regards
Frank

0
Comment actions Permalink

Please attach server logs from <server>/logs folder.

Please check build agent is able to upgrade itself, i.e. it has enough rights to update it's files. Please attach build agent's logs too.

Thanks!

0
Comment actions Permalink

Attached is the server.log and teamcity-agent.log

When looking at the teamcitys server.log is says that it can't connect to agent addresses 127.0.0.1, 10.227.102.166. That makes sense since the build agents internal address was 10.227.102.166, but that is not accessible from the outside. How can I get the agent to use the external address (in this case 79.125.34.13)?



Attachment(s):
teamcity-agent.log.zip
server.log.zip
0
Comment actions Permalink

Another thing I tried is to connect from started build agent instance to my local TeamCity server and that worked fine. So there is possible for the build agent to connect to the server.

I also have used the build agent EC2 image when I tried to do it from a EC2 based TeamCity server. And that worked fine. It could upgrade and all. So the user that runs the agent has enough rights to do the upgrade and do a build.

0
Comment actions Permalink

Do you run server and agent in the same EC2 region, availability zone?
Does build agent machine accepts connections to port 9090?

Build agent logs and server logs shows that server has failed to connect to build agent
on port 9090 to the both 10.* ip and localhost ip.

Could you please check you have specified security group of an image in the cloud profile settings page.
Please try adding a rule like: 0.0.0.0/9090

You may check the port from any web browser by opening the url http://<agent IP>:9090/RPC2

Please uncomment the following lines in <server>/conf/teamcity-server-log4j.xml file:

  <appender name="CLOUDS.LOG" >
    <param name="file" value="${teamcity_logs}teamcity-clouds.log"/>
    <param name="maxBackupIndex" value="10"/>

    <layout >
      <param name="ConversionPattern" value="[%d] %6p [%15.15t] - %30.30c - %m %n"/>
    </layout>
  </appender>


and

  <category name="jetbrains.buildServer.clouds">
    <priority value="DEBUG"/>
    <appender-ref ref="CLOUDS.LOG"/>
    <appender-ref ref="CONSOLE-ERROR"/>
  </category>


Please add:
  <category name="jetbrains.buildServer.xmlRpc" additivity="false">
     <priority value="INFO"/>
  </category>


before last <comment> element and uncomment
    <priority value="DEBUG"/>
in
  <category name="jetbrains.buildServer">
    <!--DELETE THIS LINE FOR ENABLING DEBUG LOGGING--> <!--
    <priority value="DEBUG"/>
    --> <!--DELETE THIS LINE FOR ENABLING DEBUG LOGGING-->
    <appender-ref ref="ROLL"/>
  </category>



Please try starting build agent again and that attach produced server logs.
Server will automactically update the changes in logger configuration.
0
Comment actions Permalink

eugene.petrenko wrote:

Do you run server and agent in the same EC2 region, availability zone?


I want to run the server on my local server hardware. And only use agents on EC2. This is where my problem is.

I have though tried to run the server on EC2 as well and that worked fine. It was running in the same availability zone, though that was coincidence I think, because it is not possible to specify the availability zone in TeamCity.

eugene.petrenko wrote:

Does build agent machine accepts connections to port 9090?


Yes, this is a screenshot of the security group used:

Screen shot 2009-12-29 at 12.04.24.png

eugene.petrenko wrote:

Build agent logs and server logs shows that server has failed to connect to build agent
on port 9090 to the both 10.* ip and localhost ip.


Which makes sense, because I will not have access to local or lan ip addresses from my local server.

eugene.petrenko wrote:



Please try starting build agent again and that attach produced server logs.
Server will automactically update the changes in logger configuration.


I will try that and get back with the logs

0
Comment actions Permalink

TeamCity server uses 'remote host' http attribute to get alternative build agent IP address. Build agent is unable to get EC2 extenal IP now. Please check a request from build agent to your TeamCity server contains right remote host ip address. This IP could change because of for example NAT.
Server logs with debug option will show you build agent address revieved from http attribute.

Thanks!

0
Comment actions Permalink

I'm attaching the log files.

I'm not sure what you mean about the NAT part. My local teamcity server is behind a nat, but as far as I can see in the log files it gives a correct hostname to the agent (build.teachus.dk which resolves a public IP address). The EC2 build agent is not behind a NAT as far as I know. it is a standard Amazon instance.



Attachment(s):
teamcity-server.log.zip
teamcity-clouds.log.zip
teamcity-agent.log.zip
0
Comment actions Permalink

Do you have some proxy/NAT/some other server between internet and you TeamCity server(tomcat)?

0
Comment actions Permalink

I have Apache HTTPD proxy looking like this:

# teamcity.teachus.dk <VirtualHost *>     ServerName                  teamcity.teachus.dk     ServerAlias                 *.teamcity.teachus.dk build.teachus.dk *.build.teachus.dk     ErrorLog                    /var/www/vhosts/teamcity.teachus.dk/log/error.log     CustomLog                   /var/www/vhosts/teamcity.teachus.dk/log/combined.log combined     # Proxy     ProxyPass                   / http://localhost:8111/     ProxyPassReverse            / http://localhost:8111/     <Proxy *>         Allow from all     </Proxy> </VirtualHost>

0
Comment actions Permalink

You need to make tomcat using real remote IP instead of actual remote ip of you httpd proxy.
Please have a look to tomcat valve (aka plugin):
http://code.google.com/p/xebia-france/wiki/RemoteIpValve

Installing the plugin will solve the issue.

0
Comment actions Permalink

Thanx Eugene. That helped.

For the record, I first added xebia-tomcat-extras-1.0.0.jar to <teamcity-server>/lib. I downloaded it from here:

http://code.google.com/p/xebia-france/downloads/list

I then added this to <teamcity-server>/conf/server.xml

<Valve     className="org.apache.catalina.connector.RemoteIpValve"     internalProxies="127\.0\.0\.1"     remoteIPHeader="x-forwarded-for"     remoteIPProxiesHeader="x-forwarded-by"     protocolHeader="x-forwarded-proto" />



just above

<!-- Access log processes all example.              Documentation at: /docs/config/valve.html -->

0
Comment actions Permalink

Hello Franck, hello Eugene,

Thanks for your interest in the RemoteIpValve. This valve is much easier to use since Tomcat 6.0.24 as it has been integrated in the standard distribution (see Tomcat : Valves). Saying it with other words : if TeamCity Server embeds Tomcat 6.0.24, all the customers may use this feature without downloading an extra jar.

I didn't have the time yet to merge in Tomcat documentation all the material we wrote. If you want more details about internal proxies, trusted proxies, https handling with x-forwarded-proto, etc, I suggest you to have a look at :


Cyrille

0
Comment actions Permalink

Thank you for information Cyrille. We did not notice that Tomcat 6.0.24 is already released. Will consider upgrade to new version in TeamCity 5.1.

0

Please sign in to leave a comment.