Slow reconnection for Azure based Build Agents

Folks,

Our TC instance and build agents are all in Azure.

They seem to take an eternity to come back online after inactivity shutdown. Or any shutdown for that matter...

The only suspsicious thing we found in the logs is this:
Unable to set self port. Azure integration will experience problems

Any hints on what that might be?

TIA

Ricardo

8 comments
Comment actions Permalink

Hi Ricardo,


What agent port is configured in "<build Agent>/conf/buildAgent.properties" file? It seems like some virtual machine issue which does not allow the agent process to open this port. Please investigate why it might fail, also check that everything is configured correctly according to the guide.

0
Comment actions Permalink

Hi Alina,

I'm a colleague of Ricardo's, following up on this issue. The agents ports are specified using the default value of 9090 (ownPort=9090). I did a little more digging into this today and though I haven't figured out what's going wrong, I do have some more information.

We have 3 different kinds of build agents in our azure agent cloud: One set of agents is used to build standard .NET projects, a second set of agents is used to build new .NET vNext projects using DNX and the Roslyn compiler, and a third set of agents is being used to build Erlang / Elixir projects. We are experiencing this issue only with the first set of agents -- our DNX and Erlang build agents do not have the same issue. All agents have the "Unable to set self port. Azure integration will experience problems" line appearing in their logs. However, this line does not seem to be a problem for the vNext and Erlang agents, only with the regular .NET agents.

Here is a snippit from <build Agent>/logs/wrapper.log on one of the DNX agents:
INFO   | jvm 1    | 2015/10/31 21:47:49 | Using logs directory C:\BuildAgent\logs
INFO   | jvm 1    | 2015/10/31 21:47:49 | Log4J configuration file C:\BuildAgent\bin\..\conf\teamcity-agent-log4j.xml will be monitored with interval 10 seconds.
INFO   | jvm 1    | 2015/10/31 21:48:46 | [2015-10-31 21:48:46,144]   WARN -      jetbrains.buildServer.VCS - '[C:\BuildAgent\temp] hg version --quiet' command failed.  
INFO   | jvm 1    | 2015/10/31 21:48:48 | [2015-10-31 21:48:48,781]   WARN - ds.azure.AzurePropertiesReader - Unable to set self port. Azure integration will experience problems  
INFO   | jvm 1    | 2015/10/31 21:48:50 | [2015-10-31 21:48:50,757]   WARN - .agent.AmazonPropertiesUpdater - Fetched Amazon EC2 instance metadata does not contain required values. Probably this agent was not started inside Amazon EC2. Is proxy server used?  
INFO   | jvm 1    | 2015/10/31 21:49:13 | [2015-10-31 21:49:13.415] Registering on server http://ourteamcityserver.domain.tld, AgentDetails{Name='AGENT-TC-DNX01', AgentId=null, BuildId=null, AgentOwnAddress='null', AlternativeAddresses=[172.16.0.6, ], Port=9090, Version='37176', PluginsVersion='37176-md5-d0c4458bfd7c7b007618c02323c7e213', AvailableRunners=[Ant, dotnet-tools-dupfinder, dotnet-tools-inspectcode, Duplicator, FxCop, gradle-runner, Inspection, Ipr, jb.nuget.installer, jb.nuget.pack, jb.nuget.publish, jetbrains.dotNetGenericRunner, jetbrains.mspec, jetbrains_powershell, jonnyzzz.grunt, jonnyzzz.gulp, jonnyzzz.node, jonnyzzz.npm, jonnyzzz.nvm, jonnyzzz.phantom, JPS, Maven2, MSBuild, NAnt, NUnit, rake-runner, SBT, simpleRunner, sln2003, VisualStudioTest, VS.Solution], AvailableVcs=[tfs, cvs, jetbrains.git, mercurial, svn, perforce], AuthorizationToken='[OBFUSCATED]', PingCode='[OBFUSCATED]'}
INFO   | jvm 1    | 2015/10/31 21:49:14 | [2015-10-31 21:49:14.476] Registered: id:1637, authorizationToken:[OBFUSCATED]
INFO   | jvm 1    | 2015/10/31 21:49:14 | [2015-10-31 21:49:14.476] If this is the first time this agent registered on the server make sure it is authorized by administrator in the server web UI.



Here is a snippit from <build Agent>/logs/wrapper.log on one of the Erlang agents:
INFO   | jvm 1    | 2015/10/31 21:48:44 | Using logs directory C:\BuildAgent\logs
INFO   | jvm 1    | 2015/10/31 21:48:44 | Log4J configuration file C:\BuildAgent\bin\..\conf\teamcity-agent-log4j.xml will be monitored with interval 10 seconds.
INFO   | jvm 1    | 2015/10/31 21:49:39 | [2015-10-31 21:49:39,797]   WARN -      jetbrains.buildServer.VCS - '[C:\BuildAgent\temp] hg version --quiet' command failed.  
INFO   | jvm 1    | 2015/10/31 21:49:42 | [2015-10-31 21:49:42,445]   WARN - ds.azure.AzurePropertiesReader - Unable to set self port. Azure integration will experience problems
INFO   | jvm 1    | 2015/10/31 21:49:45 | [2015-10-31 21:49:45,809]   WARN - .agent.AmazonPropertiesUpdater - Fetched Amazon EC2 instance metadata does not contain required values. Probably this agent was not started inside Amazon EC2. Is proxy server used?  
INFO   | jvm 1    | 2015/10/31 21:49:59 | [2015-10-31 21:49:59,017]  ERROR - impl.vcs.AgentVcsManagerExImpl - No TFS assemblies were found on the system. Please make sure you have Microsoft Team Explorer installed. Supported versions: 2015 2013 2012 2010 2008 2005  
INFO   | jvm 1    | 2015/10/31 21:49:59 | [2015-10-31 21:49:59.033] Registering on server http://ourteamcityserver.domain.tld, AgentDetails{Name='AGENT-TC-ELX01', AgentId=null, BuildId=null, AgentOwnAddress='null', AlternativeAddresses=[172.16.0.7, ], Port=9090, Version='37176', PluginsVersion='37176-md5-d0c4458bfd7c7b007618c02323c7e213', AvailableRunners=[Ant, dotnet-tools-dupfinder, dotnet-tools-inspectcode, Duplicator, FxCop, gradle-runner, Inspection, Ipr, jb.nuget.installer, jb.nuget.pack, jb.nuget.publish, jetbrains.dotNetGenericRunner, jetbrains.mspec, jetbrains_powershell, jonnyzzz.grunt, jonnyzzz.gulp, jonnyzzz.node, jonnyzzz.npm, jonnyzzz.nvm, jonnyzzz.phantom, JPS, Maven2, MSBuild, NAnt, NUnit, rake-runner, SBT, simpleRunner, sln2003, VisualStudioTest, VS.Solution], AvailableVcs=[cvs, jetbrains.git, mercurial, svn, perforce], AuthorizationToken='[OBFUSCATED]', PingCode='[OBFUSCATED]'}
INFO   | jvm 1    | 2015/10/31 21:49:59 | [2015-10-31 21:49:59.658] Registered: id:1638, authorizationToken:[OBFUSCATED]
INFO   | jvm 1    | 2015/10/31 21:49:59 | [2015-10-31 21:49:59.658] If this is the first time this agent registered on the server make sure it is authorized by administrator in the server web UI.



...And here is a snippit from <build Agent>/logs/wrapper.log on on of the .NET agents:
INFO   | jvm 1    | 2015/10/31 21:30:07 | Using logs directory G:\BuildAgent\logs
INFO   | jvm 1    | 2015/10/31 21:30:07 | Log4J configuration file G:\BuildAgent\bin\..\conf\teamcity-agent-log4j.xml will be monitored with interval 10 seconds.
INFO   | jvm 1    | 2015/10/31 21:31:11 | [2015-10-31 21:31:11,415]   WARN -      jetbrains.buildServer.VCS - '[D:\agent\temp] hg version --quiet' command failed.  
INFO   | jvm 1    | 2015/10/31 21:31:19 | [2015-10-31 21:31:19,195]   WARN - ds.azure.AzurePropertiesReader - Unable to set self port. Azure integration will experience problems  
INFO   | jvm 1    | 2015/10/31 21:31:20 | [2015-10-31 21:31:20,057]   WARN - .agent.AmazonPropertiesUpdater - Fetched Amazon EC2 instance metadata does not contain required values. Probably this agent was not started inside Amazon EC2. Is proxy server used?  
INFO   | jvm 1    | 2015/10/31 22:26:48 | java.lang.OutOfMemoryError: Java heap space
INFO   | jvm 1    | 2015/10/31 22:26:48 | Dumping heap to java_pid2192.hprof ...
INFO   | jvm 1    | 2015/10/31 22:26:52 | Heap dump file created [470608643 bytes in 3.970 secs]
INFO   | jvm 1    | 2015/10/31 22:26:53 | Exception in thread "Change Observer 1" [2015-10-31 22:26:53,743]   WARN - ildServer.util.EventDispatcher - Error in event handler: java.lang.OutOfMemoryError: Java heap space (enable debug to see stacktrace)  
INFO   | jvm 1    | 2015/10/31 22:26:53 | java.lang.OutOfMemoryError: Java heap space
INFO   | jvm 1    | 2015/10/31 22:26:53 | at java.lang.Object.clone(Native Method)
INFO   | jvm 1    | 2015/10/31 22:26:53 | at java.util.TimeZone.clone(Unknown Source)
INFO   | jvm 1    | 2015/10/31 22:26:53 | at sun.util.calendar.ZoneInfo.clone(Unknown Source)
INFO   | jvm 1    | 2015/10/31 22:26:53 | at java.util.TimeZone.getDefault(Unknown Source)
INFO   | jvm 1    | 2015/10/31 22:26:53 | at java.util.Calendar.getInstance(Unknown Source)
INFO   | jvm 1    | 2015/10/31 22:26:53 | at jetbrains.buildServer.util.NamedThreadUtil.time_prefix(NamedThreadUtil.java:69)
INFO   | jvm 1    | 2015/10/31 22:26:53 | at jetbrains.buildServer.util.NamedThreadUtil.getTcThreadPrefix(NamedThreadUtil.java:64)
INFO   | jvm 1    | 2015/10/31 22:26:53 | at jetbrains.buildServer.util.NamedThreadUtil$ThreadNameCookie.setNewName(NamedThreadUtil.java:114)
INFO   | jvm 1    | 2015/10/31 22:26:53 | at jetbrains.buildServer.util.NamedThreadUtil.updateThreadName(NamedThreadUtil.java:56)
INFO   | jvm 1    | 2015/10/31 22:26:53 | at jetbrains.buildServer.util.NamedThreadFactory.patchThreadName(NamedThreadFactory.java:118)
INFO   | jvm 1    | 2015/10/31 22:26:53 | at jetbrains.buildServer.util.executors.TeamCityScheduledThreadPoolExecutor.beforeExecute(TeamCityScheduledThreadPoolExecutor.java:35)
INFO   | jvm 1    | 2015/10/31 22:26:53 | at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
INFO   | jvm 1    | 2015/10/31 22:26:53 | at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
INFO   | jvm 1    | 2015/10/31 22:26:53 | at java.lang.Thread.run(Unknown Source)
INFO   | jvm 1    | 2015/10/31 22:26:54 | [2015-10-31 22:26:53,922]   WARN - LogProxyImpl$LaterExecutorTask - Failed to execute flush AgentLogProxy. Java heap space  
INFO   | jvm 1    | 2015/10/31 22:26:54 | java.lang.OutOfMemoryError: Java heap space
INFO   | jvm 1    | 2015/10/31 22:26:54 | at java.util.Arrays.copyOf(Unknown Source)
INFO   | jvm 1    | 2015/10/31 22:26:54 | at java.lang.AbstractStringBuilder.expandCapacity(Unknown Source)
INFO   | jvm 1    | 2015/10/31 22:26:54 | at java.lang.AbstractStringBuilder.ensureCapacityInternal(Unknown Source)
INFO   | jvm 1    | 2015/10/31 22:26:54 | at java.lang.AbstractStringBuilder.append(Unknown Source)
INFO   | jvm 1    | 2015/10/31 22:26:54 | at java.lang.StringBuilder.append(Unknown Source)
INFO   | jvm 1    | 2015/10/31 22:26:54 | at jetbrains.buildServer.agent.impl.AgentLogProxyImpl$LaterExecutorTask.run(AgentLogProxyImpl.java:450)
INFO   | jvm 1    | 2015/10/31 22:26:54 | at jetbrains.buildServer.util.ExceptionUtil$1.run(ExceptionUtil.java:41)
INFO   | jvm 1    | 2015/10/31 22:26:54 | at jetbrains.buildServer.agent.impl.AgentLogProxyImpl$1.run(AgentLogProxyImpl.java:55)
INFO   | jvm 1    | 2015/10/31 22:26:54 | at java.lang.Thread.run(Unknown Source)
INFO   | jvm 1    | 2015/10/31 22:27:00 | [2015-10-31 22:26:59,949]  ERROR - impl.vcs.AgentVcsManagerExImpl - No TFS assemblies were found on the system. Please make sure you have Microsoft Team Explorer installed. Supported versions: 2015 2013 2012 2010 2008 2005  
INFO   | jvm 1    | 2015/10/31 22:27:00 | [2015-10-31 22:27:00.078] Registering on server http://ourteamcityserver.domain.tld, AgentDetails{Name='AGENT-TC-02', AgentId=null, BuildId=null, AgentOwnAddress='null', AlternativeAddresses=[172.16.0.5, ], Port=9090, Version='37176', PluginsVersion='37176-md5-d0c4458bfd7c7b007618c02323c7e213', AvailableRunners=[Ant, dotnet-tools-dupfinder, dotnet-tools-inspectcode, Duplicator, FxCop, gradle-runner, Inspection, Ipr, jb.nuget.installer, jb.nuget.pack, jb.nuget.publish, jetbrains.dotNetGenericRunner, jetbrains.mspec, jetbrains_powershell, jonnyzzz.grunt, jonnyzzz.gulp, jonnyzzz.node, jonnyzzz.npm, jonnyzzz.nvm, jonnyzzz.phantom, JPS, Maven2, MSBuild, NAnt, NUnit, rake-runner, SBT, simpleRunner, sln2003, VisualStudioTest, VS.Solution], AvailableVcs=[cvs, jetbrains.git, mercurial, svn, perforce], AuthorizationToken='[OBFUSCATED]', PingCode='[OBFUSCATED]'}
INFO   | jvm 1    | 2015/10/31 22:27:01 | [2015-10-31 22:27:00.986] Registered: id:1639, authorizationToken:[OBFUSCATED]
INFO   | jvm 1    | 2015/10/31 22:27:01 | [2015-10-31 22:27:00.986] If this is the first time this agent registered on the server make sure it is authorized by administrator in the server web UI.

Note how all logs contain the "Unable to set self port. Azure integration will experience problems" lines. However, the first 2 agents establish a connection to the server within 30 seconds of experiencing this warning. The 3rd agent, on the other hand, has a period of approximately 60 minutes (varies from 45 to 90 or more minutes) where it is idle after experiencing this warning. During that log idle time, an inspection of the windows task manager shows a "Java(TM) Platform SE binary" background process with between 0.5% and 2.0% CPU usage, and that process' memory usage slowly creeps up through 400, 500, then 600 MB (like a memory leak). Eventually a memory limit is breached (somwhere around 612 MB), the CPU spikes up to 90%+ for several minutes, then finally, the logs pick back up and show a java.lang.OutOfMemory error. Shortly afterward, the agent finally registers itself with the server and connects.

I tried changing the <build Agent>/launcher/conf/wrapper.conf file to log at DEBUG rather than INFO level, but it did not appear to provide any valuable additional information. The logs were simply peppered with a lot of lines indicating that pings to the JVM all seemed to be working ok. I can tell you that the exception stack trace is not always the same. The version I posted above shows a stack including calls to Calendar & TimeZone methods, but other times it has something like the following:

INFO   | jvm 1    | 2015/10/16 09:57:32 | Using logs directory G:\BuildAgent\logs
INFO   | jvm 1    | 2015/10/16 09:57:32 | Log4J configuration file G:\BuildAgent\bin\..\conf\teamcity-agent-log4j.xml will be monitored with interval 10 seconds.
INFO   | jvm 1    | 2015/10/16 09:58:28 | [2015-10-16 09:58:28,573]   WARN -      jetbrains.buildServer.VCS - '[D:\agent\temp] hg version --quiet' command failed.  
INFO   | jvm 1    | 2015/10/16 09:58:28 | [2015-10-16 09:58:28,830]   WARN - ds.azure.AzurePropertiesReader - Unable to set self port. Azure integration will experience problems  
INFO   | jvm 1    | 2015/10/16 09:58:28 | [2015-10-16 09:58:28,936]   WARN - .agent.AmazonPropertiesUpdater - Fetched Amazon EC2 instance metadata does not contain required values. Probably this agent was not started inside Amazon EC2. Is proxy server used?  
INFO   | jvm 1    | 2015/10/16 10:40:50 | java.lang.OutOfMemoryError: Java heap space
INFO   | jvm 1    | 2015/10/16 10:40:50 | Dumping heap to java_pid1636.hprof ...
INFO   | jvm 1    | 2015/10/16 10:40:55 | Heap dump file created [470534495 bytes in 5.151 secs]
INFO   | jvm 1    | 2015/10/16 10:40:56 | Exception in thread "Agent Log Proxy" [2015-10-16 10:40:56,231]   WARN - ildServer.util.EventDispatcher - Error in event handler: java.lang.OutOfMemoryError: Java heap space (enable debug to see stacktrace)  
INFO   | jvm 1    | 2015/10/16 10:40:56 | java.lang.OutOfMemoryError: Java heap space
INFO   | jvm 1    | 2015/10/16 10:40:56 | at jetbrains.buildServer.agent.impl.AgentLogProxyImpl$1.run(AgentLogProxyImpl.java:55)
INFO   | jvm 1    | 2015/10/16 10:40:56 | at java.lang.Thread.run(Unknown Source)
INFO   | jvm 1    | 2015/10/16 10:40:57 | [2015-10-16 10:40:57,737]  ERROR - impl.vcs.AgentVcsManagerExImpl - No TFS assemblies were found on the system. Please make sure you have Microsoft Team Explorer installed. Supported versions: 2015 2013 2012 2010 2008 2005  
INFO   | jvm 1    | 2015/10/16 10:40:57 | [2015-10-16 10:40:57.760] Registering on server http://...



...and there are some times where it provides no stack trace, simly an error about TFS (like the one above in the Erlang agent log, except with a much longer delay):

INFO   | jvm 1    | 2015/10/30 15:29:39 | Using logs directory G:\BuildAgent\logs
INFO   | jvm 1    | 2015/10/30 15:29:39 | Log4J configuration file G:\BuildAgent\bin\..\conf\teamcity-agent-log4j.xml will be monitored with interval 10 seconds.
INFO   | jvm 1    | 2015/10/30 15:30:24 | [2015-10-30 15:30:24,623]   WARN -      jetbrains.buildServer.VCS - '[D:\agent\temp] hg version --quiet' command failed.  
INFO   | jvm 1    | 2015/10/30 15:30:34 | [2015-10-30 15:30:34,338]   WARN - .agent.AmazonPropertiesUpdater - Fetched Amazon EC2 instance metadata does not contain required values. Probably this agent was not started inside Amazon EC2. Is proxy server used?  
INFO   | jvm 1    | 2015/10/30 15:30:36 | [2015-10-30 15:30:36,027]   WARN - ds.azure.AzurePropertiesReader - Unable to set self port. Azure integration will experience problems  
INFO   | jvm 1    | 2015/10/30 16:22:59 | java.lang.OutOfMemoryError: Java heap space
INFO   | jvm 1    | 2015/10/30 16:22:59 | Dumping heap to java_pid2276.hprof ...
INFO   | jvm 1    | 2015/10/30 16:23:03 | Heap dump file created [468230530 bytes in 4.159 secs]
INFO   | jvm 1    | 2015/10/30 16:23:05 | [2015-10-30 16:23:04,982]   WARN - ildServer.util.EventDispatcher - Error in event handler: java.lang.OutOfMemoryError: Java heap space (enable debug to see stacktrace)  
INFO   | jvm 1    | 2015/10/30 16:23:12 | [2015-10-30 16:23:12,545]  ERROR - impl.vcs.AgentVcsManagerExImpl - No TFS assemblies were found on the system. Please make sure you have Microsoft Team Explorer installed. Supported versions: 2015 2013 2012 2010 2008 2005  
INFO   | jvm 1    | 2015/10/30 16:23:12 | [2015-10-30 16:23:12.699] Registering on server http://...

The parts that are consistent are that it takes up to an hour or longer for the JVM process to run out of memory, but after it does, the agent connects to the server. We need to find out what is going on that causes this memory leak and effectively paralyzes these agents while they are trying to register and connect. I suspect that if I were to lower the JVM memory limit, the OutOfMemory error would come more quickly and this set of agents would connect somewhat faster. Still, that is not a reasonably acceptable solution, so we're looking for Jetbrains's help finding an acceptable, permanent solution to this problem. These agents are wasting our Azure dollars waiting to connect, and this should not happen. Please help us get to the bottom of this.

Thanks In Advance,

Dan Ludwig

0
Comment actions Permalink

Hi Dan,

Thank you for investigation and provided details.
Could you please check that OS user TeamCity agent is running under has an access to C:\WindowsAzure\Config?
Could you please attach heap dump java_pid2192.hprof generated when agent crashed?

0
Comment actions Permalink

Hi Alina,

Yes, the user that the agent service runs as has access to C:\WindowsAzure\Config.

The file you requested, even when compressed, exceeds the allowable upload size for attachments on this message board. Since the last post, we changed the launcher/conf/wrapper.conf on one of the symptomatic agents using wrapper.app.parameter.3=-Xmx32m. As predicted, this has resulted in much faster connect times for that agent in particular (we have it down to around 5 minutes or less). It has also resulted in much smaller hprof files, and seems to have made the OutOfMemoryErrors a little more consistent. The last 3 do not have a stack trace, but rather a message indicating "GC overhead limit exceeded".

Here is a log excerpt from that agent today:

INFO   | jvm 1    | 2015/11/05 09:12:59 | Using logs directory G:\BuildAgent\logs

INFO   | jvm 1    | 2015/11/05 09:12:59 | Log4J configuration file G:\BuildAgent\bin\..\conf\teamcity-agent-log4j.xml will be monitored with interval 10 seconds.

INFO   | jvm 1    | 2015/11/05 09:15:00 | [2015-11-05 09:15:00,661]   WARN -      jetbrains.buildServer.VCS - '[D:\agent\temp] hg version --quiet' command failed.  

INFO   | jvm 1    | 2015/11/05 09:15:13 | [2015-11-05 09:15:13,969]   WARN -    jetbrains.buildServer.AGENT - Unable to find checkoutDir for state file 5a87ffcf43c11d36.xml: 'D:\agent\work\5a87ffcf43c11d36'(isDirectory: false). Checkout dir will be deleted  

INFO   | jvm 1    | 2015/11/05 09:15:14 | [2015-11-05 09:15:13,971]   WARN -    jetbrains.buildServer.AGENT - Unable to find checkoutDir for state file fae7c9903e55e18a.xml: 'D:\agent\work\fae7c9903e55e18a'(isDirectory: false). Checkout dir will be deleted  

INFO   | jvm 1    | 2015/11/05 09:15:14 | [2015-11-05 09:15:14,646]   WARN - ds.azure.AzurePropertiesReader - Unable to set self port. Azure integration will experience problems  

INFO   | jvm 1    | 2015/11/05 09:15:16 | [2015-11-05 09:15:16,007]   WARN - .agent.AmazonPropertiesUpdater - Fetched Amazon EC2 instance metadata does not contain required values. Probably this agent was not started inside Amazon EC2. Is proxy server used?  

INFO   | jvm 1    | 2015/11/05 09:19:11 | java.lang.OutOfMemoryError: GC overhead limit exceeded

INFO   | jvm 1    | 2015/11/05 09:19:11 | Dumping heap to java_pid3736.hprof ...

INFO   | jvm 1    | 2015/11/05 09:19:11 | Heap dump file created [41738347 bytes in 0.296 secs]

INFO   | jvm 1    | 2015/11/05 09:19:11 | [2015-11-05 09:19:11,334]   WARN - ildServer.util.EventDispatcher - Error in event handler: java.lang.OutOfMemoryError: GC overhead limit exceeded (enable debug to see stacktrace)  

INFO   | jvm 1    | 2015/11/05 09:19:27 | [2015-11-05 09:19:27,347]  ERROR - impl.vcs.AgentVcsManagerExImpl - No TFS assemblies were found on the system. Please make sure you have Microsoft Team Explorer installed. Supported versions: 2015 2013 2012 2010 2008 2005  

INFO   | jvm 1    | 2015/11/05 09:19:27 | [2015-11-05 09:19:27.524] Registering on server...

...and the file java_pid3736.hprof is attached. If you still need java_pid2192.hprof, you'll need to provide us with an alternate delivery channel to get you files ~40MB. Hopefully this hprof will tell you everything you need to know to help us get to the bottom of this.

Best,

Dan Ludwig



Attachment(s):
java_pid3736.hprof.zip
0
Comment actions Permalink

Please upload the file to our FTP. Thank you in advance!

0
Comment actions Permalink

Alina,

I have uploaded the file java_pid2192.hprof.zip to your FTP since (in /.uploads).

Best,

Dan

0
Comment actions Permalink

Alina,

Any updates on this? It's been a week since we sent the hprof files.

Thanks,

Dan

0
Comment actions Permalink

Hi Dan,

"Azure integration will experience problems" actually means that buildAgent cannot determine whether you launched this VM using TC cloud plugin for Window Azure. It actually looks like you doesn't. Please confirm

Concerning OOME - heap dump indicates to very large number of files (over 1kk) in agent's artifacts cache under (BuildAgent/system/.artifacts_cache). It will help to publish artifacts with large number of files as a single zip file. Alternatively (not recommended) you can disable artifacts caching on this agent (and similar) by setting agent property teamcity.agent.filecache.disabled=true

0

Please sign in to leave a comment.