6.5 EAP - Build 17132 - agents stuck in endless upgrade loop

My prototype server was running 6.0.1.  I stopped it and unpacked the software (made a backup of the data) and started it up. It started fine.
The agents however, were stuck in an endless upgrade loop.

I tried stopping them, wiping out the local cache/workspaces on the agents.  I also downloaded buildagent.zip and unpacked it but to no avail. It seems to want to upgrade forever.

12 comments
Comment actions Permalink

Any thoughts on this?  I just wiped out my data directory and recreated the MySQL database and I still have this issue.  In other words, I seem to have this "upgrade loop" even with a fresh installation.  Also, what is the point of downloading buildAgent.zip and unpacking it on the client if the server is going to force it to upgrade anyway?

0
Comment actions Permalink

Agents upgrade from 6.0 to 6.5 is two phased. Build agent will download updates twice.
If it failed, please attach build agent logs.

If upgrade is failed, please do the following:
- stop agent
- kill all java processes on machine
- remove folders: plugins, tools, system
- start agent again
- wait until agent is connected to server (this could take up to 10-15 minutes)
- do not logoff while waiting to the result of upgrade.

0
Comment actions Permalink

Hi Eugene,

This was  a fresh installation of the agent.  I wiped out everything and unpacked the buildAgent.zip from the TC server on the agent and started it up.  Hence my confusion why the agent needs to do an upgrade.

I just did the same steps again and generated new logfiles which are attached.



Attachment(s):
agentlogs.zip
0
Comment actions Permalink

Thank you for logs.

Please attach conf/buildAgent.properties file.

While upgrade is running, please try to copy and attach those files:

   D:\teamcity\agent\update\teamcity-agent.xml
   D:\teamcity\agent\update\teamcity-agent.xml.move

0
Comment actions Permalink

Attached is conf.zip that has the files you requested.  The configuration file is actually called buildAgent.properties.vm-ntdxbld9. In wrapper.conf, we have this set:

wrapper.app.parameter.10=d:/teamcity/conf/buildAgent.properties.vm-ntdxbld9



Attachment(s):
conf.zip
0
Comment actions Permalink

Please try reverting all config changes you've done.
Attach configs you patched.

Please revert buildAgent.properties.

Does build agent have enough rights to access both d:/teamcity/agent and d:/teamcity/agentdir folders?

Do you have files under d:/teamcity/agentdir/system? Please attach the folder.

I noticed that build agent was installed to d:/teamcity/agent, but in configuration files of agent there is d:/teamcity/agentdir specified.

I added more logging to build agent code. I'll attach a patch for you as build finishes.

0
Comment actions Permalink

Here's a diff of the agent conf file versus the original conf file:

D:\teamcity\conf>diff buildAgent.properties.vm-ntdxbld9 \orig\conf\buildAgent.dist.properties
8c8
< serverUrl=http://uxoslnxbld2.bedford.progress.com:8111/
---
> serverUrl=http://localhost:8111/
12c12
< name=vm-ntdxbld9
---
> name=
15c15
< workDir=d:/teamcity/agentdir/work
---
> workDir=../work
19c19
< tempDir=d:/teamcity/agentdir/temp
---
> tempDir=../temp
22c22
< systemDir=d:/teamcity/agentdir/system
---
> systemDir=../system
23a24
>
41c42
< authorizationToken=56a1c819648e6ccf0f96e6afc6f49c06
---
> authorizationToken=
42a44
>
54,60d55
< system.product.group=dxsi
< #system.cleartool.path=C:/Program Files/IBM/RationalSDLC/ClearCase/bin/cleartool.exe
< #system.cleartool.path=C:\\Program Files\\IBM\\RationalSDLC\\ClearCase\\bin\\cleartool
< #system.cleartool.path=C:\Program Files\IBM\RationalSDLC\ClearCase\bin\cleartool
< #system.cleartool.path=C:\\Program Files\\IBM\\RationalSDLC\\ClearCase\\bin\\cleartool
< #system.cleartool.path="C:\Program Files\IBM\RationalSDLC\ClearCase\bin\cleartool"
< #system.cleartool.path="C:\\Program Files\\IBM\\RationalSDLC\\ClearCase\\bin\\cleartool"
62d56
<
70,72c64
< #env.exampleEnvVar=example Env Value
<
<
---
> #env.exampleEnvVar=example Env Value


This is the same *.conf file that worked fine with 6.0.1 & earlier versions.

As for the system dir, there's really nothing in it.  I've gone through this exercise multiple times.  Originally, I had tried upgrading from 6.0.1 to 6.5.  Then when I had issues, I started from scratch and wiped out my 6.0.1 prototype system.


D:\teamcity\agentdir>dir
Volume in drive D is New Volume
Volume Serial Number is 2C93-F102

Directory of D:\teamcity\agentdir

03/01/2011  11:15 AM    <DIR>          .
03/01/2011  11:15 AM    <DIR>          ..
03/01/2011  11:16 AM    <DIR>          system
03/02/2011  07:13 AM    <DIR>          temp
03/01/2011  11:15 AM    <DIR>          work
               0 File(s)              0 bytes
               5 Dir(s)  18,042,429,440 bytes free

D:\teamcity\agentdir>dir system
Volume in drive D is New Volume
Volume Serial Number is 2C93-F102

Directory of D:\teamcity\agentdir\system

03/01/2011  11:16 AM    <DIR>          .
03/01/2011  11:16 AM    <DIR>          ..
03/02/2011  07:12 AM    <DIR>          .teamcity-agent
03/01/2011  11:16 AM    <DIR>          swabra
               0 File(s)              0 bytes
               4 Dir(s)  18,042,429,440 bytes free

D:\teamcity\agentdir>dir system\swabra
Volume in drive D is New Volume
Volume Serial Number is 2C93-F102

Directory of D:\teamcity\agentdir\system\swabra

03/01/2011  11:16 AM    <DIR>          .
03/01/2011  11:16 AM    <DIR>          ..
               0 File(s)              0 bytes
               2 Dir(s)  18,042,429,440 bytes free

D:\teamcity\agentdir>

0
Comment actions Permalink

So here's what I did on the agent:

1. Wiped out d:\teamcity
2. Created d:\teamcity\agent directory. Unzipped buildagent.zip into it.
3. Created d:\teamcity\conf which has all of our *.conf files for the various agents.
4. Modified d:\teamcity\agent\launcher\conf\wrapper.conf

D:\teamcity\agent\launcher\conf>diff wrapper.conf wrapper.conf.template
11c11
< wrapper.java.command=c:/java/sun/jdk1.6.0_14/bin/java.exe
---
> wrapper.java.command=java
32a33,39
> ###########################################################
> ### TeamCity agent JVM parameters
> ###
> ### NOTE: There should be no gaps in parameters numbers, if
> ### NOTE: you change parameters, you need to update numbering
> ###
> ##########################################################
35,36d41
<
< # TeamCity agent JVM parameters
39c44
< # The next line can be removed (and the rest of the lines renumbered) to preve
nt memory dumps on OutOfMemoryErrors
---
> # The next line can be removed (and the rest of the parameter names MUST BE re
numbered) to prevent memory dumps on OutOfMemoryErrors
50c55
< wrapper.app.parameter.10=d:/teamcity/conf/buildAgent.properties.vm-ntdxbld9
---
> wrapper.app.parameter.10=../conf/buildAgent.properties

D:\teamcity\agent\launcher\conf>


5. Startup the agent (d:\teamcity\agent\bin\service.start)

The agent creates the d:\teamcity\agentdir directory.  Yes, it is able to mdoify the d:\teamcity\agent directory as it creates the logs directory and when you authorize the agent, it modifies the d:\teamcity\conf\buildAgent.properties.vm-ntdxbld9 file.

When unzipping, installing the agent, I'm using the account PROGRESS\dxbld.  This is the same account that the agent runs as an NT service using the logon privs of PROGRESS\dxbld.

0
Comment actions Permalink

Any update on this issue?

0
Comment actions Permalink

Sorry for delay,

I managed to reproduce your issue. I created an issue for that in the issue tracker at http://youtrack.jetbrains.net/issue/TW-15720. Please vote for it.
The workaround is to avoid pathing agent's config files and make all settings in <agent>/conf/buildAgent.properties file.

Thanks!

0
Comment actions Permalink

There might be another bug here.  Why is the agent deciding to upgrade?  I took the buildAgent.zip from the server and unzipped it.  There shouldn't be anything that needs to change or be upgraded.

I can understand the agent doing this if it was running 6.0.2 or some other version but the buildAgent.zip was from the TC server running 6.5 EAP - Build 17132.

0
Comment actions Permalink

The bug was in non-defailt systemDir parameter in buildAgent.properties file. I've fixed the regression. The fix will be available in the next after next EAP.

0

Please sign in to leave a comment.