Disconnected agents after upgrade

I upgraded from 4.0.1 to 4.0.2 today. After upgrading, I noticed no agents were connected -- I'm assuming when you uninstall it shuts down the agent services. So I started them up (I have 3). I waited awhile because I know from past experience the first thing they do is check for upgrades ... but they never connected to my server. So I checked the log files and it looked like they failed to upgrade themselves.

So I downloaded the latest build agent bin from my server, uninstalled their services, and copy/pasted the new code over top my old agents. I then copied my old buildAgent.properties over top of the updated one and I also edited my wrapper.conf file to have unique service names (I forgot to back them up). Once I did that I added the services ... but now they don't start.

I checked out my wrapper.conf file and found that the agent service cannot locate java as this thread also experienced:
http://www.jetbrains.net/devnet/thread/279382

But my problem is that I already do have a jre folder in my agent folder. So I'm at a loss for ideas. Here are some details:

Directory of D:\ContinuousIntegration\Tools\Agent1
02/04/2009  05:13 PM    <DIR>          bin
02/04/2009  05:13 PM                 0 BUILD_8222
02/04/2009  05:14 PM    <DIR>          conf
12/03/2008  09:29 AM    <DIR>          jre
02/04/2009  05:13 PM    <DIR>          launcher
02/04/2009  05:13 PM    <DIR>          lib
02/04/2009  04:59 PM    <DIR>          logs
02/04/2009  04:29 PM    <DIR>          temp
12/03/2008  09:31 AM            87,010 Uninstall.exe
02/04/2009  05:00 PM    <DIR>          update
02/03/2009  04:32 PM    <DIR>          work


Directory of D:\ContinuousIntegration\Tools\Agent1\jre\bin
05/01/2007  05:22 PM            49,248 java.exe
05/01/2007  07:01 PM            45,171 javacpl.exe
05/01/2007  05:23 PM            53,346 javaw.exe
... others omitted ...


D:\ContinuousIntegration\Tools\Agent1\conf>more buildAgent.properties
#Generated properties (see buildAgent.dist.properties for description)
#Wed Dec 03 09:31:38 EST 2008
name=DEVACI01 - Agent 1
ownPort=9090
serverUrl=http\://localhost\:88
workDir=D\:\\ContinuousIntegration\\Tools\\Agent1\\work
tempDir=D\:\\ContinuousIntegration\\Tools\\Agent1\\temp
#User defined properties
#Wed Dec 03 09:31:38 EST 2008
env.TEAMCITY_JRE=D\:\\ContinuousIntegration\\Tools\\Agent1\\jre
authorizationToken=02c79c41d4203614a7f3cd64fd904269


The only thing I changed in my wrapper.conf file are these properties:
# Name of the service
wrapper.ntservice.name=TCBuildAgent1
# Display name of the service
wrapper.ntservice.displayname=TeamCity Build Agent 1 Service
# Description of the service
wrapper.ntservice.description=Build Agent 1


Any ideas on how to make my agent see my jre folder? Once I saw that env.TEAMCITY_JRE I was out of ideas.


Thanks,

Dan Lash

6 comments
Comment actions Permalink

Build agent is tarted usgin Java Service Wrapper. You may leave wrapper.java.command= property blank to make JSW search for any installed Java on you machine.
buildagent.zip contain more defailt vertsion of wrapper.conf file.
The most stable solution is to set there path to any JRE you have on your machine.
Windows installed/Java web start installer does this job for you.

BTW. How long have you been waiting for build agents to come back after upgrading server? Agent upgrade may take up to several minutes, and while this process is running you may see that agent service is stopped. Do you have any logs from that broken agents?

Thanks!

0
Comment actions Permalink

Unfortunatley when I copy/pasted the agent files, I overwrote my logs. I did look at them beforehand though, and I saw some things about 'failed to upgrade' and the like which is why I decided to do it manually. I had waited about 5-7 minutes before checking the logs after starting the services.

You mentioned that I can point to any java installed on my machine -- I do not have any other java other than what is included in the agent package (that I know of). And as I mentioned it looked like my agents were pointing correctly using env.TEAMCITY_JRE inside of their buildAgent.properties files.

I noticed in the buildAgent.dist.properties file that the paths were using relative notation ..\work etc ... but my buildAgent.properties files use absolute notation ... could that be a problem? Also does maybe the env.TEAMCITY_JRE need to point to the bin sub directory instead of the jre directory?

Here is my wraper.log file:
STATUS | wrapper  | 2009/02/05 08:58:08 | --> Wrapper Started as Service
STATUS | wrapper  | 2009/02/05 08:58:08 | Launching a JVM...
FATAL  | wrapper  | 2009/02/05 08:58:08 | Unable to execute Java command.  The system cannot find the file specified. (0x2)
FATAL  | wrapper  | 2009/02/05 08:58:08 |     "java" -Xrs -Djava.library.path="../launcher/lib;../launcher/bin" -classpath "../launcher/lib/wrapper.jar;../launcher/lib/launcher.jar" -Dwrapper.key="ztAh2LghWtP76cjC" -Dwrapper.port=32000 -Dwrapper.jvm.port.min=31000 -Dwrapper.jvm.port.max=31999 -Dwrapper.pid=3728 -Dwrapper.version="3.2.3" -Dwrapper.native_library="wrapper" -Dwrapper.service="TRUE" -Dwrapper.cpu.timeout="10" -Dwrapper.jvmid=1 org.tanukisoftware.wrapper.WrapperSimpleApp jetbrains.buildServer.agent.StandAloneLauncher -ea -Xmx512m -XX:+HeapDumpOnOutOfMemoryError -Xrs -Dlog4j.configuration=file:../conf/teamcity-agent-log4j.xml -Dteamcity_logs=../logs/ jetbrains.buildServer.agent.AgentMain -file ../conf/buildAgent.properties
FATAL  | wrapper  | 2009/02/05 08:58:08 | Critical error: wait for JVM process failed

Thanks,

Dan Lash

0
Comment actions Permalink

I have found my upgrade.log file which made me want to do the upgrade manually:

[2009-02-04 17:02:32,113]   INFO - r.agent.upgrade.UpgradeRunBase - Start agent upgrade.
[2009-02-04 17:02:32,128]   INFO - r.agent.upgrade.UpgradeRunBase - Agent home folder: D:\ContinuousIntegration\Tools\Agent2
[2009-02-04 17:02:32,128]   INFO - r.agent.upgrade.UpgradeRunBase - Run command line (arguments separated with '|': cmd.exe|/c|D:\ContinuousIntegration\Tools\Agent2\bin\service.stop.bat| (at D:\ContinuousIntegration\Tools\Agent2\bin)
[2009-02-04 17:02:32,800]   WARN - .agent.run.ProcessOutputLogger - 'D:\ContinuousIntegration\Tools\Agent2\bin\service.stop.bat' is not recognized as an internal or external command,
[2009-02-04 17:02:32,800]   WARN - .agent.run.ProcessOutputLogger - operable program or batch file.
[2009-02-04 17:02:32,800]   WARN - .agent.run.ProcessOutputLogger - 
[2009-02-04 17:02:32,800]  ERROR - r.agent.upgrade.UpgradeRunBase - Upgrade failed...
java.io.IOException: Failed to start process. Return code was 1 not equal to zero.
at jetbrains.buildServer.agent.upgrade.UpgradeRunBase.runCmdLine(UpgradeRunBase.java:66)
at jetbrains.buildServer.agent.upgrade.NTServiceUpgrade.exec(NTServiceUpgrade.java:36)
at jetbrains.buildServer.agent.upgrade.NTServiceUpgrade.beforeUpgrade(NTServiceUpgrade.java:22)
at jetbrains.buildServer.agent.upgrade.UpgradeRunBase.run(UpgradeRunBase.java:28)
at jetbrains.buildServer.agent.upgrade.UpgradeMode$2.run(UpgradeMode.java:26)
at jetbrains.buildServer.agent.upgrade.Upgrade2.main2(Upgrade2.java:33)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at jetbrains.buildServer.agent.ClazzLoader.call(ClazzLoader.java:59)
at jetbrains.buildServer.agent.ClazzLoader.callMain2(ClazzLoader.java:19)
at jetbrains.buildServer.agent.upgrade.Upgrade.main(Upgrade.java:16)
[2009-02-04 17:05:31,467]   INFO - r.agent.upgrade.UpgradeRunBase - Run command line (arguments separated with '|': cmd.exe|/c|D:\ContinuousIntegration\Tools\Agent2\bin\service.start.bat| (at D:\ContinuousIntegration\Tools\Agent2\bin)
[2009-02-04 17:05:31,514]  ERROR - r.agent.upgrade.UpgradeRunBase - java.io.IOException: Failed to start process. Return code was 1 not equal to zero.
java.io.IOException: Failed to start process. Return code was 1 not equal to zero.
at jetbrains.buildServer.agent.upgrade.UpgradeRunBase.runCmdLine(UpgradeRunBase.java:66)
at jetbrains.buildServer.agent.upgrade.NTServiceUpgrade.exec(NTServiceUpgrade.java:36)
at jetbrains.buildServer.agent.upgrade.NTServiceUpgrade.startAgain(NTServiceUpgrade.java:26)
at jetbrains.buildServer.agent.upgrade.UpgradeRunBase.run(UpgradeRunBase.java:40)
at jetbrains.buildServer.agent.upgrade.UpgradeMode$2.run(UpgradeMode.java:26)
at jetbrains.buildServer.agent.upgrade.Upgrade2.main2(Upgrade2.java:33)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at jetbrains.buildServer.agent.ClazzLoader.call(ClazzLoader.java:59)
at jetbrains.buildServer.agent.ClazzLoader.callMain2(ClazzLoader.java:19)
at jetbrains.buildServer.agent.upgrade.Upgrade.main(Upgrade.java:16)

0
Comment actions Permalink

Here is my launcher.log if that helps:

[2009-02-04 17:02:27,085]   INFO - ldServer.agent.run.AgentRunner - Agent process finished
[2009-02-04 17:02:27,086]   INFO - ldServer.agent.run.AgentRunner - Agent has exited with code: 10
[2009-02-04 17:02:27,088]   INFO - ldServer.agent.run.AgentRunner - Updates directory exists
[2009-02-04 17:02:27,137]   INFO - buildServer.agent.LauncherUtil - Listing files in D:\ContinuousIntegration\Tools\Agent2\update\launcher\lib\launcher.jar to check if jetbrains/buildServer/agent/upgrade/Upgrade.class contained.
[2009-02-04 17:02:27,208]  DEBUG - r.agent.upgrade.UpgradeStarter - Found compatible launcher version.
[2009-02-04 17:02:27,230]   INFO - ildServer.agent.run.JavaRunner - Starting new process with command line D:\ContinuousIntegration\Tools\Agent2\jre\bin\java -classpath D:\ContinuousIntegration\Tools\Agent2\update\launcher\lib\launcher.jar jetbrains.buildServer.agent.upgrade.Upgrade NT_SERVICE D:\ContinuousIntegration\Tools\Agent2 -ea -Xmx512m -XX:+HeapDumpOnOutOfMemoryError -Xrs -Dlog4j.configuration=file:../conf/teamcity-agent-log4j.xml -Dteamcity_logs=../logs/ jetbrains.buildServer.agent.AgentMain -file ../conf/buildAgent.properties 
[2009-02-04 17:02:27,286]   INFO - ldServer.agent.run.AgentRunner - Launcher is waiting to be killed by upgrade process

0
Comment actions Permalink

Ok I seem to have found the problem ... for whatever reason the wrapper.java.command property has changed in this upgrade -- or at least it stopped working the way it used to. All of my agents were configured to use 'java' as that property, but after doing an install from the agent MSI, I noticed it had a different value. So I set my agent's to use that new value and they all seem to be working -- it takes 5 minutes for them to update themselves, but they're working.

Then new value is:   ../jre/bin/java


Hope that helps anyone else with that problem.


Dan Lash

0
Comment actions Permalink

Use ../jre/bin/java as the wrapper.java.command value in wrapper.conf

0

Please sign in to leave a comment.