Custom checkout directory causes strange errors
Answered
TeamCity Professional 2019.2.3 (build 72031)
Setting custom checkout directory to %teamcity.build.default.checkoutDir% or to %system.teamcity.buildType.id% works fine, however setting it to %teamcity.build.default.checkoutDir%%system.teamcity.buildType.id% causes an out of memory error.
Full checkout enforced. Reason: [Checkout directory is empty or doesn't exist]
Will perform clean checkout. Reason: Checkout directory is empty or doesn't exist
Checkout directory: D:\TeamCity\buildAgent\work\db5cda79c6004db8<ProjectName>_FullBuild
Updating sources: auto checkout (on agent) 3m:33s
Will use agent side checkout
VCS Root: Plastic 3m:33s
Failed to perform checkout on agent: Problem while checkout on agent: java.lang.OutOfMemoryError: Java heap space
Error message is logged
The absolute path D:\%teamcity.build.default.checkoutDir%%system.teamcity.buildType.id% seems to work fine as well. Which leads me to believe (after a few other tests) that there is some issue with perhaps path length in the custom checkout directory field. Could be a TeamCity issue or a plug-in issue or (most likely) some configuration issue on our part.
Please sign in to leave a comment.
Are you using the same field for the custom and absolute path as in this screenshot? If not, where are you entering the custom path?
Yes I'm using this custom directory, and amazon latest version of Java as recommended. Java memory looks fine and is set to high numbers. Shorter resulting paths seem to work.
%teamcity.agent.home.dir%/../%system.teamcity.buildType.id%/%straykite.build.branch.clean%
> Setting custom checkout directory to %teamcity.build.default.checkoutDir% or to %system.teamcity.buildType.id% works fine, however setting it to %teamcity.build.default.checkoutDir%%system.teamcity.buildType.id% causes an out of memory error.
Can you provide an example of each parameter that work independently, but produces the error when used together? Or is it that you're using %teamcity.agent.home.dir%/../%system.teamcity.buildType.id%/%straykite.build.branch.clean% to cause the error?
> Failed to perform checkout on agent: Problem while checkout on agent: java.lang.OutOfMemoryError: Java heap space
Since this problem appears to be occurring during checkout on the agent, what are your Java memory settings for the agent? You can look in the teamcity-agent.log file to verify the memory settings. It should be available in <build agent directory>/logs. You should find something like this when the agent starts up and should contain the JVM parameters for the agent. What is your Xmx setting?
> Error message is logged
You may also find some useful information on the out of memory error you're seeing in this logfile. Do you see the error in the teamcity-agent.log? If so, what do you see?
It doesn't seem to have anything to do with the parameters specifically but rather just the length of the path. If I hard-code the path, I get the same issue. There is an example in the error log below of one that does not work. If you split it up and use either half, it works.
Memory settings:
Error from the build agent log file:
You say you get the same out of memory error if you use this as the Custom Path:
D:\TeamCity\buildAgent\work\db5cda79c6004db8Hydrogen_FullBuild
But earlier you said if you used this, it worked:
D:\%teamcity.build.default.checkoutDir%%system.teamcity.buildType.id%
Looking at your build log, it appears your %teamcity.build.default.checkoutDir% is D:\TeamCity\buildAgent\work\db5cda79c6004db8 and your %system.teamcity.buildType.id% is Hydrogen_FullBuild. The length of the path in your log file is not very long, I would be surprised if you're running into a limit here. I have seen much longer file paths used successfully. What is the resulting path when it works?
What other testing have you done that leads you to believe it is the custom path causing your out of memory error? As an experiment, could you try increasing the Agent jvm settings from -Xmx512m to -Xmx1g?
If you're running the Build Agent as a service, this setting is in the wrapper config file. See here for reference: https://www.jetbrains.com/help/teamcity/configuring-build-agent-startup-properties.html#ConfiguringBuildAgentStartupProperties-BuildAgentIsRunAsService
teamcity.build.default.checkoutDir is defined as the checkout directory as if no custom checkout directory had been specified, so in your example case, this should resolve to db5cda79c6004db8. Are you saying that parameter resolves to an absolute path and not just the directory name?
Shorter paths such as D:\checkout\Hydrogen_FullBuild work fine. This is my current work-around and all builds have been fine for the last two weeks. If I change it to make it longer again, it breaks with the out of memory error. If I turn off custom paths, it works fine. So while it might not be that setting by itself, that is the cause as far as I am concerned as an end user.
Increased to 1g and restored a longer path:
%teamcity.agent.home.dir%/tempcheckout/trysomethinglong/%system.teamcity.buildType.id%/%teamcity.build.branch%
Reset path to:
%teamcity.agent.home.dir%/../%system.teamcity.buildType.id%/%straykite.build.branch.clean%
with no other changes, build successful.
Current wrapper memory settings on the agent I tested on below. I don't think most of these are necessary changes, I was just trying everything to make this error go away. Would love to know what's needed / unnecessary.
Sorry, you are correct on the parameter values. I'll chalk that one up to it being Monday. Regarding the wrapper.conf file, there are two sections, one for the launcher and one for the agent JVM. The most important setting is the -Xmx memory, which is the maximum memory allocation for the JVM. How much memory is available on the agent server? It looks like it is probably running on the TeamCity Server, is this right? If that is the case, can you screenshot the Administration | Diagnostics | Memory Usage chart? It would be especially helpful while the build is running.
These settings are for the launcher and are not normally needed to be changed at all. I'd recommend changing these back to the defaults (see my settings further below):
These are the agent JVM settings and would be much more likely to have an impact on your build:
Here are the settings from my wrapper.conf, which are about the most basic settings you can use if you find it helpful:
Okay I've changed everything to match your warpper.conf except for -Xmx1g and -server (assuming this is meant to be set if the agent is on the server? if not I'll remove this as well).
We do have two other agents that are not on the server which both exhibit the same issue. The server has 128GB of RAM (most of which is unused, currently at 38/128 with no build running), the other agents have 64GB.
Prior to build starting:
During build:
After build failed to start:
Thanks for the additional information on your environment and the charts. I am not sure you need to use the -server option, but it is unlikely to affect your issue either way. My understanding of the -server option is that it will allow for faster throughput at the cost of longer startup times.
One more quick experiment would be to try setting your Xmx to an extreme value like 4gb?
If you still get the out of memory error at that point, we can look at your memory dump from the error for anything useful. There should be one created with each error in the <build agent home>/bin directory. Look for a file like: java_pidxxxx.hprof. You can upload it to me on our FTP, instructions are here: https://www.jetbrains.com/help/teamcity/reporting-issues.html#ReportingIssues-UploadingLargeDataArchives. Just be sure to let me know the exact filename so I can find it.
Increased to 4g and got:
Thanks so much, this is very helpful. It looks like you have uncovered something I will need to discuss with our developers. I just have a couple of follow-up questions. Which type of VCS are you using for this build configuration? Has this same build configuration worked in earlier releases of TeamCity?
No problem.
Plastic SCM is the VCS.
This has not worked for us previously, but we just started with TeamCity about a month ago.
Should I keep the memory setting at 4g?
The limitation you're seeing is a component of the Plastic SCM plugin. While it looks like a workaround would be to use a shorter checkout directory, as you had already discovered, I would suggest reaching out to the vendor of the plugin to let them know of the issue so they can have the opportunity to patch it. If you're using the Plastic SCM from our plugin repository, it looks to be directly from the makers of Plastic SCM and they have support on their website here: https://www.plasticscm.com/documentation/technical-articles/how-to-integrate-plastic-scm-with-teamcity-ci.
The memory setting of 4gb would be a bit higher than average, however, it did resolve your out of memory error. Since it appears that you have plenty of memory available on the server, you won't hurt anything by leaving it there.
Okay, thanks Eric. Were you able to reproduce the issue?
I have a support thread going with them on this issue as well. Didn't know where the problem was so figured it was best to look at it from both ends.
No problem, it was an interesting issue to look at and I think we got your java settings on your build agent straightened out if nothing else. I was not able to reproduce your issue on my end, but I do not have any Plastic SCM VCS roots to experiment on.