Setting environment variables from build fails silently sporadically
Hi,
for many of my builds, I set a environment variable from a metarunner. It has 2 "runners", one that sets the variable, one that dumps all vars to double check ("Get-ChildItem Env:" in Powershell). See below for the metarunner definition.
Usually, this works fine,and I can see in the log both the "##" line that sets the variable, and the corresponding variable in the dump.
However, once in while (like every 40 runs), I can see the "##" line as usual, but in the next step, it's not in the dump. Since subsequent build steps depend on this variable, the whole builds fail, and the upstream builds too.
Has anyone see this before ? Any idea about the cause or further troubleshooting steps ?
Cheers
Olivier
Edit: using TeamCity 8.1 (build 29879)
<?xml version="1.0" encoding="UTF-8"?>
<meta-runner name="Extend TeamCity environment variables">
<description>Extend TeamCity environment variables</description>
<settings>
<parameters />
<build-runners>
<runner name="Extend TeamCity environment variables" type="jetbrains_powershell">
<parameters>
<param name="jetbrains_powershell_bitness" value="x86" />
<param name="jetbrains_powershell_execution" value="STDIN" />
<param name="jetbrains_powershell_noprofile" value="true" />
<param name="jetbrains_powershell_script_code"><![CDATA[$host.UI.RawUI.BufferSize = New-Object System.Management.Automation.Host.Size(8192,50)
echo "##teamcity[setParameter name='env.VAR' value='...']"
# End of PowerShell]]></param>
<param name="jetbrains_powershell_script_mode" value="CODE" />
<param name="teamcity.step.mode" value="default" />
</parameters>
</runner>
<runner name="Print environment variable with PowerShell" type="jetbrains_powershell">
<parameters>
<param name="jetbrains_powershell_bitness" value="x86" />
<param name="jetbrains_powershell_execution" value="STDIN" />
<param name="jetbrains_powershell_noprofile" value="true" />
<param name="jetbrains_powershell_script_code" value="Get-ChildItem Env:" />
<param name="jetbrains_powershell_script_mode" value="CODE" />
<param name="teamcity.step.mode" value="default" />
</parameters>
</runner>
</build-runners>
<requirements />
</settings>
</meta-runner>
Please sign in to leave a comment.
Olivier,
Would it be possible to attach raw build logs (avaialble for download when viewing build log with "Verbose" level) for several builds which do not have the environment set and for several working in due way?
Hello Yegor,
please find below the raw log. I just replaced internal names by placeholders.
Best regards
Olivier
One of our nightly builds failed early today because of this issue :( I checked the server logs, there isn't anything suspicious there.
I just set up a troubleshooting build configuration that runs every minute, trying to reproduce the issue.
Is there anything else I can do on my side to pinpoint the origin of the problem ?
In case it makes any difference, the build configurations run on 4 agents running on the same Windows 7 machine, 32bit.
Olivier,
Would it be possible to attach raw build log files? These come in an archive, see my previous comment.
Could you please also attach those for both bad and good builds?
Please also attach .teamcity/properties/build.finish.properties.gz files from the hidden build artifacts for the builds.
So when this reproduces none of the set environment variables are passed to te next step?
Would it be possible to use plan command line runner instead of PowerShell to eliminate possible PowerShell issues?
Please also keep us updated on the results you have with the testing build.
Thanks for the feedback. I just sent you the requested files by email.
The values of the variables are computed dynamically, and the runners do even more than that, so CMD is not an option. It's too cumbersome and limited for this purpose.
Olivier,
Quite strange. I cannot reproduce this locally...
How often does this reprodcue?
What are the results of the troubleshooting build you have set up?
Seems that this is the consequence of an internal "NoRunningBuildException" error.
I have filed the case as http://youtrack.jetbrains.com/issue/TW-35871
Thank you Yegor. I hope it will be an easy bug to fix :)
A few questions:
1) Does the problem occur for all configurations or the only one? Or all ones that contain meta-runner you've mentioned?
2) Does it happen for all build agents, or just a single one?
I suspect, it might be a networking issue, when agent receives signal to stop the build, while it's in process of execution. And since build type uses service messages, it fails.
It happened last night again. The same exception could be found in the agent log (NoRunningBuildException).
We have 4 agents running on the same machine, the problem occurs on at least 3.
All our builds are based on a template that uses this meta-runner. Several of these builds have been impacted in the past.
The "troubleshooting build configuration" never failed so far (it's running every 5 minutes). It's based on the same template as the productive builds.
The network issue sounds interesting, I will investigate in this direction. However I don't see why an agent would receive a signal to stop the build.
Please attach build log and teamcity-agent.log to the issue http://youtrack.jetbrains.com/issue/TW-35871.
Actually, I fixed the issue. Will have a build for you to try tonight or tomorrow.
That's great news, thanks !
Please try this: ftp://ftp.intellij.net/pub/.teamcity/TW-35871/TeamCity-29981.tar.gz
Thanks Sergey. I have a few questions, to help us defining our stategy to test and deploy this build.
This build is a build from branch that includes this bugfix (and other bugfixes as well).
As long as this one works properly, I would recommend you to update to 8.1.2 which is going to be released this week and that cointains the bugfix as well.
We will wait for 8.1.2. Thank you for the great support.
I have installed 8.1.2 three days ago, and I didn't see the problem since. It looks good. Thanks !