Automated termination of related builds

We build C/C++ code cross-platform on Windows, Linux, and Mac. Some of our builds are faster than others. I am investigating if it is possible to terminate a build or multiple builds that may be either still queued on running on other platforms and/or on the same machine. We also build for 32-bit and 64-bit, and have separate builds for each.

Our source control is handled by Team Foundation Server.

What I want to do is if any personal build fails on any platform, the remaining builds should be terminated from that point forward. The thought is that if it failed in one place, it will more than likely fail in every build. So to conserve server resources, I would like to see if I might be able to add a build step at the end of each build that checks to see if the build was successful up to that point. If it failed, then terminate the other related builds. Is this possible, and if so, can anyone provide any guidance on how to proceed with this task.

I think in order for this to work, the custom script would need to get the version of the code in TFS, the name or version of any shelveset code for the changes, and perhaps the name of the person that queued the changes. Assuming that can all be obtained through the build info, is it possible with the TeamCity API to then go and programmatically terminate other running or queued builds?

Thanks

1
2 comments

Hi Walter,

 

there are some facilities within builds that would ease your task via a custom script:

-You have the revisions being run by the build by accessing the %build.vcs.number%, and there are several sub-parameters for each vcs root in the case you are using multiple.

-You can use the REST API to stop/cancel builds. This article tells you what you would need: https://www.jetbrains.com/help/teamcity/rest-api.html#RESTAPI-BuildCanceling/Stopping

0
Avatar
Permanently deleted user

I am close to having this done. I was able to query the TeamCity API to get the build information I need. I ended up having to query the API to get the status of the build the utilty was running in because this information is not available in the build steps themselves as an environment variable nor is it [currently] possible to have a build step that will only run when a failure has occurred. More information on this can be found here:
https://youtrack.jetbrains.com/issue/TW-17975

 

That was easy enough to work around.

 

I also learned that there's problems with using the provided TeamCityBuildID and password as part of the build to use with authenticating with the API. This has been resolved where the build is trying to prematurely terminate itself. See the following:
https://youtrack.jetbrains.com/issue/TW-39206

 

My problem now is that I am trying to kill OTHER running or queued builds from within TeamCity. Since the fix for TW-39206 does not address my use case and because I was getting HTTP error code 402, I decided to switch to other known credentials. After changing to known credentials, my requests are failing with an HTTP status code of 401: Unauthorized.

 

I know this code works because I tested it in a separate .NET Core console application using both my personal credentials and a service account set up to run the TeamCity build agents. Yet when I run it under the build, it fails.

 

Is there some security issue I am running up against here? Why is this not working under the build?

 

I should note that we are currently using 2018.2.4 (build 61678). I know there is a newer version, but I have not tested this against it yet.

 

Here's the C# code I'm using:

 

string queryString = string.Empty;
string failureComment = $"Terminated due to build failure of build {failedBuildNumber}";

using (HttpClient client = new HttpClient())
{
client.DefaultRequestHeaders.Accept.Clear();

var byteArray = System.Text.Encoding.ASCII.GetBytes(string.Format("{0}:{1}", username, password));
client.DefaultRequestHeaders.Authorization = new System.Net.Http.Headers.AuthenticationHeaderValue("Basic", Convert.ToBase64String(byteArray));
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/xml"));
client.DefaultRequestHeaders.Add("Origin", @"http://teamcity.leadtools.com");

StringContent data = new StringContent($"<buildCancelRequest comment='{failureComment}' readdIntoQueue='false' />");
data.Headers.ContentType = new MediaTypeWithQualityHeaderValue("application/xml");

foreach (Build bld in buildList)
{
queryString = $"{Constants.BASE_URL_TEAMCITY}{Constants.TEAMCITY_URL_VERSION}/builds/id:{bld.ID}";

if (killTheBuild)
{
try
{
var stringTask = client.PostAsync(queryString, data);

var msg = await stringTask;
if (msg == null)
Console.WriteLine($"HTTP Response in KillBuilds() is null for the following query: \"{queryString}\"");
else if (msg.StatusCode == System.Net.HttpStatusCode.OK)
{
Console.WriteLine($"Kill message for {bld.ID} sent successfully");
}
else
{
Console.WriteLine("++++++++++++++++++++++++++++++++++++++++++++++++++++++++");
Console.WriteLine($"Kill message info for {bld.ID}: {msg.ToString()}");
Console.WriteLine("++++++++++++++++++++++++++++++++++++++++++++++++++++++++");
}
}
catch (Exception ex)
{
Console.WriteLine($"KillBuilds() queryString: {queryString}");
Console.WriteLine($"Exception: {ex.Message}");
throw ex;
}
}
}
}

 

0

Please sign in to leave a comment.