Force TeamCity to fetch additional branches

Answered

On the builds server I have set up TeamCity (8.1.1) so that  it executes the build process if there are changes in either the master,  one of the feature branches or one of the pull request branches using  the branch specifier:


    +:refs/heads/*     +:refs/pull/(*/merge)


I have turned on the build agent option:


    teamcity.git.use.local.mirrors=true


which clones the repository in a directory outside the build directory and then pulls from that local repository.

The build process needs access to the git repository and the master  branch, even for builds of one of the feature branches or pull request  branches. However TeamCity only has the branch that contains the changes  in the local repository thereby making my builds fail, e.g. when the  change was on the issue/mycoolissue branch then that is the only branch  that exists in the git repository in the TeamCity working space.

I have tried performing a local

git fetch
to get the  master branch but because the local repository does not have the master  branch this fails. While I could add a remote pointing to the origin (a  github private repository) that would mean that I'd have to handle  credentials as well and I'd rather have TeamCity take care of all of  that for me.

My question is whether there is a way to tell TeamCity to just pull  all the branches into both the local repository and the working  repository?
8 comments
Comment actions Permalink

Hi, can you please clarify this: "However TeamCity only has the branch that contains the changes  in the local repository thereby making my builds fail, e.g. when the  change was on the issue/mycoolissue branch then that is the only branch  that exists in the git repository in the TeamCity working space.

I have tried performing a local

git fetch
to get the  master branch but because the local repository does not have the master  branch this fails. While I could add a remote pointing to the origin (a  github private repository) that would mean that I'd have to handle  credentials as well and I'd rather have TeamCity take care of all of  that for me."

I didn't get it.
0
Comment actions Permalink

When execute a

    git branch

command on the working directory I get

    > git branch
    * issue/mybranch

where issue/mybranch is the branch that had the changes. So as far as I can tell that means that TeamCity only pulled in the issue/mybranch down from GitHub. In order to succeed my build needs access to the master branch (specifically it needs the tags on the master branch). Without that branch in the local repository the build will fail.

I'm currently trying to write a script that will fetch the master branch from GitHub and make it available for my build but I'm running into a lot of problems, especially if I tell TeamCity to clone the repository to the build agents local directory (build agent property teamcity.git.use.local.mirrors set to true). In that case the origin (the local clone on the build agent) doesn't actually have the master branch so I can't fetch it from anywhere. Hence my question whether it was possible to tell TeamCity to fetch all the branches.

Thanks

Patrick

0
Comment actions Permalink

So, as far as I understand, your build needs access to the feature branch and master of the same repository. I'm not sure, though, this is a correct way to build.

Anyway, you need to add another copy of the same VCS instance to the build that points to master only (no branch spec). You can use checkout rules to put your branch and master into different directories.

0
Comment actions Permalink

You are correct in your assumption. There are two tools that run during my build process that need access to the master branch (these are https://github.com/JakeGinnivan/GitHubFlowVersion and https://github.com/JakeGinnivan/GitReleaseNotes, for getting the current semantic version and the release notes respectively).

While it may not be a standard workflow to have when using the standard GitHub approach (develop on branch, pull request, merge) in combination with semantic versioning it seems quite a logical approach.

For the time being I've solved the problem by running an additional MsBuild script that fetches the contents of the master branch if there is a need for it. That script looks like:

    <?xml version="1.0" encoding="utf-8"?>
    <Project ToolsVersion="4.0"
             DefaultTargets="Run"
             xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
        <PropertyGroup>
            <DirWorkspace>$(MSBuildProjectDirectory)</DirWorkspace>
            <DirRepository Condition=" '$(DirRepository)' == '' ">$(DirWorkspace)</DirRepository>
            <DirGit Condition=" '$(DirGit)' == '' ">c:\Program Files (x86)\Git\bin</DirGit>      
        </PropertyGroup>
      
        <Import Project="$(DirWorkspace)\GitHasMasterBranch.msbuild"
                Condition="Exists('$(DirWorkspace)\GitHasMasterBranch.msbuild')"/>  
        <Import Project="$(DirWorkspace)\GitGetMasterBranch.msbuild"
                Condition="Exists('$(DirWorkspace)\GitGetMasterBranch.msbuild')"/>  
      
        <Target Name="Run" DependsOnTargets="_DisplayInfo;_FetchOriginMasterIfNotExists">
            <!-- Do nothing here -->
        </Target>
      
        <!-- Display info -->
        <Target Name="_DisplayInfo">
            <Message Text="Preparing workspace ..." />
        </Target>
      
        <PropertyGroup>
            <ExeGit>$(DirGit)\git.exe</ExeGit>
        </PropertyGroup>
        <Target Name="_FetchOriginMasterIfNotExists" DependsOnTargets="_DisplayInfo">
            <GitHasMasterBranch LocalPath="$(DirRepository)">
                <Output TaskParameter="HasMaster" PropertyName="HasMaster" />
            </GitHasMasterBranch>
      
            <Message Text="Not fetching master branch because it already exists" Condition="($(HasMaster))" />
            <Message Text="Fetching master branch because it does not exist" Condition="(!$(HasMaster))" />
            <GitGetMasterBranch LocalPath="$(DirRepository)" Condition="(!$(HasMaster))"/>
        </Target>
     </Project>


Where the GitHasMasterBranch MsBuild script looks like:

    <Project xmlns='http://schemas.microsoft.com/developer/msbuild/2003'
             ToolsVersion="4.0">
        <UsingTask TaskName="GitHasMasterBranch"
                   TaskFactory="CodeTaskFactory"
                   AssemblyFile="$(MSBuildToolsPath)\Microsoft.Build.Tasks.v4.0.dll">
            <ParameterGroup>
                <LocalPath ParameterType="System.String" Required="true" />
                <HasMaster ParameterType="System.Boolean" Output="true" />
            </ParameterGroup>
            <Task>
                <Code Type="Method" Language="cs">
                    <![CDATA[
                        public override bool Execute()
                        {
                            var info = new System.Diagnostics.ProcessStartInfo
                                    {
                                        FileName = "git",
                                        Arguments = "branch",
                                        WorkingDirectory = LocalPath,
                                        UseShellExecute = false,
                                        RedirectStandardOutput = true,
                                        RedirectStandardError = true,
                                    };
                           
                            var text = new System.Text.StringBuilder();
                            var process = new System.Diagnostics.Process();
                            process.StartInfo = info;
                            process.OutputDataReceived +=
                                (s, e) =>
                                {
                                    text.Append(e.Data);
                                };
                            process.ErrorDataReceived +=
                                (s, e) =>
                                {
                                    if (!string.IsNullOrWhiteSpace(e.Data))
                                    {
                                        Log.LogError(e.Data);
                                    }
                                };
                            process.Start();

                            process.BeginOutputReadLine();
                            process.BeginErrorReadLine();
                            process.WaitForExit();
                           
                            HasMaster = text.ToString().Contains("* master");
                           
                            // Log.HasLoggedErrors is true if the task logged any errors -- even if they were logged
                            // from a task's constructor or property setter. As long as this task is written to always log an error
                            // when it fails, we can reliably return HasLoggedErrors.
                            return !Log.HasLoggedErrors;
                        }
                    ]]>
                </Code>
            </Task>
        </UsingTask>
    </Project>


And the GitGetMasterBranch MsBuild script looks like:


    <Project xmlns='http://schemas.microsoft.com/developer/msbuild/2003'

             ToolsVersion="4.0">
        <UsingTask TaskName="GitGetMasterBranch"
                   TaskFactory="CodeTaskFactory"
                   AssemblyFile="$(MSBuildToolsPath)\Microsoft.Build.Tasks.v4.0.dll">
            <ParameterGroup>
                <LocalPath ParameterType="System.String" Required="true" />
            </ParameterGroup>
            <Task>
                <Code Type="Method" Language="cs">
                    <![CDATA[
                        public override bool Execute()
                        {
                            // Get the name of the current branch
                            var info = new System.Diagnostics.ProcessStartInfo
                                    {
                                        FileName = "git",
                                        Arguments = "symbolic-ref --short -q HEAD",
                                        WorkingDirectory = LocalPath,
                                        UseShellExecute = false,
                                        RedirectStandardOutput = true,
                                        RedirectStandardError = true,
                                    };
                           
                            var text = new System.Text.StringBuilder();
                            var process = new System.Diagnostics.Process();
                            process.StartInfo = info;
                            process.OutputDataReceived +=
                                (s, e) =>
                                {
                                    text.Append(e.Data);
                                };
                            process.Start();


                            process.BeginOutputReadLine();
                            process.BeginErrorReadLine();
                            process.WaitForExit();
                           
                            var currentBranch = text.ToString().Trim();
                           
                            // git fetch
                            info = new System.Diagnostics.ProcessStartInfo
                                    {
                                        FileName = "git",
                                        Arguments = "fetch origin",
                                        WorkingDirectory = LocalPath,
                                        UseShellExecute = false,
                                        RedirectStandardOutput = true,
                                        RedirectStandardError = true,
                                    };
                           
                            process = new System.Diagnostics.Process();
                            process.StartInfo = info;
                            process.OutputDataReceived +=
                                (s, e) =>
                                {
                                    if (!string.IsNullOrWhiteSpace(e.Data))
                                    {
                                        Log.LogMessage(MessageImportance.High, e.Data);
                                    }
                                };
                            process.Start();
                           
                            process.BeginOutputReadLine();
                            process.BeginErrorReadLine();
                            process.WaitForExit();
                           
                            // git checkout master
                            info = new System.Diagnostics.ProcessStartInfo
                                    {
                                        FileName = "git",
                                        Arguments = "checkout master",
                                        WorkingDirectory = LocalPath,
                                        UseShellExecute = false,
                                        RedirectStandardOutput = true,
                                        RedirectStandardError = true,
                                    };
                           
                            process = new System.Diagnostics.Process();
                            process.StartInfo = info;
                            process.OutputDataReceived +=
                                (s, e) =>
                                {
                                    if (!string.IsNullOrWhiteSpace(e.Data))
                                    {
                                        Log.LogMessage(MessageImportance.High, e.Data);
                                    }
                                };
                            process.Start();
                           
                            process.BeginOutputReadLine();
                            process.BeginErrorReadLine();
                            process.WaitForExit();
                           
                            // git pull
                            info = new System.Diagnostics.ProcessStartInfo
                                    {
                                        FileName = "git",
                                        Arguments = "pull",
                                        WorkingDirectory = LocalPath,
                                        UseShellExecute = false,
                                        RedirectStandardOutput = true,
                                        RedirectStandardError = true,
                                    };
                           
                            process = new System.Diagnostics.Process();
                            process.StartInfo = info;
                            process.OutputDataReceived +=
                                (s, e) =>
                                {
                                    if (!string.IsNullOrWhiteSpace(e.Data))
                                    {
                                        Log.LogMessage(MessageImportance.High, e.Data);
                                    }
                                };
                            process.Start();
                           
                            process.BeginOutputReadLine();
                            process.BeginErrorReadLine();
                            process.WaitForExit();
                           
                            // git checkout <CURRENT_BRANCH>
                            info = new System.Diagnostics.ProcessStartInfo
                                    {
                                        FileName = "git",
                                        Arguments = string.Format("checkout {0}", currentBranch),
                                        WorkingDirectory = LocalPath,
                                        UseShellExecute = false,
                                        RedirectStandardOutput = true,
                                        RedirectStandardError = true,
                                    };
                           
                            process = new System.Diagnostics.Process();
                            process.StartInfo = info;
                            process.OutputDataReceived +=
                                (s, e) =>
                                {
                                    if (!string.IsNullOrWhiteSpace(e.Data))
                                    {
                                        Log.LogMessage(MessageImportance.High, e.Data);
                                    }
                                };
                            process.Start();
                           
                            process.BeginOutputReadLine();
                            process.BeginErrorReadLine();
                            process.WaitForExit();
                           
                            // Log.HasLoggedErrors is true if the task logged any errors -- even if they were logged
                            // from a task's constructor or property setter. As long as this task is written to always log an error
                            // when it fails, we can reliably return HasLoggedErrors.
                            return !Log.HasLoggedErrors;
                        }
                    ]]>
                </Code>
            </Task>
        </UsingTask>
    </Project>


Just in case anybody ever needs to do something like this :)

0
Comment actions Permalink

I see this is old, but it's still an issue; we used to do this in Jenkins to generate changelogs baked into each build so we could see differences mainly from production.

> So, as far as I understand, your build needs access to the feature branch and master of the same repository. I'm not sure, though, this is a correct way to build.

It's correct for us, since we already have the tools, but will find some other way to do it.

0
Comment actions Permalink

Hi Kristoffer,

There is no support in TeamCity to checkout several branches in one build. The only available workaround at the moment is to perform checkout in the first build step.

0
Comment actions Permalink

I also have a problem with this using the JGitFlow Maven plugin when doing a release-finish.

But in my case it looks like the branches are available but not up to date, and the plugin does exactly this validation in the current release branch and on the master branch.

The plugin closes the current release branch, merges it into the master branch and then into the develop branch.

1
Comment actions Permalink

The build log shows an impressive sequence of git commands that only fetch the triggered branch. And that makes sense for what the agent requires, but our git flow implementation requires the fetching of additional branches.

Our ant scripts support git flow, but unfortunately not all the required branches are fetched such as master. when, for example, we're releasing something on the develop branch. But unless TC is officially supporting git flow,  I guess it doesn't make sense to fetch addtional branches the TC agent doesn't require. But what about offering a clone option instead of a fetch?  Or add support for the required additional branches that an extension or build script requires?  

For the time being, I'm going to modify our ant scripts to fetch the required git flow branches, but it's prefereable to utilize TC's intilligent fetch paradigm that could also leverage the mirror repo.

0

Please sign in to leave a comment.