SSH Agent troubles

TeamCity On-Premises 2023.11.3. 

At the end of our builds, we like to push changes to our git repo hosted on BitBucket cloud. Recently, app passwords on one of our build agents stopped working, so we wanted to make the change to SSH. We made a key pair and uploaded the private key to TeamCity, as well as registering the public key with BitBucket. Then we modified the fetch URL so that the repo is cloned using SSH. We also added a ~/.ssh/config file to the build agent that disables StrictHostKeyChecking. That all worked well.

The problem is that the git operations at the end of the build still did not work. We added an “SSH Agent” build feature with the same key as the one used by the VCS root. We have done the git commands in command line or Powershell. The most recent version we were testing is in command line:

echo on
ssh-add -L
ssh -v
git status
git fetch

The logs are at the bottom of this post. ssh-add -L shows the right key. ssh -v  gets authenticated. And then the git fetch still doesn't work. 

We ended up just putting the private key on the build Agent for now, which worked. But we would prefer to use the SSH Agent feature.

>ssh-add -L
ssh-ed25519 {key} {email_address}

>ssh -v
OpenSSH_for_Windows_8.1p1, LibreSSL 3.0.2
debug1: Reading configuration data C:\\Users\\{user}/.ssh/config
debug1: C:\\Users\\{user}/.ssh/config line 1: Applying options for
Pseudo-terminal will not be allocated because stdin is not a terminal.
debug1: Connecting to {ip and port}.
debug1: Connection established.
debug1: identity file C:\\Users\\{user}/.ssh/id_rsa type -1
debug1: identity file C:\\Users\\{user}/.ssh/id_rsa-cert type -1
debug1: identity file C:\\Users\\{user}/.ssh/id_dsa type -1
debug1: identity file C:\\Users\\{user}/.ssh/id_dsa-cert type -1
debug1: identity file C:\\Users\\{user}/.ssh/id_ecdsa type -1
debug1: identity file C:\\Users\\{user}/.ssh/id_ecdsa-cert type -1
debug1: identity file C:\\Users\\{user}/.ssh/id_ed25519 type -1
debug1: identity file C:\\Users\\{user}/.ssh/id_ed25519-cert type -1
debug1: identity file C:\\Users\\{user}/.ssh/id_xmss type -1
debug1: identity file C:\\Users\\{user}/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_for_Windows_8.1
debug1: Remote protocol version 2.0, remote software version conker_1443369b08 {hash}
debug1: no match: conker_1443369b08 {hash}
debug1: Authenticating to as 'git'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm:
debug1: kex: host key algorithm: ssh-ed25519
debug1: kex: server->client cipher: MAC: <implicit> compression: none
debug1: kex: client->server cipher: MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ssh-ed25519 SHA256:{key}
debug1: Host '' is known and matches the ED25519 host key.
debug1: Found key in C:\\Users\\{user}/.ssh/known_hosts:1
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: {email_address} ED25519 SHA256:{key} agent
debug1: Will attempt key: C:\\Users\\{user}/.ssh/id_rsa
debug1: Will attempt key: C:\\Users\\{user}/.ssh/id_dsa
debug1: Will attempt key: C:\\Users\\{user}/.ssh/id_ecdsa
debug1: Will attempt key: C:\\Users\\{user}/.ssh/id_ed25519
debug1: Will attempt key: C:\\Users\\{user}/.ssh/id_xmss
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ecdsa-sha2-nistp256,ecdsa-sha2-nistp521,,ecdsa-sha2-nistp384,,,,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering public key: {email_address} ED25519 SHA256:{key} agent
debug1: Server accepts key: {email_address} ED25519 SHA256:{key} agent
debug1: Authentication succeeded (publickey).
Authenticated to {ip and port}.
debug1: channel 0: new [client-session]
debug1: Entering interactive session.
debug1: pledge: network
debug1: client_input_global_request: rtype want_reply 0
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
authenticated via ssh key.
debug1: channel 0: free: client-session, nchannels 1

Transferred: sent 2012, received 2144 bytes, in 0.1 seconds
You can use git to connect to Bitbucket. Shell access is disabled
Bytes per second: sent 28658.9, received 30539.1
debug1: Exit status 0

>git status
On branch teamcity-ssh
Your branch is up to date with 'origin/teamcity-ssh'.

You are in a sparse checkout with 15% of tracked files present.

Untracked files:
  (use "git add <file>..." to include in what will be committed)

nothing added to commit but untracked files present (use "git add" to track)

>git fetch Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
Process exited with code 128
Hi Jordan,

I am currently in the process of verifying how the git fetch command should work when explicitly called from the PowerShell script with the team.

The git fetch command downloads commits, files, and refs from a remote repository into the local repo, and if the build configuration uses automatic checkout, the agent executes this command automatically at the "Updating sources: auto checkout (on agent)" stage when needed (you can see it in the build logs), so the additional call of git fetch in the script seems redundant.

You have mentioned that you would like to push changes to your remote git repo at the end of the build. Could you provide more details on why you need to call git fetch in this scenario? In the meantime, I will continue looking for possible solutions for this issue.

Best regards,

Hi Anton,

At the beginning of each build, we update the AssemblyFileVersion in our solution. At the end of the build, we commit the version update, tag the commit, pull, and push. The reason we pull is so that if there are new commits since the build started, we can still push.

I used fetch in this testing because it seemed like a lighter-weight way to reproduce the issue.

I just tried changing the build step in question to the below, but the result was the same: Permission denied.

echo on
ssh-add -L
ssh -v
git commit --allow-empty -m "empty commit"
git push
Hi Jordan,

I was able to reproduce the issue and found a solution that worked on my side. The issue may be in the following: the ssh command uses a built-in Windows ssh client, but git uses C:\Program Files\Git\usr\bin\ssh.exe. 

To fix it:
1. Open the .gitconfig file for the user that runs a Build Agent Windows Service (if the service runs under the Local System account, the file is located in C:\Windows\System32\config\systemprofile; for other users it is C:\Users\). If there is no .gitconfig file, create it.
2. Add the following to the file or replace this setting if there is something there already:
    sshCommand = C:/Windows/System32/OpenSSH/ssh.exe
3. Try to run the build configuration, where the git fetch is called.

As an alternative, you can change the PATH environment variable on a build agent, for the ssh command to use git built-in ssh client (C:\Program Files\Git\usr\bin\ssh.exe).

The idea is to make both git and ssh to use the same ssh client.

Please let me know the results.

Best regards,
Dear Jordan,

Just to add to my previous reply, I tested git push as well with the proposed solution.

Best regards,

Thanks Anton! Changing the PATH worked for me. The confusing thing is that I had tried that before and it hadn't worked. Perhaps I needed to restart the OpenSSH service after changing the path. Anyway, appreciate the help :D

Dear Jordan,

I am glad to hear it worked! Thank you for using TeamCity, and feel free to post any questions that may arise :).

Best regards,

Please sign in to leave a comment.