SSH Agent troubles
TeamCity On-Premises 2023.11.3.
At the end of our builds, we like to push changes to our git repo hosted on BitBucket cloud. Recently, app passwords on one of our build agents stopped working, so we wanted to make the change to SSH. We made a key pair and uploaded the private key to TeamCity, as well as registering the public key with BitBucket. Then we modified the fetch URL so that the repo is cloned using SSH. We also added a ~/.ssh/config file to the build agent that disables StrictHostKeyChecking. That all worked well.
The problem is that the git operations at the end of the build still did not work. We added an “SSH Agent” build feature with the same key as the one used by the VCS root. We have done the git commands in command line or Powershell. The most recent version we were testing is in command line:
echo on
ssh-add -L
ssh -v git@bitbucket.org
git status
git fetchThe logs are at the bottom of this post. ssh-add -L shows the right key. ssh -v git@bitbucket.org gets authenticated. And then the git fetch still doesn't work.
We ended up just putting the private key on the build Agent for now, which worked. But we would prefer to use the SSH Agent feature.
>ssh-add -L
ssh-ed25519 {key} {email_address}
>ssh -v git@bitbucket.org
OpenSSH_for_Windows_8.1p1, LibreSSL 3.0.2
debug1: Reading configuration data C:\\Users\\{user}/.ssh/config
debug1: C:\\Users\\{user}/.ssh/config line 1: Applying options for bitbucket.org
Pseudo-terminal will not be allocated because stdin is not a terminal.
debug1: Connecting to bitbucket.org {ip and port}.
debug1: Connection established.
debug1: identity file C:\\Users\\{user}/.ssh/id_rsa type -1
debug1: identity file C:\\Users\\{user}/.ssh/id_rsa-cert type -1
debug1: identity file C:\\Users\\{user}/.ssh/id_dsa type -1
debug1: identity file C:\\Users\\{user}/.ssh/id_dsa-cert type -1
debug1: identity file C:\\Users\\{user}/.ssh/id_ecdsa type -1
debug1: identity file C:\\Users\\{user}/.ssh/id_ecdsa-cert type -1
debug1: identity file C:\\Users\\{user}/.ssh/id_ed25519 type -1
debug1: identity file C:\\Users\\{user}/.ssh/id_ed25519-cert type -1
debug1: identity file C:\\Users\\{user}/.ssh/id_xmss type -1
debug1: identity file C:\\Users\\{user}/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_for_Windows_8.1
debug1: Remote protocol version 2.0, remote software version conker_1443369b08 {hash}
debug1: no match: conker_1443369b08 {hash}
debug1: Authenticating to bitbucket.org:22 as 'git'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: ssh-ed25519
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ssh-ed25519 SHA256:{key}
debug1: Host 'bitbucket.org' is known and matches the ED25519 host key.
debug1: Found key in C:\\Users\\{user}/.ssh/known_hosts:1
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: {email_address} ED25519 SHA256:{key} agent
debug1: Will attempt key: C:\\Users\\{user}/.ssh/id_rsa
debug1: Will attempt key: C:\\Users\\{user}/.ssh/id_dsa
debug1: Will attempt key: C:\\Users\\{user}/.ssh/id_ecdsa
debug1: Will attempt key: C:\\Users\\{user}/.ssh/id_ed25519
debug1: Will attempt key: C:\\Users\\{user}/.ssh/id_xmss
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ecdsa-sha2-nistp256,ecdsa-sha2-nistp521,ssh-rsa-cert-v01@openssh.com,ecdsa-sha2-nistp384,ssh-dss-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp521-cert-v01@openssh.com>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering public key: {email_address} ED25519 SHA256:{key} agent
debug1: Server accepts key: {email_address} ED25519 SHA256:{key} agent
debug1: Authentication succeeded (publickey).
Authenticated to bitbucket.org {ip and port}.
debug1: channel 0: new [client-session]
debug1: Entering interactive session.
debug1: pledge: network
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
authenticated via ssh key.
debug1: channel 0: free: client-session, nchannels 1
Transferred: sent 2012, received 2144 bytes, in 0.1 seconds
You can use git to connect to Bitbucket. Shell access is disabled
Bytes per second: sent 28658.9, received 30539.1
debug1: Exit status 0
>git status
On branch teamcity-ssh
Your branch is up to date with 'origin/teamcity-ssh'.
You are in a sparse checkout with 15% of tracked files present.
Untracked files:
(use "git add <file>..." to include in what will be committed)
Ci_Cd_Scripts/
nothing added to commit but untracked files present (use "git add" to track)
>git fetch
git@bitbucket.org: Permission denied (publickey).
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
Process exited with code 128Please sign in to leave a comment.
I am currently in the process of verifying how the git fetch command should work when explicitly called from the PowerShell script with the team.
The git fetch command downloads commits, files, and refs from a remote repository into the local repo, and if the build configuration uses automatic checkout, the agent executes this command automatically at the "Updating sources: auto checkout (on agent)" stage when needed (you can see it in the build logs), so the additional call of git fetch in the script seems redundant.
You have mentioned that you would like to push changes to your remote git repo at the end of the build. Could you provide more details on why you need to call git fetch in this scenario? In the meantime, I will continue looking for possible solutions for this issue.
Best regards,
Anton
Hi Anton,
At the beginning of each build, we update the AssemblyFileVersion in our solution. At the end of the build, we commit the version update, tag the commit, pull, and push. The reason we pull is so that if there are new commits since the build started, we can still push.
I used fetch in this testing because it seemed like a lighter-weight way to reproduce the issue.
I just tried changing the build step in question to the below, but the result was the same: Permission denied.
I was able to reproduce the issue and found a solution that worked on my side. The issue may be in the following: the ssh command uses a built-in Windows ssh client, but git uses C:\Program Files\Git\usr\bin\ssh.exe.
To fix it:
1. Open the .gitconfig file for the user that runs a Build Agent Windows Service (if the service runs under the Local System account, the file is located in C:\Windows\System32\config\systemprofile; for other users it is C:\Users\). If there is no .gitconfig file, create it.
2. Add the following to the file or replace this setting if there is something there already:
[core]
sshCommand = C:/Windows/System32/OpenSSH/ssh.exe
3. Try to run the build configuration, where the git fetch is called.
As an alternative, you can change the PATH environment variable on a build agent, for the ssh command to use git built-in ssh client (C:\Program Files\Git\usr\bin\ssh.exe).
The idea is to make both git and ssh to use the same ssh client.
Please let me know the results.
Best regards,
Anton
Just to add to my previous reply, I tested git push as well with the proposed solution.
Best regards,
Anton
Thanks Anton! Changing the PATH worked for me. The confusing thing is that I had tried that before and it hadn't worked. Perhaps I needed to restart the OpenSSH service after changing the path. Anyway, appreciate the help :D
I am glad to hear it worked! Thank you for using TeamCity, and feel free to post any questions that may arise :).
Best regards,
Anton