upgraded teamcity and can no longer do agent-side checkouts

Hi,

The error seems to indicate some kind of issue with the SSH key algorithm. What version of Java is your build agent running on and is it 32-bit or 64-bit? What version of TeamCity did you upgrade from?

upgraded from  2021.1 to latest

64-bit

interestingly the TC server seems ok.  it can poll checkins and trigger builds.  its the agents that are barfed.

Build agents according to the logs are using a jre in the agent directory, and running Java 1.8.0_31

When the agents are automatically upgraded, the Java version is not affected. Your Java does look to be somewhat old, would you be able to replace the Java in the agent jre directory on one of the build agents to see if it helps? We are currently recommending Amazon Corretto JDK, either 8 or 11, which is available at https://aws.amazon.com/corretto/.

For details on upgrading Java on the build agents, please refer to https://www.jetbrains.com/help/teamcity/configure-java-for-agent.html#Upgrading+Java+on+Agents.

The TeamCity Server uses jgit for the connection to Git servers, whereas the build agent uses the version of Git installed on the build agent (see https://www.jetbrains.com/help/teamcity/git.html#agentGitPath). This could account for the difference you're seeing between the two.

Aside from upgrading Java, it could be that the version of ssh-server (SSH-2.0-OpenSSH_5.3) on your old Git server only supports SHA1. I think SHA2 was introduced in OpenSSH 5.8, but I'm not positive. Since the SHA1 protocol is no longer considered secure, it is not supported by TeamCity by default. You could upgrade the version of ssh-server on your Git server or you could try a patch created to enable SHA1 on TeamCity, which can be downloaded from https://youtrack.jetbrains.com/issue/TW-75102#focus=Comments-27-5813243.0-0. After the patch is installed, you can add an Internal Property to your TeamCity server to allow for the use of SHA1 on the specific domain your repository is located as described in https://youtrack.jetbrains.com/issue/TW-75102#focus=Comments-27-5852251.0-0.

 teamcity.git.ssh.domainsWithEnforcedSha1Signature=.azure.com,.visualstudio.com,<hostname of your Git server>.

I have attempted the following one at a time.  none have worked

1. i have installed the plugin and configured the internal property.

2. i migrated all my repo to a new recently patched GiT server

3. i installed a new jre and pointed the Agents at this.

4. i can ssh from the agent to the GiT server without issue

5. if i add the same ssh key on the build agent, I can issue a GiT Fetch without any problem, from the build agent to git.

since doing all this im getting a slightly different error during agent side checkout though

15:45:33 Updating sources: agent side checkout

15:45:33   Full checkout enforced. Reason: ["Delete all files before the build" turned on]

15:45:33   VCS Root: 7 Systest Branch

15:45:33     checkout rules: =>dest; revision: fd331fcb93eb23f3a67333a383835174deb74f96

15:45:33     Mirrors enabled via VCS root settings

15:45:33     Git version: 2.30.0.0

15:45:33     Update git mirror (F:\TeamCityAgent\system\git\git-0B3F5B5F.git)

15:45:39     Update checkout directory (F:\TeamCityAgent\work\d8195e1d65c3ada\dest)

15:45:39       The .git directory is missing in 'F:\TeamCityAgent\work\d8195e1d65c3ada\dest'. Running 'git init'...

15:45:39       "C:\Program Files\Git\bin\git.exe" init --initial-branch=main

15:45:39       "C:\Program Files\Git\bin\git.exe" config lfs.storage F:\TeamCityAgent\system\git\git-0B3F5B5F.git\lfs

15:45:39       "C:\Program Files\Git\bin\git.exe" config core.sparseCheckout true

15:45:39       "C:\Program Files\Git\bin\git.exe" config http.sslCAInfo

15:45:39       "C:\Program Files\Git\bin\git.exe" show-ref

15:45:39       "C:\Program Files\Git\bin\git.exe" -c credential.helper= ls-remote origin

15:45:39       [15:45:39.648] INFO SSH command to run: git-upload-pack ' /mirror.git'

15:45:39       cipher aes256-ctr is not available

15:45:39       fatal: Could not read from remote repository.

15:45:39      

15:45:39       Please make sure you have the correct access rights

15:45:39       and the repository exists.

15:45:39       Failed to list remote repository refs, outdated local refs will not be cleaned

15:45:39       "C:\Program Files\Git\bin\git.exe" show-ref refs/remotes/origin/v7-systest

15:45:40       "C:\Program Files\Git\bin\git.exe" log -n1 --pretty=format:%H%x20%s fd331fcb93eb23f3a67333a383835174deb74f96 --

15:45:40       No 'git fetch' required: commit 'fd331fcb93eb23f3a67333a383835174deb74f96' is in the local repository clone pointed by 'refs/remotes/origin/v7-systest'.

15:45:40       "C:\Program Files\Git\bin\git.exe" branch

15:45:40       "C:\Program Files\Git\bin\git.exe" update-ref refs/heads/v7-systest fd331fcb93eb23f3a67333a383835174deb74f96

15:45:40       "C:\Program Files\Git\bin\git.exe" -c credential.helper= checkout -q -f v7-systest

15:45:49       "C:\Program Files\Git\bin\git.exe" branch --set-upstream-to=refs/remotes/origin/v7-systest

15:45:49       Cleaning 7 Systest Branch in F:\TeamCityAgent\work\d8195e1d65c3ada\dest the file set ALL_UNTRACKED

15:45:49       "C:\Program Files\Git\bin\git.exe" clean -f -d -x

15:45:49   VCS Root: 7 Branch

15:45:49     checkout rules: =>src; revision: 0e01b895a6d099c6f69edcfacd75687290298a71

15:45:49     Mirrors enabled via VCS root settings

15:45:49     Git version: 2.30.0.0

15:45:49     Update git mirror (F:\TeamCityAgent\system\git\git-0B3F5B5F.git)

15:45:49       "C:\Program Files\Git\bin\git.exe" config http.sslCAInfo

15:45:49       "C:\Program Files\Git\bin\git.exe" show-ref

15:45:49       "C:\Program Files\Git\bin\git.exe" -c credential.helper= ls-remote origin

15:45:50       [15:45:49.856] INFO SSH command to run: git-upload-pack '/mirror.git'

15:45:50       cipher aes256-ctr is not available

15:45:50       fatal: Could not read from remote repository.

15:45:50      

15:45:50       Please make sure you have the correct access rights

15:45:50       and the repository exists.

15:45:50       Failed to list remote repository refs, outdated local refs will not be cleaned

15:45:50       "C:\Program Files\Git\bin\git.exe" show-ref refs/heads/v7

15:45:50       show-ref command failed, empty result will be returned: "C:\Program Files\Git\bin\git.exe" show-ref refs/heads/v7 command failed.

      exit code: 1

15:45:50       'git fetch' required: 'refs/heads/v7' is not found in the local repository clone.

15:45:50       "C:\Program Files\Git\bin\git.exe" -c credential.helper= fetch --progress --recurse-submodules=no origin +refs/heads/v7:refs/heads/v7

15:45:50         [15:45:50.438] INFO SSH command to run: git-upload-pack '/mirror.git'

15:45:50         cipher aes256-ctr is not available

15:45:50         fatal: Could not read from remote repository.

15:45:50        

15:45:50         Please make sure you have the correct access rights

15:45:50         and the repository exists.

15:45:50   Failed to perform checkout on agent: "C:\Program Files\Git\bin\git.exe" -c credential.helper= fetch --progress --recurse-submodules=no origin +refs/heads/v7:refs/heads/v7 command failed.

  exit code: 128

  stderr: [15:45:50.438] INFO SSH command to run: git-upload-pack '/mirror.git'

  cipher aes256-ctr is not available

  fatal: Could not read from remote repository.

  Please make sure you have the correct access rights

  and the repository exists.

>>>>>>>>>>>>>>>>>>>>>>

ive checked the new GiT server, and it definately supports the cipher aes256-ctr

Have you removed the Internal Property and plugin that you added earlier? I think the error might mean the client-side does not have aes256-ctr. What version of OpenSSH is installed on the build agent?

Also, would you mind sharing the section of the teamcity-agent.log showing the Java being used by the build agent? It should look something like:

INFO - s.buildServer.agent.AgentMain2 - TeamCity Build Agent 2021.2 (build 99542)
INFO - s.buildServer.agent.AgentMain2 - OS: Windows 10, version 10.0, x86, Current user: LAPTOP-123$, Time zone: CST (UTC-06:00)
INFO - s.buildServer.agent.AgentMain2 - Java: 1.8.0_302, OpenJDK Server VM (32 bit) (25.302-b08, mixed mode), OpenJDK Runtime Environment (1.8.0_302-b08), Amazon.com Inc.; JVM parameters: -ea -Xmx512m -XX:+HeapDumpOnOutOfMemoryError -Xrs -Dlog4j.configuration=file:../conf/teamcity-agent-log4j.xml -Dteamcity_logs=../logs/

thank you for your suggestions.

for anyone else that finds this post.  after trying everything else i could think of, and all the suggestions in this thread, i completely removed the build agent software and working directories, and installed fresh build agents on the same servers.

suddenly everything just works again. its using the build in jre that ships with the agent fine.  it even connects fine to my old 2010 GiT server (and the new one i moved everything to).

So if someone else gets the same problems, and only as a last resort - uninstll and fresh install the agents.

Thanks for reporting back with your results!