"Auth Fail" with SSH access to GIT repo on on-premise TFS

We have running TeamCity Professional 2019.2 (build 71499 running on Java8). It connects to our GIT repos on on-premise TFS (Version 16.122.28313.3) via SSH and Uploaded SSH keys.

Recently we tried to upgrade to TC (2021.2.3). After some struggles we made it work with Correto 11. But we were unable to get SSH connection to GIT/TFS (above) run. We got only "Auth Fail" message (stack below). In the end we rolled back. 

Then we installed fresh instance TC (2021.2.3) with packaged Correto 11. Everything seems to work but the SSH connection to GIT/TFS. (In the end we always get the same exception listed below).

The problem is not related to agent/server checkout. We get the exception when pressing "Test connection" button in the VCS definition.

We tried:

* to issue new key for TFS - works with old TC but not with the new TC

* use key with or without password - works with old TC but not with the new TC

* use "Custom private key" with or without password

* use "Custom private key" and specify wrong path to the key - we got different/specific error. So our problem is not related to file acces to key.

* use "Custom private key" and specify wrong password to the key - we got different/specific error. So our problem is not related to wrong pasword to key.

* use different usernames - (none), "git", username, domain\username (we are in Windows environment)

* use ssh from commandline from the TC server with the key to connect to GIT/TFS - connection was successful

Any clues what to do? 

The exception we get with full stack from the teamcity-vcs.log (we didnt get anything better using the VCS debug preset) (the exception is the same with any of the cases above) :

[2022-03-16 10:26:44,328]   INFO [nio-8111-exec-2] -      jetbrains.buildServer.VCS - Error occurred in test connection: jetbrains.buildServer.vcs.VcsException: org.eclipse.jgit.errors.TransportException: ssh://theusername@tfsserveraddress:22/tfs/TFS2008%20Collection/CTSGIT/_git/CTS.BuildTools: Auth fail
    at jetbrains.buildServer.buildTriggers.vcs.git.command.impl.GitRepoOperationsImpl.getRemoteRefsJGit(GitRepoOperationsImpl.java:154)
    at jetbrains.buildServer.buildTriggers.vcs.git.command.impl.GitRepoOperationsImpl.access$100(GitRepoOperationsImpl.java:34)
    at jetbrains.buildServer.buildTriggers.vcs.git.command.impl.GitRepoOperationsImpl$1.lsRemote(GitRepoOperationsImpl.java:106)
    at jetbrains.buildServer.buildTriggers.vcs.git.GitVcsSupport.getRemoteRefs(GitVcsSupport.java:513)
    at jetbrains.buildServer.buildTriggers.vcs.git.GitVcsSupport.getRemoteRefs(GitVcsSupport.java:499)
    at jetbrains.buildServer.buildTriggers.vcs.git.GitVcsSupport.lambda$getCurrentState$0(GitVcsSupport.java:178)
    at jetbrains.buildServer.buildTriggers.vcs.git.RepositoryManagerImpl.runWithDisabledRemove(RepositoryManagerImpl.java:256)
    at jetbrains.buildServer.buildTriggers.vcs.git.GitVcsSupport.getCurrentState(GitVcsSupport.java:174)
    at jetbrains.buildServer.buildTriggers.vcs.git.TestConnectionCommand.checkFetchConnection(TestConnectionCommand.java:103)
    at jetbrains.buildServer.buildTriggers.vcs.git.TestConnectionCommand.lambda$null$0(TestConnectionCommand.java:73)
    at jetbrains.buildServer.serverSide.impl.BaseAccessChecker.runWithDisabledChecks(BaseAccessChecker.java:8)
    at jetbrains.buildServer.serverSide.impl.SecondaryNodeSecurityManager.executeSafe(SecondaryNodeSecurityManager.java:30)
    at jetbrains.buildServer.serverSide.IOGuardInitializer$IOGuardDelegateImpl.allowNetworkAndCommandLine(IOGuardInitializer.java:7)
    at jetbrains.buildServer.serverSide.IOGuard.allowNetworkAndCommandLine(IOGuard.java:130)
    at jetbrains.buildServer.buildTriggers.vcs.git.TestConnectionCommand.lambda$testConnection$1(TestConnectionCommand.java:62)
    at jetbrains.buildServer.buildTriggers.vcs.git.JSchLoggers.evaluateWithLoggingLevel(JSchLoggers.java:43)
    at jetbrains.buildServer.buildTriggers.vcs.git.TestConnectionCommand.testConnection(TestConnectionCommand.java:60)
    at jetbrains.buildServer.buildTriggers.vcs.git.GitVcsSupport.lambda$testConnection$2(GitVcsSupport.java:317)
    at jetbrains.buildServer.buildTriggers.vcs.git.RepositoryManagerImpl.runWithDisabledRemove(RepositoryManagerImpl.java:256)
    at jetbrains.buildServer.buildTriggers.vcs.git.GitVcsSupport.testConnection(GitVcsSupport.java:314)
    at jetbrains.vcs.api.services.impl.TestConnectionServiceProvider$1.testConnection(TestConnectionServiceProvider.java:2)
    at jetbrains.buildServer.controllers.admin.projects.TestConnectionCommand.lambda$runTestConnection$0(TestConnectionCommand.java:27)
    at jetbrains.buildServer.serverSide.impl.BaseAccessChecker.runWithDisabledChecks(BaseAccessChecker.java:8)
    at jetbrains.buildServer.serverSide.impl.SecondaryNodeSecurityManager.executeSafe(SecondaryNodeSecurityManager.java:30)
    at jetbrains.buildServer.serverSide.IOGuardInitializer$IOGuardDelegateImpl.allowNetworkAndCommandLine(IOGuardInitializer.java:7)
    at jetbrains.buildServer.serverSide.IOGuard.allowNetworkAndCommandLine(IOGuard.java:130)
    at jetbrains.buildServer.controllers.admin.projects.TestConnectionCommand.runTestConnection(TestConnectionCommand.java:24)
    at jetbrains.buildServer.controllers.admin.projects.TestConnectionCommand.doTestConnection(TestConnectionCommand.java:28)
    at jetbrains.buildServer.controllers.admin.projects.EditVcsRootsController.doPost(EditVcsRootsController.java:278)
    at jetbrains.buildServer.controllers.BaseFormXmlController$1.handleRequest(BaseFormXmlController.java:55)
    at jetbrains.buildServer.controllers.AjaxRequestProcessor.processRequest(AjaxRequestProcessor.java:48)
    at jetbrains.buildServer.controllers.BaseFormXmlController.doHandle(BaseFormXmlController.java:52)
    at jetbrains.buildServer.controllers.BaseController.handleRequestInternal(BaseController.java:114)
    at org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:174)
    at jetbrains.buildServer.controllers.BaseController.handleRequest(BaseController.java:93)
    at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:50)
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
    at jetbrains.buildServer.maintenance.WebDispatcherServlet.doService(WebDispatcherServlet.java:9)
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
    at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:681)
    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:764)
    at jetbrains.buildServer.maintenance.TeamCityDispatcherServlet.processedByMainServlet(TeamCityDispatcherServlet.java:27)
    at jetbrains.buildServer.maintenance.TeamCityDispatcherServlet.service(TeamCityDispatcherServlet.java:48)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at jetbrains.buildServer.web.jsp.JspPrecompilerFilter.doFilter(JspPrecompilerFilter.java:125)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at jetbrains.buildServer.web.DisableSessionIdFromUrlFilter.doFilter(DisableSessionIdFromUrlFilter.java:4)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at jetbrains.buildServer.web.UserIdProviderFilter.doFilter(UserIdProviderFilter.java:5)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at jetbrains.buildServer.web.BannedIPsFilter.doFilter(BannedIPsFilter.java:3)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at jetbrains.buildServer.web.NodeInfoHeaderFilter.doFilter(NodeInfoHeaderFilter.java:9)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:107)
    at jetbrains.buildServer.diagnostic.web.DiagnosticFilter.doFilter(DiagnosticFilter.java:15)
    at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:112)
    at jetbrains.buildServer.web.DependencyParametersCalculationContextFilter.doFilter(DependencyParametersCalculationContextFilter.java:3)
    at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:112)
    at jetbrains.buildServer.diagnostic.web.HttpRequestsDurationMetricsReporter.doFilter(HttpRequestsDurationMetricsReporter.java:5)
    at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:112)
    at jetbrains.buildServer.web.HttpSecurityHeadersFilter.doFilter(HttpSecurityHeadersFilter.java:3)
    at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:112)
    at jetbrains.buildServer.controllers.filters.DisableSessionCookieTokenAuthFilter.doFilter(DisableSessionCookieTokenAuthFilter.java:10)
    at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:112)
    at jetbrains.buildServer.controllers.filters.ProxyDetailsFilter.doFilter(ProxyDetailsFilter.java:5)
    at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:112)
    at jetbrains.buildServer.controllers.filters.ClearSecurityContextFilter.doFilter(ClearSecurityContextFilter.java:23)
    at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:112)
    at org.springframework.web.filter.CompositeFilter.doFilter(CompositeFilter.java:73)
    at jetbrains.buildServer.web.DelegatingFilter.doFilter(DelegatingFilter.java:74)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at jetbrains.buildServer.web.ResponseFragmentFilter.doFilter(ResponseFragmentFilter.java:42)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:196)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:542)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:364)
    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:624)
    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:831)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1650)
    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
    at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
    at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.eclipse.jgit.errors.TransportException: ssh://CTS-GROUP\tvetrovsky@ctstfs2013.cts-group.local:22/tfs/TFS2008%20Collection/CTSGIT/_git/CTS.BuildTools: Auth fail
    at org.eclipse.jgit.transport.JschConfigSessionFactory.getSession(JschConfigSessionFactory.java:162)
    at org.eclipse.jgit.transport.SshTransport.getSession(SshTransport.java:107)
    at org.eclipse.jgit.transport.TransportGitSsh$SshFetchConnection.<init>(TransportGitSsh.java:254)
    at org.eclipse.jgit.transport.TransportGitSsh.openFetch(TransportGitSsh.java:144)
    at jetbrains.buildServer.buildTriggers.vcs.git.command.impl.GitRepoOperationsImpl$2.call(GitRepoOperationsImpl.java:128)
    at jetbrains.buildServer.buildTriggers.vcs.git.command.impl.GitRepoOperationsImpl$2.call(GitRepoOperationsImpl.java:116)
    at jetbrains.buildServer.buildTriggers.vcs.git.Retry.retry(Retry.java:58)
    at jetbrains.buildServer.buildTriggers.vcs.git.command.impl.GitRepoOperationsImpl.lambda$getRemoteRefsJGit$1(GitRepoOperationsImpl.java:116)
    at jetbrains.buildServer.serverSide.impl.BaseAccessChecker.runWithDisabledChecks(BaseAccessChecker.java:8)
    at jetbrains.buildServer.serverSide.impl.SecondaryNodeSecurityManager.executeSafe(SecondaryNodeSecurityManager.java:30)
    at jetbrains.buildServer.serverSide.IOGuardInitializer$IOGuardDelegateImpl.allowNetworkCall(IOGuardInitializer.java:9)
    at jetbrains.buildServer.serverSide.IOGuard.allowNetworkCall(IOGuard.java:82)
    at jetbrains.buildServer.buildTriggers.vcs.git.command.impl.GitRepoOperationsImpl.getRemoteRefsJGit(GitRepoOperationsImpl.java:114)
    ... 103 more
Caused by: com.jcraft.jsch.JSchException: Auth fail
    at com.jcraft.jsch.Session.connect(Session.java:523)
    at org.eclipse.jgit.transport.JschConfigSessionFactory.getSession(JschConfigSessionFactory.java:115)
    ... 115 more