On-prem TeamCity Server to Bitbucket Server SSH Auth Cancel error.
Hi:
* Team City Version: 2020.2.4 build 86063
* Bitbucket/Stash Version: 3.7.1
* Windows Server 2012 R2 with patches
My on-prem TeamCity Server has recently developed an issue communicating with our on-prem Bitbucket Server.
This issue only happens when the VCSRoot is configured to use SSH. The error being returned is Auth cancel which is being thrown by jetbrains.buildServer.vcs.VcsException.
This all started sometime between 4th April and 12th April 2023. Today I receive an answer from the infrastructure team that error is the SSH Server changed from SSHD-CORE-0.13.0 to OpenSSH_5_3.
While I can't rule out a change to the Team City instance on the Windows server, I suspect the change happened at the Bitbucket instance.
However, I need to get more details of the error to continue to resolve the issue.
Therefore, I would like to enable more detailed logging. From reading all the [documentation](https://www.jetbrains.com/help/teamcity/2020.2/teamcity-server-logs.html#Reading+Logs), I know that is possible, I just don't know:
1. What log level I should set: Debug, Trace, or some custom level. Which will provide the most helpful information.
2. Is there a way to capture the actual SSH Server implementation being used along with the actual request and response data?
3. Where to actually make that change. I have never used Log4j, and I am a bit confused by the TeamCity configuration file set up. Do I change <TeamCity Server home>/conf/teamcity-server-log4j.xml or <TeamCity Data home>/config/_logging/debug.vcs.xml or debug.vcs.xml.dist .
4. Do I only need to change the <category ...> element for jetbrains.buildServer.vcs or are there other categories that need to be changed?
Thank you for your help.
-marc
P.S.: If needed I can provide the full stack dump from several calls between TeamCity and Bitbucket.
Please sign in to leave a comment.
Hello Marc,
In most cases changing the logging configuration via logging preset is enough - this is a UI operation you may perform via Administration | Diagnostics | Troubleshooting | Debug logging. You would probably need the debug-vcs preset.
Where exactly do you see the issue, on the agent side (e.g. only when the builds are started) or on the server side (e.g. the changes from repository are not collected, and Test Connection option in the VCS root settings throws an exception)?
Please mind that finer details of the communication process (like the SSH server in use on the remote side) will not be available in the TeamCity logs. I suggest to set up a traffic capture tool (Wireshark, tcpdump, etc.) on the machine (server or agent, depending on where you are seeing the error) and try to reproduce the issue while the capture is running.
Please let me know if this helps.
@... thank you. The issue was resolved. Our internal Bitbucket/Stash server uses a non-standard port for SSH communications.
It seems that when the TeamCity server was set up and given access to the Bitbucket server, some place in the routing between the two servers, was a port forward rule from the standard SSH port to the non-standard port.
I've looked at the Administration | Diagnostics | Troubleshooting | Debug logging section. The Active logging preset was already configured to be debug-all. Is it safe to assume that setting would also haven been applied to the VCS log?
Is that the most detail available?
Thanks
-marc
Hello Marc,
Thank you for the update, glad to hear the issue was resolved!
debug-all is indeed the most detailed preset if we are talking about the TeamCity-specific logging. It is also possible to add -Djavax.net.debug=all startup parameter - this parameter will make JVM log a lot of details about the network communication into catalina.out log, but due to it being extremely verbose, capturing traffic is usually an easier alternative.
Speaking of that, debug-all is also quite verbose - unless it is needed, I would suggest to keep the default preset on and only switch to a debug preset when it is required.