Selective Git Authentication Errors

Hey there!

We're experiencing some trouble with authentication errors to our VCS server, which appears to be only effecting certain agents. Normally I'd imagine what we have in the error messages below are relatively normal when the authentication for the VCS roots are truly configured incorrectly, however these errors appear to be happening on only a select few agents. A task that fails with these authentication errors on one agent will be picked up and processed correctly on most other agents; these agents are all part of the same of the same cloud profile and that should be configured identically. We're working with Github to see if this could be anything on their end, but we were curious if there was anything that we could do to ensure the agents/git plugin is functioning correctly.

Error messages:
teamcity-agent.log

[2017-11-27 18:34:55,023] INFO - pl.patch.ProjectSourcesOnAgent - Updating sources for root id=7151; name=GitHub Pull Requests; revision: 24de7fe72a96c0495671164b9700a62e8d27badb
[2017-11-27 18:34:55,059] INFO - vcs.git.agent.PluginConfigImpl - Use the mirrors option specified in the VCS root
[2017-11-27 18:34:55,064] INFO - cs.git.agent.UpdaterWithMirror - Update local mirror of root GitHub Pull Requests at <agent_root>/system/git/git-468126D7.git
[2017-11-27 18:34:55,136] INFO - cs.git.agent.UpdaterWithMirror - Init local mirror of root GitHub Pull Requests at <agent_root>/system/git/git-468126D7.git
[2017-11-27 18:34:55,298] INFO - jetbrains.buildServer.AGENT - Updating agent parameters on the server: AgentDetails{Name='EC2-i-<ec2_id>', AgentId=124517, BuildId=657900, AgentOwnAddress='null', AlternativeAddresses=[<agent_ip>, 172.17.0.1], Port=<agent_port>, Version='42538', PluginsVersion='42538-md5-e26e16160c4e13a3a0e8b2871504734c', AvailableRunners=[Ant, cargo-deploy-runner, Duplicator, ftp-deploy-runner, FTPUpload, gradle-runner, Inspection, Ipr, JPS, Maven2, rake-runner, SBT, simpleRunner, smb-deploy-runner, ssh-deploy-runner, ssh-exec-runner], AvailableVcs=[tfs, cvs, jetbrains.git, mercurial, svn, perforce], AuthorizationToken='<auth_token>', PingCode='a3FUQIwPoGYgJO6nCRUctDWF469EeiTe'}
[2017-11-27 18:34:56,400] WARN - l.patch.AbstractSourcesUpdater - Error while checkout on agent: '/usr/bin/git -c core.askpass=<agent_root>/temp/buildTmp/pass1082133811176248193 fetch --progress origin +refs/pull/6326/head:refs/pull/6326/head' command failed.
exit code: 128
stderr: remote: Invalid username or password.
fatal: Authentication failed for 'https://<user>@<github_enterprise_vcs_server>/<repo>/<repo>.git/'
jetbrains.buildServer.vcs.VcsException: '/usr/bin/git -c core.askpass=<agent_root>/temp/buildTmp/pass1082133811176248193 fetch --progress origin +refs/pull/6326/head:refs/pull/6326/head' command failed.
exit code: 128
stderr: remote: Invalid username or password.
fatal: Authentication failed for 'https://<user>@<github_enterprise_vcs_server>/<repo>/<repo>.git/'
at jetbrains.buildServer.buildTriggers.vcs.git.agent.command.impl.CommandUtil.commandFailed(CommandUtil.java:71)
at jetbrains.buildServer.buildTriggers.vcs.git.agent.command.impl.CommandUtil.checkCommandFailed(CommandUtil.java:38)
at jetbrains.buildServer.buildTriggers.vcs.git.agent.command.impl.CommandUtil.runCommand(CommandUtil.java:117)
at jetbrains.buildServer.buildTriggers.vcs.git.agent.GitCommandLine.run(GitCommandLine.java:113)
at jetbrains.buildServer.buildTriggers.vcs.git.agent.command.impl.FetchCommandImpl.call(FetchCommandImpl.java:101)
at jetbrains.buildServer.buildTriggers.vcs.git.agent.UpdaterImpl.fetch(UpdaterImpl.java:614)
at jetbrains.buildServer.buildTriggers.vcs.git.agent.UpdaterWithMirror.fetchMirror(UpdaterWithMirror.java:166)
at jetbrains.buildServer.buildTriggers.vcs.git.agent.UpdaterWithMirror.updateLocalMirror(UpdaterWithMirror.java:116)
at jetbrains.buildServer.buildTriggers.vcs.git.agent.UpdaterWithMirror.updateLocalMirror(UpdaterWithMirror.java:69)
at jetbrains.buildServer.buildTriggers.vcs.git.agent.UpdaterWithMirror.doUpdate(UpdaterWithMirror.java:64)
at jetbrains.buildServer.buildTriggers.vcs.git.agent.UpdaterImpl.update(UpdaterImpl.java:122)
at jetbrains.buildServer.buildTriggers.vcs.git.agent.GitAgentVcsSupport.updateSources(GitAgentVcsSupport.java:104)
at jetbrains.buildServer.agent.impl.vcs.AgentVcsManagerExImpl$CheckoutSupportImpl.updateSources(AgentVcsManagerExImpl.java:108)
at jetbrains.buildServer.agent.impl.patch.ProjectSourcesOnAgent$1.run(ProjectSourcesOnAgent.java:186)
at java.lang.Thread.run(Thread.java:748)

output.log
[2017-11-27 18:34:53,293] INFO - buildServer.AGENT.registration - If this is the first time this agent registered on the server make sure it is authorized by administrator in the server web UI.
[2017-11-27 18:34:55,192] WARN - jetbrains.buildServer.VCS - '/usr/bin/git show-ref refs/pull/6326/head' command failed.
exit code: 1
[2017-11-27 18:34:55,843] WARN - jetbrains.buildServer.VCS - '/usr/bin/git -c core.askpass=<agent_root>/temp/buildTmp/pass3043145072725885474 fetch --progress origin +refs/pull/6326/head:refs/pull/6326/head' command failed.
exit code: 128
stderr: remote: Invalid username or password.
fatal: Authentication failed for 'https://<user>@<github_enterprise_vcs_server>/<repo>/<repo>.git/'
[2017-11-27 18:34:56,398] WARN - jetbrains.buildServer.VCS - '/usr/bin/git -c core.askpass=<agent_root>/temp/buildTmp/pass1082133811176248193 fetch --progress origin +refs/pull/6326/head:refs/pull/6326/head' command failed.
exit code: 128
stderr: remote: Invalid username or password.
fatal: Authentication failed for 'https://<user>@<github_enterprise_vcs_server>/<repo>/<repo>.git/'
[2017-11-27 18:34:56,400] WARN - l.patch.AbstractSourcesUpdater - Error while checkout on agent: '/usr/bin/git -c core.askpass=<agent_root>/temp/buildTmp/pass1082133811176248193 fetch --progress origin +refs/pull/6326/head:refs/pull/6326/head' command failed.
exit code: 128
stderr: remote: Invalid username or password.
fatal: Authentication failed for 'https://<user>@<github_enterprise_vcs_server>/<repo>/<repo>.git/'
jetbrains.buildServer.vcs.VcsException: '/usr/bin/git -c core.askpass=<agent_root>/temp/buildTmp/pass1082133811176248193 fetch --progress origin +refs/pull/6326/head:refs/pull/6326/head' command failed.
exit code: 128
stderr: remote: Invalid username or password.
fatal: Authentication failed for 'https://<user>@<github_enterprise_vcs_server>/<repo>/<repo>.git/'
at jetbrains.buildServer.buildTriggers.vcs.git.agent.command.impl.CommandUtil.commandFailed(CommandUtil.java:71)
at jetbrains.buildServer.buildTriggers.vcs.git.agent.command.impl.CommandUtil.checkCommandFailed(CommandUtil.java:38)
at jetbrains.buildServer.buildTriggers.vcs.git.agent.command.impl.CommandUtil.runCommand(CommandUtil.java:117)
at jetbrains.buildServer.buildTriggers.vcs.git.agent.GitCommandLine.run(GitCommandLine.java:113)
at jetbrains.buildServer.buildTriggers.vcs.git.agent.command.impl.FetchCommandImpl.call(FetchCommandImpl.java:101)
at jetbrains.buildServer.buildTriggers.vcs.git.agent.UpdaterImpl.fetch(UpdaterImpl.java:614)
at jetbrains.buildServer.buildTriggers.vcs.git.agent.UpdaterWithMirror.fetchMirror(UpdaterWithMirror.java:166)
at jetbrains.buildServer.buildTriggers.vcs.git.agent.UpdaterWithMirror.updateLocalMirror(UpdaterWithMirror.java:116)
at jetbrains.buildServer.buildTriggers.vcs.git.agent.UpdaterWithMirror.updateLocalMirror(UpdaterWithMirror.java:69)
at jetbrains.buildServer.buildTriggers.vcs.git.agent.UpdaterWithMirror.doUpdate(UpdaterWithMirror.java:64)
at jetbrains.buildServer.buildTriggers.vcs.git.agent.UpdaterImpl.update(UpdaterImpl.java:122)
at jetbrains.buildServer.buildTriggers.vcs.git.agent.GitAgentVcsSupport.updateSources(GitAgentVcsSupport.java:104)
at jetbrains.buildServer.agent.impl.vcs.AgentVcsManagerExImpl$CheckoutSupportImpl.updateSources(AgentVcsManagerExImpl.java:108)
at jetbrains.buildServer.agent.impl.patch.ProjectSourcesOnAgent$1.run(ProjectSourcesOnAgent.java:186)
at java.lang.Thread.run(Thread.java:748)