GitHub PR Failed to retrieve pull request information; ... status code 403: Forbidden message
I'm using TeamCity 2018.2.2 (build 61245). We have the GitHub Pull Request and Commit Status Publisher plugins configured for a bunch of our jobs. We have a fairly long chain of jobs that get this message:
Failed to retrieve pull request information; Exception: java.io.IOException: Pull request information retrieval failure due to HTTP error, status code 403: Forbidden
The Pull Request builds still kick off from our trigger (on a single job that has dependencies to the other chained jobs). I also had to setup the GitHub PR Plugin on all the chained jobs so they could see the PR information (I don't remember seeing that info in documentation).
I haven't been able to figure out what's causing this message to appear. All the chained jobs are configured the same way with the same account. When testing the connection to GitHub it always passes.
The GitHub account this is tied to, has the correct permissions as far as I can see since PR jobs do kick off and run - and TC Reports it's status back to the PR's.
My first guess was that it was being shown when a PR was opened that has the message that it cannot be automatically merged, but I haven't been able to prove this yet.
I would love it if someone had some idea of why we keep seeing this.
Thanks.
Chris
TLDR;
I just cannot figure out what's causing this message:
Failed to retrieve pull request information; Exception: java.io.IOException: Pull request information retrieval failure due to HTTP error, status code 403: Forbidden
Hide stacktrace
at jetbrains.buildServer.pullRequests.impl.github.GitHubPullRequestProvider$AbstractHttpResponseProcessor.processResponse(GitHubPullRequestProvider.java:260)
at jetbrains.buildServer.pullRequests.impl.github.GitHubPullRequestProvider$MultiplePRsResponseProcessor.processResponse(GitHubPullRequestProvider.java:324)
at jetbrains.buildServer.pullRequests.impl.HttpHelper$2.consume(HttpHelper.java:59)
at jetbrains.buildServer.util.HTTPRequestBuilder$ApacheClient43RequestHandler.doRequest(HTTPRequestBuilder.java:69)
at jetbrains.buildServer.pullRequests.impl.HttpHelper.call(HttpHelper.java:77)
at jetbrains.buildServer.pullRequests.impl.HttpHelper.post(HttpHelper.java:93)
at jetbrains.buildServer.pullRequests.impl.github.GitHubPullRequestProvider.lambda$fetchPullRequests$0(GitHubPullRequestProvider.java:146)
at jetbrains.buildServer.serverSide.impl.BaseAccessChecker.runWithDisabledCheck(BaseAccessChecker.java:45)
at jetbrains.buildServer.serverSide.impl.SecondaryNodeSecurityManager.runSafeNetworkOperation(SecondaryNodeSecurityManager.java:35)
at jetbrains.buildServer.pullRequests.impl.github.GitHubPullRequestProvider.fetchPullRequests(GitHubPullRequestProvider.java:145)
at jetbrains.buildServer.pullRequests.impl.PullRequestManagerImpl.retrievePullRequests(PullRequestManagerImpl.java:493)
at jetbrains.buildServer.pullRequests.impl.PullRequestManagerImpl$4.run(PullRequestManagerImpl.java:463)
at jetbrains.buildServer.util.NamedThreadFactory.executeWithNewThreadName(NamedThreadFactory.java:76)
at jetbrains.buildServer.util.ExceptionUtil$1.run(ExceptionUtil.java:41)
at jetbrains.buildServer.serverSide.impl.executors.SimpleExecutorServices$RunnableWrapper.run(SimpleExecutorServices.java:2)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at jetbrains.buildServer.util.executors.ScalingThreadPoolExecutor$RunnableFutureWrapper.run(ScalingThreadPoolExecutor.java:161)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Please sign in to leave a comment.
Hi Chris,
Would you mind to provide a bit more details about your case, especially how many pull requests are you receiving on a daily basis? How many repositories do you access from your TeamCity instance under the same credentials? How frequent are the builds (both in terms of chains and each individual build)?
We have receieved reports before that may indicate that in some cases the Pull Requests plugin breaks GitHub's abuse rate limit without clear indication that this is happening. This is different to regular request rate limit that the plugin deals with correctly in our knowledge. We are looking into this problem on our side. But such situations seemingly emerge when the same credentials are used to access many repositories with relatively many new pull requests appearing on a daily basis in each. Does this describe your case?
Thanks,
Anton
Hi Chris,
thanks a lot for the very detailed explanation. We had a very similar report of this error showing up (correctly) once but then being impossible to get rid of, even removing the PR feature from the build. While we haven't fully figured out yet what produces it, I've reported it to the dev of this feature and he's checking it. It seems like it's the same issue, so will hopefully be fixed at the same time.
Hi Anton,
Jobs tied to GitHub PRs…
5 jobs don’t get PR’s very often maybe once every few weeks.
1 single job – Since the May 10 – this has kicked off 16 PR jobs, and one master branch job.
Long build chain:
10 jobs in the chain that can kick off and are configured with the GitHub PR Plugin, and one of those jobs has the Publish Commit Status plugin.
We do have other jobs in our TC system that poll GitHub as well for their branch builds that do not have the PR plugin tied to their jobs.
Since the May 10 those other jobs have probably run 146 times.
During the day I do see some of our jobs queue up with multiple commits to different PR’s – that will queue up 9+ jobs at a time for each job.
I’m not sure if this is a lot for GitHubs api or not. It kind of makes sense that we might be running up against their throttling limit occasionally during the day. I have had a few developers over the last 2 weeks ask me why their PR hasn’t kicked off. I can see the job knows there are PR changes that it should have kicked off, but the job just didn’t kick off. It’s during those times it could be we have been throttled…
Does this seem like a lot? I’m not sure what their thresholds are.
Thanks.
Chris
Thanks for such a comprehensive response Chris. No this is definitely not a lot, but it is possible that the pull requests plugin makes a number of requests to GitHub GraphQL API concurrently (or almost concurrently) and this may break their abuse rate limit. Please follow this issue on our issue tracker: https://youtrack.jetbrains.com/issue/TW-59723
Thanks for the quick response. I have followed that issue.
Thanks again.
Chris