Gitlab commit status publisher fails when build is failing

Answered

Permanently deleted user

Created February 13, 2017 20:33

Hi,

I'm using teamcity v10.0.3 (build 42434), and I came across a problem with the gitlab commit status publisher.
I have configured the gitlab URL as following: http:///api/v3/ and the private token from my gitlab account.
When I push a commit that compiles - there is no problem. The commit status is published to gitlab and all is well.
But when the build fails, there is an error publishing the status:
"Publish status error in build ...: java.lang.RuntimeException: 500 Internal Server Error"

So I checked the logs on the teamcity server(teamcity-server.log) but they didn't help:
"WARN - tpBasedCommitStatusPublisher$1 - Failed to execute posting commit status. Commit status publishing HTTP request failed
Jetbrains.buildServer.commitPublisher.PublishError: Commit status publishing HTTP request failed
at jetbrains.buildServer.commitPublisher.HttpBasedCommitStatusPublisher$1.run(HttpBasedCommitStatusPublisher.java:71)
...
Caused by: java.lang.RuntimeException: 500 Internal Server Error
at jetbrains.buildServer.commitPublisher.HttpBasedCommitStatusPublisher.processResponse(HttpBasedCommitStatusPublisher.java:140)

Is this a teamcity or gitlab problem?
Thanks in advance

7 comments

Denis Lapuente

Created February 14, 2017 18:26

Hi Hecht,

an error 500 usually means an error from the server. In this case, the server is Gitlab and TeamCity is the client trying to connect to it, so I would suggest checking in gitlab to find out what is going on. In case the server is throwing the error due to a malformed request, we would need to know more details on the build configuration and maybe on the commit, if the issue happens on only a few commits and not all.

Permanently deleted user

Created February 21, 2017 19:10

Hi Denis,

We are experiencing similar problems with our TeamCity Commit Status Publisher for GitLab as well. We have observed this same error in the same condition.

We are also experiencing other symptoms as follows:

GitLab reports an external pipeline build in the "Running" state for each build configuration with the commit status publisher enabled. In our case, we have two configurations. Then, when the build completes on TeamCity, it publishes the status that it "Failed/Succeeded" into a new build ID on GitLab. So for every set of triggers, we're guaranteed to see 4 builds in a single pipline on GitLab.

One other symptom we see is TeamCity does not seem to report certain build errors as failure thus leaving GitLab in the "Running" state until the build is manually cancelled on GitLab itself. I have seen this most prominently in the MSBuild Step when a build fails in a post build event (project script).

Does any of this sound suspicious to the TeamCity side?

Anton Zamolotskikh

Created February 22, 2017 15:56

Hi Oren,

Could you confirm please that you are using gitlab.com? Or is it some private installation of GitLab? And in the latter case what version is it?

Also would you mind to check if either owner or repository name in your VCS root URL contains dots? There was a GitLab issue with it. If this issue affects you case you can download Commit Status Publisher for TC 10.0.x from the artifacts of the most recent build here: https://teamcity.jetbrains.com/viewType.html?buildTypeId=TeamCityPluginsByJetBrains_CommitStatusPublisher_TeamCity100x - Please download it and install in the TeamCity plugin directory.

And as Denis suggests here, would you mind to provide us more info?

Thanks

Anton Zamolotskikh

Created February 24, 2017 08:09

Tanner,

regarding your first issue, it seems to be strange behaviour of GitLab, that always creates "external job" when status arrives (it seems they don't expect a single build to report its status more than once, which is rather strange it seems to be relatively recent change of behaviour). I have contacted them on Twitter and they suggested to record an issue with them, so I did: https://gitlab.com/gitlab-org/gitlab-ce/issues/28656 We'll see how it goes.

As for the issue post-build events, we will investigate it and come back to you. Do you see similar behaviour (holding "Running" status forever) in other types of configs BTW?

Thanks

Anton Zamolotskikh

Created February 27, 2017 13:12

Hi Tanner, GitLab guys say (see that issue tracker link in my previous comment) the issue with GitLab creating a new Job Id (or Build Id) arises when status description sent to GitLab differs between statuses (and it should differ if the statuses are different). They say they will fix it on their side.

Anton Zamolotskikh

Created February 27, 2017 13:47

As for the second issue you report, Tanner, would you mind to reproduce it with a debug logging turned on in TeamCity and send us relevant fragment(s) of the server log? (either by reporting an issue in https://youtrack.jetbrains.com/issues/TW and making a private attachment to it, or by uploading the logs to our FTP server - see https://confluence.jetbrains.com/display/TCD10/Reporting+Issues#ReportingIssues-UploadingLargeDataArchives).

Thanks a lot!

Permanently deleted user

Created February 27, 2017 15:18

Hi Anton,

We created a relevant issue on the GitLab side and they have done some investigating on it as well. I will link it here and you can find all the relevant logs we were able to pull in that thread.

https://gitlab.com/gitlab-org/gitlab-ce/issues/28600

GitLab is returning an error 500 because GitLab is trying to invoke a method that it cannot find called reverse_merge. This error only happens on a failed build in general. We've found that it is not only done on the post build steps, but in any build configuration that simply fails.

Please sign in to leave a comment.