2020.1 Failure Loading Builds List

Answered

Version: TeamCity Professional 2020.1 (build 78475)

UI Action: RECEIVE_BUILDS_DATA

Error Page indicates this url

 

https://<snip>.com/app/rest/ui/builds?locator=defaultFilter%3Afalse%2CsnapshotDependency(to%3A(id%3A15567))%2Cor%3A(personal%3Afalse%2Cand%3A(personal%3Atrue%2Cuser%3Acurrent))%2Ccount%3A-1&fields=nextHref,count,build(id,number,branchName,defaultBranch,queuedDate,startDate,finishDate,history,composite,links(link(type,relativeUrl)),comment(text,timestamp,user(id,name,username)),statusChangeComment(text,timestamp,user(id,name,username)),statusText,status,state,failedToStart,personal,pinned,pinInfo(text,timestamp,user(id,name,username)),user(id,name,username),canceledInfo(text,user(id,name,username)),agent(name,id,links(link(type,relativeUrl)),environment(osType),typeId,connected,pool(id,name)),tags(tag(name,private),$locator(private:any,owner:current)),artifacts($locator(count:1),count:($optional)),limitedChangesCount($optional),buildType(id,paused,internalId,projectId,name,type,links(link(type,relativeUrl))),snapshot-dependencies(count:(1)),running-info(percentageComplete,elapsedSeconds,estimatedTotalSeconds,leftSeconds,probablyHanging,lastActivityTime,outdated,outdatedReasonBuild(number,links(link(type,relativeUrl)))),waitReason,queuePosition,startEstimate,triggered(date,displayText,build(id,number,links(link(type,relativeUrl))),buildType(id,paused,internalId,projectId,name,type,links(link(type,relativeUrl)))))

 

and that it's returning a 400 Bad Request as an html error handling page.

The Branches tab just has no data, the Builds page has an "Error while loading builds" message.

 

0
10 comments
Avatar
Fedor Rumyantsev

Hello!

It could be related to this issue: https://youtrack.jetbrains.com/issue/TW-66341
Could you please let me know if there are any special chars (i.e. "#", parentheses etc.) in the branch names of the project in question? Is that the only project affected or do you see the issue for every project?

0

This is occurring with every project.

As for specifically branch names though I have this occurring for projects that do contain some special characters (forward slash and hyphen), but also for projects with no special branches - just master. Our Github Organization name is hyphenated though in case that's relevant.

Turning on debug-rest and pulling the teamcity-rest.log file doesn't show me any failure indications interestingly. It seems perfectly happy with all 200s - though that does seem to be specifically the Branch lookups, not builds.

0
Avatar
Fedor Rumyantsev

Hello,

Thank you for the details. No, hyphens/slashes do not count as illegal characters so this might be different issue. 
May I ask you to:
1) capture HAR file (for example, as per this instruction: https://toolbox.googleapps.com/apps/har_analyzer/)
2) via developer tools on your browser, access the console and export the log
3) send them over?

You could do so in a private manner via Submit a Request form above; please mention this thread on the request if possible so this gets routed to me.
In case of any questions, please let me know.


0

While I was looking to capture a har without some of our enterprise cookies present I think I stumbled onto some clarifications to be made on this.

 

Opened an Incognito Chrome window and logged in as Guest, no issues.

I then logged into my user account using Github Enterprise as an oauth provider with the teamcity-oauth plugin found here: https://github.com/pwielgolaski/teamcity-oauth/ and the issue returned.

Logging out and returning to the Guest account made the issue go away.

 

At the moment I think that's a good enough indicator to me that this is a conflict with the plugin and not something tied to the content or any additional baggage cookies I might be getting by having this deployed on an enterprise subdomain.

 

0
Avatar
Fedor Rumyantsev

Hello,

Thank you for this detail - it`s really appreciated! oAuth plugin above is, unfortunately, a third-party development so there is little I could suggest in terms of fixing its logic. Still, if you would consider to fork it and would need any insight on the TeamCity logic, please let me know.

We also have a feature request regarding GHE oAuth out-of-the-box support: https://youtrack.jetbrains.com/issue/TW-65724; please feel free to vote for it.

0

Alright, had another person with the same issue on their server. They are not using the oauth plugin - just teamcity's built-in account management.

The other common link is that we're both using an nginx reverse proxy in front of these servers. They were able to get a result hitting the server directly with auth. I was able to get a result using Teamcity's guest account through the reverse proxy. At this point it could be the reverse proxy's handling of some auth cookie/etc, but I'm fiddling with it to figure that out.

I'm having issues catching this in the teamcity logs though - I can't seem to find a log that indicates an issue.

0
Avatar
Fedor Rumyantsev

Hello,

I would suggest starting with catalina.<date_of_issue>.log files; chances are that the header issues would be reflected there, if any. Please feel free to send those my way via Submit a request so I will check them further.

0

Seeing logs now that indicate a "Request header is too large" coming from tomcat.

I have a 6416 character cookie for some other enterprise headers and I'm guessing that was just exceeding the tomcat default maxHttpHeaderSize of 8kb. I just modified the tomcat server.xml directly with a ghetto sed, but I'll go back and read if there's a more approved way to load that in some teamcity configuration property.

Just clicking around it appears to have resolved the issue.

 

sed -i -e "s/<Connector /<Connector maxHttpHeaderSize=\"16384\" /g" server.xml
0
Avatar
Fedor Rumyantsev

Hello!

As far as I could see, no, this approach is perfectly fine - quick shuffle through the documentation shows that we do not have an option to control header max size from TeamCity configuration. I will check internally on that matter, though, and will get back to you if there is an option which could be kept within TC configuration.|

Please let me know if the issue reoccurs for you.

0

Thanks for checking, Fedor.

We rebuild servers fairly regularly so I've just added that replacement to our scripts and it'll work just fine for me.

0

Please sign in to leave a comment.