TeamCity nuget feed completely broken

Answered

I turned on a nuget feed in TeamCity and tried to push a nuget package to it. 

However `nuget push` gave me this error:

Pushing foo.nupkg to 'https://teamcity.company.com/httpAuth/app/nuget/feed/ham/eggs/'...
PUT https://teamcity.company.com/httpAuth/app/nuget/feed/ham/eggs/
RequestEntityTooLarge https://teamcity.company.com/httpAuth/app/nuget/feed/ham/eggs/ 14858ms
Response status code does not indicate success: 413 (Request Entity Too Large).

The nuget package is a mere 30 MB's. What gives?

0
10 comments
Avatar
Fedor Rumyantsev

Hello,

Could you please let me know:
1) if it was a nuget push in context of TeamCity build? 
2) what is the maximum artifact size, specified in Administration | Global Settings | Maximum build artifact file size?

0

1. No it was not the result of a nuget push from a teamcity build. It was from the command line where I called "Nuget Push <http endpoint> <nuget_package.nupkg>"

2. My maximum size is 300 MB's. 

0
Avatar
Fedor Rumyantsev

Thank you!

Unfortunately, it is only possible to push to the TeamCity-based NuGet feed from the context of the build (due to the way we handle package/build metadata internally). I have reached out internally to update our documentation so to explicitly mention this limitation. If you would use the nuget push as part of NuGet Publish runner (https://www.jetbrains.com/help/teamcity/nuget-publish.html), it would work just fine.

Speaking of the 413 response code - I believe this could be linked to the known issue about IIS reverse proxy configured in front of TeamCity (in case there is one and you try to push the package to the server, IIS by default will not allow files larger than 30 Mb - see the details here: https://confluence.jetbrains.com/display/TCD8/Known+Issues#KnownIssues-FailuretopublishartifactstoserverbehindIISreverseproxy).

0

Thanks for the clarification on the docs. 

As for a reverse proxy, I have no idea about that. But hopefully that helps someone else.

0

So I tried this in the context of a teamcity nuget push.

I added a build step to do a nuget push, specified the api key and the source feed ID. pointed it to the file and I still got the same result.

Note I am using nuget version 5.6.0.6591

[12:36:29]Response status code does not indicate success: 413 (Request Entity Too Large).
 
 
0
Avatar
Fedor Rumyantsev

Hello!

The root of the issue appears to be the same - i.e. IIS proxy set up in front of the TeamCity. If agent instance would connect to the server via the proxy as well, IIS limitations on size would still apply.

Could you please check internally if there is indeed IIS-based proxy in your setup?

0

Talk in plain language man. THis has nothing to do with IIS proxy. What is too big? Tell me: WHAT is too big? Is is the nuget package? Is it bugs bunnies gum? Your error message here is worthless here, and you tell me about some stupid IIS proxy that has nothing to do with the network setup.

-1
Avatar
Fedor Rumyantsev

Let me rephrase - earlier, I have asked what is the maximal size of the artifact configured on your TeamCity instance. TeamCity NuGet feed will refuse to process the pushed package only if it is larger than that (so, 300 MB by default). You are operating with 30 MB file, so I believe TeamCity limitations do not apply here.

However, if this is not the TeamCity who responds with 413, next thing which looks logical is that there is an intermediate resource - proxy, load balancer, any sort of middleman - which might respond with 413. I have asked you about the IIS because it looks like the most fitting candidate (as it is known not to pass through files larger than 30 MB without reconfiguration) - am I correct to assume that you do not have it on your setup? Are there any other proxies present?

Code-wise, I see that exceptions encountered during the nuget push processing on the TeamCity side should be reflected on the logs. Could you please retry the nuget push in context of the build and check if "Unhandled error while processing NuGet package" occurs on the teamcity-nuget.log (found at <TeamCity Server>/logs)? If it does, please send over the log file via Submit a request form (which allows to do so in a private manner). If it does not, though, this would prove that the issue might occur outside of TeamCity. 

On a side note - unfortunately, I did not get the reference about the gum, and I could not google it either. I would appreciate if you could explain that part to me.  

0

Our maximum artifact size is 300 MB's. 

Our Teamcity server is actually running out of an official docker container which itself is running on an ubuntu VM. All running in the google cloud. 

As for log files, there is no such thing as a teamcity-nuget.log file:

All I have is this:

```
me@machine_name:/opt/jetbrains/teamcity-server/logs$ ls -l
total 14020
-rw-r----- 1 root root 39526 Jun 29 23:43 catalina.2020-06-29.log
-rw-r----- 1 root root 16756 Jun 30 15:39 catalina.2020-06-30.log
-rw-r----- 1 root root 1832163 Jul 22 18:57 catalina.2020-07-22.log
-rw-r----- 1 root root 0 Jun 29 16:37 host-manager.2020-06-29.log
-rw-r----- 1 root root 0 Jun 30 00:13 host-manager.2020-06-30.log
-rw-r----- 1 root root 0 Jul 22 18:56 host-manager.2020-07-22.log
-rw-r----- 1 root root 0 Jun 29 16:37 localhost.2020-06-29.log
-rw-r----- 1 root root 0 Jun 30 00:13 localhost.2020-06-30.log
-rw-r----- 1 root root 0 Jul 22 18:56 localhost.2020-07-22.log
-rw-r----- 1 root root 0 Jun 29 16:37 manager.2020-06-29.log
-rw-r----- 1 root root 0 Jun 30 00:13 manager.2020-06-30.log
-rw-r----- 1 root root 0 Jul 22 18:56 manager.2020-07-22.log
-rw-r----- 1 root root 388235 Jul 23 16:54 teamcity-activities.log
-rw-r----- 1 root root 284877 Jul 23 16:27 teamcity-auth.log
-rw-r----- 1 root root 350013 Jul 23 03:00 teamcity-cleanup.log
-rw-r----- 1 root root 721679 Jul 23 16:31 teamcity-clouds.log
-rw-r----- 1 root root 24605 Jul 23 04:57 teamcity-diagnostics.log
-rw-r----- 1 root root 2128 Jun 29 21:39 teamcity-javaLogging-2020-06-29.log
-rw-r----- 1 root root 6578 Jun 30 15:40 teamcity-javaLogging-2020-06-30.log
-rw-r----- 1 root root 3431 Jul 22 18:57 teamcity-javaLogging-2020-07-22.log
-rw-r----- 1 root root 4952 Jul 22 18:57 teamcity-ldap.log
-rw-r--r-- 1 root root 0 Jul 22 18:56 teamcity.lock
-rw-r----- 1 root root 2957 Jul 22 18:57 teamcity-mavenServer.log
-rw-r----- 1 root root 1606 Jul 16 13:51 teamcity-nodes.log
-rw-r----- 1 root root 20082 Jul 23 16:54 teamcity-notifications.log
-rw-r--r-- 1 root root 5 Jul 22 18:57 teamcity.pid
-rw-r----- 1 root root 10209 Jul 22 18:57 teamcity-rest.log
-rw-r----- 1 root root 1724359 Jul 23 16:54 teamcity-server.log
-rw-r----- 1 root root 784 Jul 22 18:56 teamcity-tfs.log
-rw-r----- 1 root root 8626347 Jul 23 16:58 teamcity-vcs.log
-rw-r----- 1 root root 173178 Jul 22 20:49 teamcity-vcs-states.log
-rw-r--r-- 1 root root 1279 Jul 22 18:56 teamcity-wrapper.log
-rw-r----- 1 root root 16733 Jul 22 23:24 teamcity-ws.log
drwxr-x--- 2 root root 4096 Jul 22 22:37 threadDumps-2020-07-22
```

0
Avatar
Fedor Rumyantsev

Hello,

Thank you for the details. Indeed, the artifact size setting should not prevent you from uploading the package, from TeamCity perspective. 
Regarding the teamcity-nuget.log - apologies, I have checked first that the relevant entries should appear on this log, but I did not check that it would also require a specific logging preset. Could you please:
1) change logging preset (https://www.jetbrains.com/help/teamcity/teamcity-server-logs.html#Changing+Logging+Configuration) to debug-all;
2) rerun the configuration?
This preset will allow nearly every debug entry into the logs, so I would advice to turn it off once the log file is available

Where the agents are installed network-wise? (e.g. are they also on the cloud in the same subnet, or do they connect from an external location?) 

0

Please sign in to leave a comment.