Internal Server Error

Are you sure you do not have any proxy servers between agent and server?

The server and all agents are on a LAN, with no proxy between them. All systems here do use a proxy for accessing the internet though, is there anyway TeamCity could be grabbing this option (from Start -> Settings -> Control Panel -> Internet Options -> Connections -> LAN Settings) and attempting to use it? We use a configuration script for the proxy access if that makes any difference (the script just specifies that localhost requests should go direct, anything that resolves to an internal IP goes direct (which would include the TeamCity server), and anything that goes within our parent company goes direct. All other requsts go through a cache.

Steve

Agent connects to the server by HTTP, using URL provided in buildAgent.properties file.
Does the problem occur with any agent, any artifact?

I've just checked the build log for one of the builds:

Total artifacts published (attempted) 6095
Total internal server errors 147

The particular configuration that generates the above build only runs on one agent as it requires specialist hardware only available to that agent.

I checked another couple of more 'standard' configurations (just doing compiling), and found:

Total artifacts published (attempted) 24
Total internal server errors 0

I checked a number of builds for the above configuration and all were fine - so not sure if that indicates there is something wrong with the agent of the build configuration?

By the way, we are now in the process of setting up a server with the latest version of TeamCity, in hope we can migrate everything over (wanted to upgrade anyway), and have the problem disappear. Could still be useful to know what is going on here though.


Steve

We had some fixes related to uploading of large artifacts, but as I remember the errors reported were a bit different. So I am not sure why the problem stopped appearing.
Another possible cause is installed antivirus software.

Hi there,

We have done a fresh install of the latest version of TeamCity onto a Linux server (previously we were using 4.5 on a Windows server), and I've just setup two projects, to make sure this internal server error is gone, and I am sad to say it is not. In the last build I had four of these errors:

Do you see the same error in the server log?

Nope, seem to just have errors to do with permissions (Permission denied), for example:

localhost.2010-04-19.log:SEVERE: Exception Processing ErrorPage[exceptionType=java.lang.Exception, location=/runtimeError.jsp]
localhost.2010-04-19.log:       at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
localhost.2010-04-19.log:Caused by: java.io.FileNotFoundException: /opt/TeamCity/work/Catalina/localhost/_/TC_TeamCity-main/org/apache/jsp/runtimeError_jsp.java (Permission denied)

teamcity-javaLogging-2010-04-16.log:SQL error when doing: Querying for a single string
teamcity-javaLogging-2010-04-16.log:    at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1122)

teamcity.log:[2010-04-19 11:38:34,685]  ERROR - rverSide.impl.BuildNumbersImpl - java.io.FileNotFoundException: /data/TeamCity/BuildServer/config/TeamCity Dev/bt3.buildNumbers.properties (Permission denied)
teamcity.log:[2010-04-19 11:39:58,127]   WARN - facts.ArtifactUploadController - Failed to upload build artifact due to error: java.io.FileNotFoundException: /data/TeamCity/BuildServer/system/artifacts/TeamCity Dev/Harry_GAMEBUILD/22/logs/env_vars_at_start.txt (No such file or directory)
teamcity.log:[2010-04-19 11:40:01,378]   WARN - facts.ArtifactUploadController - Failed to upload build artifact due to error: java.io.FileNotFoundException: /data/TeamCity/BuildServer/system/artifacts/TeamCity Dev/Harry_GAMEBUILD/22/logs/AccurevPopulateTTBuilder.log (No such file or directory)

teamcity-javaLogging-2010-04-16.log:SQL error when doing: Querying for a single string
teamcity-javaLogging-2010-04-16.log:    at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1122)

I have now asked support to look in to this (I am an Enterprise customer), but just to keep this thread up to date, the server is reporting errors like:

[2010-05-17 12:49:32,335]  ERROR -   jetbrains.buildServer.SERVER - Error in JSP, request dump:

Path: /runtimeError.jsp

Method: POST

org.springframework.web.multipart.MultipartException: Could not parse multipart servlet request; nested exception is org.apache.commons.fileupload.FileUploadBase$IOFileUploadException: Processing of multipart/form-data request failed. Stream ended unexpectedly

      at org.springframework.web.multipart.commons.CommonsMultipartResolver.parseRequest(CommonsMultipartResolver.java:172)

      at org.springframework.web.multipart.commons.CommonsMultipartResolver.resolveMultipart(CommonsMultipartResolver.java:149)

      at jetbrains.spring.web.TeamCityMultipartResolver.resolveMultipart(TeamCityMultipartResolver.java:8)

      at org.springframework.web.servlet.DispatcherServlet.checkMultipart(DispatcherServlet.java:1015)

      at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:851)

      at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:807)

      at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:571)

      at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:511)

      at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)

      at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)

      at jetbrains.buildServer.rootDispatcher.TeamCityDispatcherServlet.service(TeamCityDispatcherServlet.java:36)

      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)

      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

      at jetbrains.buildServer.web.SetThreadNameFilter.runChainWithModifiedThreadName(SetThreadNameFilter.java:20)

      at jetbrains.buildServer.web.SetThreadNameFilter.doFilter(SetThreadNameFilter.java:12)

      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)

      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

      at jetbrains.buildServer.web.ResponseFragmentFilter.doFilter(ResponseFragmentFilter.java:2)

      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)

      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

      at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)

      at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)

      at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)

      at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)

      at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)

      at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)

      at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:852)

      at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)

      at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)

      at java.lang.Thread.run(Thread.java:619)

Caused by: org.apache.commons.fileupload.FileUploadBase$IOFileUploadException: Processing of multipart/form-data request failed. Stream ended unexpectedly

      at org.apache.commons.fileupload.FileUploadBase.parseRequest(FileUploadBase.java:359)

      at org.apache.commons.fileupload.servlet.ServletFileUpload.parseRequest(ServletFileUpload.java:126)

      at org.springframework.web.multipart.commons.CommonsMultipartResolver.parseRequest(CommonsMultipartResolver.java:165)

      ... 29 more

Caused by: org.apache.commons.fileupload.MultipartStream$MalformedStreamException: Stream ended unexpectedly

      at org.apache.commons.fileupload.MultipartStream$ItemInputStream.makeAvailable(MultipartStream.java:964)

      at org.apache.commons.fileupload.MultipartStream$ItemInputStream.read(MultipartStream.java:887)

      at java.io.InputStream.read(InputStream.java:85)

      at org.apache.commons.fileupload.util.Streams.copy(Streams.java:94)

      at org.apache.commons.fileupload.util.Streams.copy(Streams.java:64)

      at org.apache.commons.fileupload.FileUploadBase.parseRequest(FileUploadBase.java:354)

      ... 31 more

Which match up more or less with the original errors (from our Windows, TeamCity 4.5x box, so it appears changing Server OS and TeamCity version did not fix this).

Steve

Are you sure you do not have antivirus software on agent or on server?

We do have anti-virus software on our agents, we are part of a multinational organisation that forces all machines on the domain to run anti-virus software. I imagine any large Enterprise level organisation will have the same restriction.

However, I server is now running on Linux, and there is no 'anti-virus' software installed on there. What leads you to think it is the anti-virus software based on the call stack I have pasted above by the way? It seems weird to me that anti-virus software would case a Java exception such as the above to occur.

Steve

Seems to be other people having somewhat similar problems:

http://osdir.com/ml/lang.groovy.grails.user/2007-05/msg01346.html

Although that was quite a while ago - and I'm not entirely sure how much of it applies here.

Steve

According to error message:
Processing of multipart/form-data request failed. Stream ended  unexpectedly

The upload stream was aborted on the client (agent). Chances are it was caused by antivirus. I would try to run several builds on some agent with disabled antivirus.

We can not have any machines logged in to our domain without antivirus software installed, furthermore, I've just looked in the logs for the anti-virus program, and it is not reporting any activity for today, so it really doesn't look like it is part of the equation.

Steve

I've just spoke to IT, they say it should be possible to get our parent company to exclude certain directories from the AntiVirus software, so I'll get them to exclude the TeamCity build agents working directories tomorrow.

Steve

What antivirus do you use? Maybe you can disable antivirus for some time, disable agent, and run several builds manually on this agent?

In fact there aren't so many options here:
- proxy (you told you don't have one)
- antivirus

Some antivirus software also install proxies to catch internet traffic. If you have such an antivirus, disabling of scanning in some folders may not help.

We use Symantec Antivirus - the builds themselves are 100% successful. The only thing that is failing is publishing artifacts to TeamCity. The artifacts in question are plain text files, just containing the output from the compiler linker. While I understand Antivirus programs can get a bit overexcited (I've seen them quarantine compiler intermediate files), it seems very unlikely they would falsely detect a plain ASCII .txt file of this nature.

I'll see about getting the TeamCity build agent folders excluded from the antivirus and take it from there - but as I say, the antivirus log files do not show any hits for today.

Steve

Well, if Symantec Antivirus installs some proxy (redirects network operations to its' scanner), there easily can be a bug in antivirus itself, especially if many files are transferred through such proxy. I would not say it is unlikely because antivirus software may not be tested for such cases. How often regular users upload many files via HTTP?

Let's also add some debug logging on the agent, try to add the following category to <agent inst dir>/conf/teamcity-agent-log4j.xml:

  <category name="org.apache.commons.httpclient">
    <priority value="DEBUG"/>
    <appender-ref ref="ROLL"/>
  </category>

Then, if error happens again, send logs to teamcity-feedback[at]jetbrains.com

I have heard back from IT:

'OK sounds like the antivirus is a red herring then. Symantec is only doing real-time scanning  and does not use any proxy even a local one. We don't have the web filtering component installed precisely because as its known to cause these sort of problems'

Any further thoughts?

Steve

Please try to enable logging, as I recommended in the previous post. Let's see what's happening at the lower level.

Okay - I have enabled the debug logs and will post when it next happens.

Thanks,

Steve

On TeamCity logs for build:

[



2010-05-18 17:38:19,197]  ERROR -   jetbrains.buildServer.SERVER - Error in JSP, request dump:
Path: /runtimeError.jsp
Method: POST
 
org.springframework.web.multipart.MultipartException: Could not parse multipart servlet request; nested exception is org.apache.commons.fileupload.FileUploadBase$IOFileUploadException: Processing of multipart/form-data request failed. Stream ended unexpectedly
     at org.springframework.web.multipart.commons.CommonsMultipartResolver.parseRequest(CommonsMultipartResolver.java:172)
     at org.springframework.web.multipart.commons.CommonsMultipartResolver.resolveMultipart(CommonsMultipartResolver.java:149)
     at jetbrains.spring.web.TeamCityMultipartResolver.resolveMultipart(TeamCityMultipartResolver.java:8)
     at org.springframework.web.servlet.DispatcherServlet.checkMultipart(DispatcherServlet.java:1015)
     at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:851)
     at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:807)
     at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:571)
     at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:511)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
     at jetbrains.buildServer.rootDispatcher.TeamCityDispatcherServlet.service(TeamCityDispatcherServlet.java:36)
     at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
     at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
     at jetbrains.buildServer.web.SetThreadNameFilter.runChainWithModifiedThreadName(SetThreadNameFilter.java:20)
     at jetbrains.buildServer.web.SetThreadNameFilter.doFilter(SetThreadNameFilter.java:12)
     at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
     at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
     at jetbrains.buildServer.web.ResponseFragmentFilter.doFilter(ResponseFragmentFilter.java:2)
     at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
     at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
     at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
     at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
     at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
     at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
     at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
     at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
     at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:852)
     at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
     at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
     at java.lang.Thread.run(Thread.java:619)
Caused by: org.apache.commons.fileupload.FileUploadBase$IOFileUploadException: Processing of multipart/form-data request failed. Stream ended unexpectedly
     at org.apache.commons.fileupload.FileUploadBase.parseRequest(FileUploadBase.java:359)
     at org.apache.commons.fileupload.servlet.ServletFileUpload.parseRequest(ServletFileUpload.java:126)
     at org.springframework.web.multipart.commons.CommonsMultipartResolver.parseRequest(CommonsMultipartResolver.java:165)
     ... 29 more
Caused by: org.apache.commons.fileupload.MultipartStream$MalformedStreamException: Stream ended unexpectedly
     at org.apache.commons.fileupload.MultipartStream$ItemInputStream.makeAvailable(MultipartStream.java:964)
     at org.apache.commons.fileupload.MultipartStream$ItemInputStream.read(MultipartStream.java:887)
     at java.io.InputStream.read(InputStream.java:85)
     at org.apache.commons.fileupload.util.Streams.copy(Streams.java:94)
     at org.apache.commons.fileupload.util.Streams.copy(Streams.java:64)
     at org.apache.commons.fileupload.FileUploadBase.parseRequest(FileUploadBase.java:354)
     ... 31 more

I will report this issue to support directly too.

Steve

Steve, please provide agent logs (teamcity-agent.log file and others), or send them to teamcity-feedback[at]jetbrains.com.

I have emailed them, as there is some confidential information stored within. I'm not sure the agent that the build was running on in this case had debug logging enabled (the server did), I have enabled for this agent now though (we have ~10 agents).

Cheers,

Steve

Just had the problem again, and have sent more logs to the email address provided.

Steve

Hi Pavel,

Was there ever any resolution to this issue? We are still having this problem.

Thanks,
Jamie.

It seems there is nothing to fix on our side. TeamCity does not terminate HTTP connections, this can be done by various proxies, antivirus and other software. At the moment we do not think this is our bug, and we obviously can't do anything with such software.

Hi Pavel,

I think I have fixed the problem. The problem was we were repeatedly overwriting and re-publishing the same file (intentionally) but sometimes a prior publish must still have been in operation when the file was newly overwritten, thus breaking the send of the file over the network. The problem went away when we made the filenames unique (by adding a counter into the filename).

Thanks for your help.
Jamie.

I'm getting this issue on  EAP 7.0 too. Running on Windows and on an OLD box. Agent and Server are on same machine.  It could very well be that it is just too dang slow.  But the errors are consistent on the publishing of artifacts. I'll check into using the same files like the last post mentioned.

getting issue in 9.1.8