Failed to publish artifacts: Failed to upload artifact for build with id: 42406, error: java.io.IOException: There are no more files

Answered

I started getting this java.io.IOException last week. We have multiple builds setup and running for years and there's only one that's throwing this error.

Here's what I did:

1. I rebooted our build machine, and I was able to get a good build. But it showed up again, and now even reboot won't fix it. 

2. I checked the source and destination drives, and both have plenty of free space.

3. I checked the virus scanner, and there's no issue logged.

4. I modified the "Artifact paths" to only copy 1 sub-folder, instead of the whole bin folder, and it's good. But I need all of them.

Any idea what might be causing it and what I can do to fix it?

Here's the TeamCity version:

    env.TEAMCITY_JRE C:Program FilesJavajdk1.8.0_25
    env.TEAMCITY_VERSION 2019.1.1 (build 66192)

This is from build log before the issue occurred:

[00:12:28]Publishing artifacts (15m:25s)
[00:12:28][Publishing artifacts] Collecting files to publish: [bin => bin, -:bin\**\*.bsc => bin, -:bin\**\*.exp => bin, -:bin\**\*.iobj=> bin, -:bin\**\*.ipdb => bin]
[00:12:30][Publishing artifacts] Publishing 1305 files using [WebPublisher]: bin => bin
[00:12:30][Publishing artifacts] Publishing 1305 files using [ArtifactsCachePublisher]: bin => bin
 
This is after:
 
[03:23:09]Publishing artifacts (11m:30s)
[03:23:09][Publishing artifacts] Collecting files to publish: [bin => bin, -:bin\**\*.bsc => bin, -:bin\**\*.exp => bin, -:bin\**\*.iobj=> bin, -:bin\**\*.ipdb => bin]
[03:23:15][Publishing artifacts] Publishing 1305 files using [WebPublisher]: bin => bin
[03:23:15][Publishing artifacts] Publishing 1305 files using [ArtifactsCachePublisher]: bin => bin
[03:34:40][Publishing artifacts] Failed to publish artifacts: Failed to upload artifact for build with id: 42276, error: java.io.IOException: There are no more files
[03:34:40]Failed to publish artifacts: Failed to upload artifact for build with id: 42276, error: java.io.IOException: There are no more files
 
0
9 comments

Hi Richard,

First of all, allow me to mention that you are using an outdated TeamCity version that is no longer supported. So the first thing you should try is to upgrade to a more recent version. I suggest 2020.1.1, which is the latest version available as of now. 

If the upgrade doesn't help, try the following:

  • Scan hard drives for errors on your build machine. It is possible that there is a bad index somewhere that causes the error.
  • Despite the fact that your antivirus didn't have any records about the file, try completely disabling the AV software.
  • Check that the total number of opened files on your build machine doesn't exceed the maximum allowed number (16384 for Windows) - this should be checked during the build execution
0

Hi, I'm still getting this error intermittently. I've done the following:

  • updated to the latest version of TeamCity 2020.1.2 (build 78726),.
  • scan hard drives, with no error
  • checked that the whole build folder is except from antivirus 
  • I've also tried to install the latest version of OpenJDK and modify TeamCity config to use it, made no difference
  • delete the whole checkout folder for the failed build under BuildAgent\work\, the few times I've tried this, the next build would be successful, But it seems the error would come back after a while.

I do not know how to monitor number of open files. Do you know any tool that we can use?

Also I've seen the same error with smaller builds on another build machine that publish about 50 files. Although that is rare. We are seeing the error more often with builds that ran for more than 1 hour, and that publish several hundred to over a thousand files.

Here's the new log, if I break the copy to several folders, it failed in 1 of them but succeeded in others.

[20:00:16]Publishing artifacts (12m:27s)
[20:00:16][Publishing artifacts] Collecting files to publish: [bin\any cpu => bin\any cpu, bin\Win32 => bin\Win32, bin\x64 => bin\x64]
[20:00:21][Publishing artifacts] Publishing 595 files using [ArtifactsCachePublisher]: bin/any cpu => bin/any cpu
[20:00:21][Publishing artifacts] Publishing 595 files using [WebPublisher]: bin/any cpu => bin/any cpu
[20:01:12][Publishing artifacts] Publishing 441 files using [ArtifactsCachePublisher]: bin/Win32 => bin/Win32
[20:01:12][Publishing artifacts] Publishing 441 files using [WebPublisher]: bin/Win32 => bin/Win32
[20:06:57][Publishing artifacts] Failed to publish artifacts: Failed to upload artifact for build with id: 43252, error: java.io.IOException: There are no more files
[20:06:57]Failed to publish artifacts: Failed to upload artifact for build with id: 43252, error: java.io.IOException: There are no more files
[20:06:57][Publishing artifacts] Publishing 363 files using [ArtifactsCachePublisher]: bin/x64 => bin/x64
[20:06:57][Publishing artifacts] Publishing 363 files using [WebPublisher]: bin/x64 => bin/x64
[20:12:42]Build finished
 
Also, in my original post, the order of publish was [WebPublisher], then [ArtifactsCachePublisher], and the error was after [ArtifactsCachePublisher]. Now the order if reversed, and the error showed up after [WebPublisher]. Is this order changed with the latest version of TeamCity?
0

Hi Richard,

Could you please upload full build logs and teamcity-agent.log from the corresponding agent to https://uploads.jetbrains.com/

Please also let me know if the agent in question is based on Windows, Linux, or MacOS?

0

Hi Mikhail Efremov ,

Still we are facing same issue, As per your comment I have attached the logs. 

Please find the the id 2020_09_15_FChoddd2nw7RGes1.

0

Hi Kumar,

Sorry for the delay. Unfortunately, the build log alone does not shed the light on the problem. Could you please provide another example: a full build log, teamcity-agent.log from the corresponding agent, and teamcity-server.log from the server-side.

0

Hi Mikhail Efremov,

Could you please tell me the steps how to get these logs.

full build log, teamcity-agent.log, tenacity-server.log. It would be great if you provide me this information as soon as possible.

0

Hi Kumar,

This article will help you to locate server-side logs, and this article will help to find the agent logs.

0

Hi Mikhail Efremov

I have added server logs, agent logs, client logs, attached. Still we are facing the issue multiple times.

Please find the logs under 2021_02_05_EYxdvHDtFKZFR7pw upload id.

Please help us in getting a resolution.

 

Thank you,

Kumar

0

Hello Kumar,

It looks like the problem didn't appear in the build log you provided. But I found a record for another build in the server logs:

[2021-02-02 21:03:59,894] WARN - jetbrains.buildServer.SERVER - Failed to upload artifact for build with id: 48410: java.io.IOException: There are no more files (enable debug to see stacktrace)

Build 48410 uses 48409 (the one you provided) as its dependency. Could you please upload the corresponding build and agent logs?

Additionally, you mentioned that this problem happens only in a particular build configuration. Is that correct? Does it happen only with specific build agent(s) or on different agents?

0

Please sign in to leave a comment.