Clean checkout takes much longer time by using shared VCS root in TeamCity?

Hi All,

I have changed the build configurations in TeamCity to use a shared VCS root according to the main branch in Version Control System. With the new configuration, clean checkout is failing repeatedly, needs much longer time and needs number of retries to succeed. Sometimes, it also generates error message: ‘Error while applying patch: Problem while loading patch data stream’.

It would be great if someone faced similar situation in the past or JetBrains developers could suggest something.

 

Thanks,

Sobit Thapa

18 comments
Comment actions Permalink

Hi Sobit,


Could you please attach a build log with the error, VCS settings used and teamcity-vcs.log This will allow us to start looking into the issue.

0
Comment actions Permalink

Hi Yegor,

Thank you very much for your reply. We have Perforce as VCS and following is an example of the VCS root configurations:

Client Mapping: 
//team/abc/libs/... //team-city-agent/...

Checkout Rules:
-:.
+:/xyz/main/



We are using Server Side Checkout. It is taking significantly longer time to perform clean checkout than using previous configuration (eg: //team/abc/libs/xyz/main/...........).

An example of error log when it fails while applying patch is following:

[10:02:17]Checking for changes
[10:02:19]Publishing internal artifacts
[10:02:19][Publishing internal artifacts] Sending build.start.properties.gz file
[10:02:19]Clearing temporary directory: /Applications/buildAgent/temp/buildTmp
[10:02:19]Checkout directory: /Applications/buildAgent/work/ee1dsbab94221fbd
[10:02:19]Updating sources: server side checkout (5m)
[10:02:19][Updating sources] Will perform clean checkout. Reason: Agent doesn't have any version of the project sources
[10:02:19][Updating sources] Building clean patch for VCS root: ABC_LIBS
[10:07:19][Updating sources] Problem while loading patch data stream: Connection reset
[10:07:19]Will repeat attempt when server will be available, number of attempts left: 2
[10:07:29]Updating sources: server side checkout (5m)
[10:07:29][Updating sources] Will perform clean checkout. Reason: Agent doesn't have any version of the project sources
[10:12:30][Updating sources] Problem while loading patch data stream: Connection reset
[10:12:30]Will repeat attempt when server will be available, number of attempts left: 1
[10:12:40]Updating sources: server side checkout (5m:02s)
[10:12:40][Updating sources] Will perform clean checkout. Reason: Agent doesn't have any version of the project sources
[10:17:42][Updating sources] Problem while loading patch data stream: Connection reset
[10:17:42]Problem while loading patch data stream: Connection reset
[10:17:42]
java.net.SocketException: Connection reset
java.net.SocketException: Connection reset
 at java.net.SocketInputStream.read(SocketInputStream.java



Looking forward to hearing from you.

Thanks,

Sobit
0
Comment actions Permalink

Hi Sobit,


Please also attach  teamcity-vcs.log and teamcity-server.log form the servr covering the issue.

0
Comment actions Permalink

Hi Yegor,

Thank you very much for your response. Please find the attached log files from the server covering the failure.

Looking forward to hearing from you soon.

Thanks,

Sobit



Attachment(s):
teamcity-server.log.zip
teamcity-vcs.log.zip
0
Comment actions Permalink

Hi Sobit,

Looks like http://youtrack.jetbrains.com/issue/TW-23749

Could you please check and let us know:
- Is your server running on Windows?
- If yes, do you have Windows updates KB2735855 and KB2750841 installed?
- Do you have any anti-virus program installed?

0
Comment actions Permalink

Hi Yegor,

Thank you very much for your reply.

Our TeamCity server is running on Debian 5 and we are not using any anti-virus software. Moreover, the problem started after using shared VCS root rather than upgrading to TeamCity 7.1. Before the change, everything was working fine.

Thanks,

Sobit

0
Comment actions Permalink

Hi Sobit,

Thank you for the details.

What OS does the agent run?

The issue from the server logs seems like this:
To obtain a patch from the server, agent sends an HTTP request to download the data. At some point server encounters "java.net.SocketException: Connection reset" Java-reported error while streming data to the agent.

From your first message it seems that the agent receives the same error.

Could you please also attach relevant agent logs?
Actually, I'd appreciate if you can attach complete server logs, agents logs and build logs covering the same occurrence as I am not sure the server logs you provided correspond to the build log included into the first message.
There are ways not to make the logs public if you do not want that.

Most probable reason for the errors is some external (external to TeamCity) activity interfering with the HTTP connection. Can that be the case? Do you have proxies/firewalls/etc. in between the agent and the server?
The build log you included originally seem to get the error after exactly 5 minutes of downloading and there are no such liitations in TeamCity.

Can you please also describe how often does it fail with the error andare all the agents affected?

0
Comment actions Permalink

Hi Yegor,

Sure. I will send you agent logs as well as other information.

Anyway, our agents run on all popular operating systems like Windows, Mac, Unix etc. We have encountered several other issues including this one after changing VCS root configurations to use a shared VCS root. Before that, our build infrastructure was stable.

Thanks,

Sobit

0
Comment actions Permalink

Sobit,

> We have encountered several other issues including this one after changing VCS root configurations to use a shared VCS root. Before that, our build infrastructure was stable.

We would need to look into all the issues in detail (and separately). Please make sure you describe them.
We have no known stability issues related to VCS roots reuse, so this still might be a coincedence.

0
Comment actions Permalink

Hi Yegor,

Thank you very much for the reply. Unfortunately, I could not send the complete log files at this time.

Regarding the build failure and agent details, it fails consistently in Mac agents for certain projects. We were facing similar issues on other OS, but builds succeeded after couple of retries. Other than the failure, clean checkout takes much longer time after sharing a VCS root anyway.



Thanks,

Sobit
0
Comment actions Permalink

Hi Yegor,

We are using default Tomcat which comes with the TeamCity. I was wondering is there any default settings regarding socket connection reset?

Looking forward to hearing from you soon.

Thanks,

Sobit

0
Comment actions Permalink

Hi Sobit,

> I was wondering is there any default settings regarding socket connection reset?

Not that I know about.

0
Comment actions Permalink

Hi Yegor,

I found a strange solution to this problem. After setting an option in 'Build Failure Condition' tab of project configurations, as ' it runs longer than 90 minutes', our problem has gone. I have to increase it even more like 120/270 minutes for some projects.

It looks completely different setting, but somehow Socket Connection Reset Exception problem has been solved after this step. Another strange thing is that after making this option enable, some of our builds finished before 5 minutes on which we were getting exceptions.

So, could you please investigate more on it and find out how these things are related? If I am assuming something wrong, please let me know.

Looking forward to hearing from you soon.

Thanks,
Sobit

0
Comment actions Permalink

Hi Sobit,

Any relation between build fialure condition and "Connection reset" errors on checkout would be utterly strange.

Are you absolute sure this is not a coincedence?

Could you please double-check the relation and let us know the results? For example you can try removing the failure condition and try to reproduce the issue, then adding it back again and doing the same.

0
Comment actions Permalink

Hi Yegor,

Yes, I can reproduce same error by removing the setting I mentioned. But, this does not help when 'enable clean checkout' option is selected.

It may be coincidence but helped us somehow (I am also not convinced with this solution/result). I would like to remind you that only Mac build agents are failing builds with these errors but started working after this change.

Thanks,

Sobit

0
Comment actions Permalink

Sobit,

I have nothing more to add to my previous reply.

If you want us to look into the isuse please attach full server and agent logs covering the issue (preferably with debug turned on for both server and agent).

0
Comment actions Permalink

Hi Yegor,

Although, we upgraded the TeamCity to its latest version (7.1.4) and moved it to more reliable and powerful server, the issue "error while applying patch" is still there. (Reminder: this issue was produced when we started using the shared VCS root configuration.)

So, I would like to re-open our discussion by sending the log files covering the issues. Please let me know if this issue is fixed by the JetBrains recently.

Thanks,
Sobit

0
Comment actions Permalink

Hi Sobit,

There is no one issue related to the case, they might be totally different, even if they seem to be the same.

Please create an issue in the tracker and describe all the relevant details from scratch with the data and the logs noted before.

0

Please sign in to leave a comment.