svn: No buffer space available

Please Help! I got a strange error after Subversion upgrade to 3.1.1. TeamCity Server works well about 12 hours, then brings this message: "svn: No buffer space available (maximum connections reached?): JVM_Bind". In previous version all was all right. After rebooting all build agents run again next 12 hours. There is some details in Log:

Failed for the root 'svn: http://svn2.fancom.lv/svn/sams/Servants/trunc' #1: Checking changes for checkout rule '.=>.' failed with erorr: Cannot request dated revision from svn: svn: PROPFIND request failed on '/svn/sams/Servants/trunc' svn: No buffer space available (maximum connections reached?): JVM_Bind

jetbrains.buildServer.vcs.VcsException: Checking changes for checkout rule '.=>.' failed with erorr: Cannot request dated revision from svn: svn: PROPFIND request failed on '/svn/sams/Servants/trunc'
svn: No buffer space available (maximum connections reached?): JVM_Bind
at jetbrains.buildServer.vcs.VcsSupportUtil.collectBuildChanges(VcsSupportUtil.java:33)
at jetbrains.buildServer.buildTriggers.vcs.svn.SvnSupport.collectBuildChanges(SvnSupport.java:18)
at jetbrains.buildServer.buildTriggers.vcs.VcsChangesLoader.collectBuildChanges(VcsChangesLoader.java:16)
at jetbrains.buildServer.buildTriggers.vcs.VcsChangesLoader.loadChanges(VcsChangesLoader.java:41)
at jetbrains.buildServer.vcs.impl.VcsManagerImpl.loadChanges(VcsManagerImpl.java:305)
at jetbrains.buildServer.vcs.impl.VcsManagerImpl.checkForModifications(VcsManagerImpl.java:400)
at jetbrains.buildServer.serverSide.impl.auth.SecuredVcsManager.checkForModifications(SecuredVcsManager.java:85)
at jetbrains.buildServer.serverSide.impl.VcsModificationChecker$1.run(VcsModificationChecker.java:5)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:417)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
at java.util.concurrent.FutureTask.run(FutureTask.java:123)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
at java.lang.Thread.run(Thread.java:595)
Caused by: jetbrains.buildServer.vcs.VcsException: Cannot request dated revision from svn: svn: PROPFIND request failed on '/svn/sams/Servants/trunc'
svn: No buffer space available (maximum connections reached?): JVM_Bind
at jetbrains.buildServer.buildTriggers.vcs.svn.SvnConnection.getRevision(SvnConnection.java:467)
at jetbrains.buildServer.buildTriggers.vcs.svn.SvnConnection.getEntries(SvnConnection.java:235)
at jetbrains.buildServer.buildTriggers.vcs.svn.SvnConnection.addChanges(SvnConnection.java:452)
at jetbrains.buildServer.buildTriggers.vcs.svn.SvnSupport.collectBuildChanges(SvnSupport.java:135)
at jetbrains.buildServer.vcs.VcsSupportUtil.collectBuildChanges(VcsSupportUtil.java:27)
... 13 more
Caused by: org.tmatesoft.svn.core.SVNException: svn: PROPFIND request failed on '/svn/sams/Servants/trunc'
svn: No buffer space available (maximum connections reached?): JVM_Bind
at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:55)
at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:40)
at org.tmatesoft.svn.core.internal.io.dav.DAVUtil.findStartingProperties(DAVUtil.java:124)
at org.tmatesoft.svn.core.internal.io.dav.DAVUtil.getVCCPath(DAVUtil.java:148)
at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.getDatedRevision(DAVRepository.java:164)
at jetbrains.buildServer.buildTriggers.vcs.svn.SvnConnection.getRevision(SvnConnection.java:227)
... 17 more
jetbrains.buildServer.vcs.VcsException: Cannot request dated revision from svn: svn: PROPFIND request failed on '/svn/sams/Servants/trunc'
svn: No buffer space available (maximum connections reached?): JVM_Bind
at jetbrains.buildServer.buildTriggers.vcs.svn.SvnConnection.getRevision(SvnConnection.java:467)
at jetbrains.buildServer.buildTriggers.vcs.svn.SvnConnection.getEntries(SvnConnection.java:235)
at jetbrains.buildServer.buildTriggers.vcs.svn.SvnConnection.addChanges(SvnConnection.java:452)
at jetbrains.buildServer.buildTriggers.vcs.svn.SvnSupport.collectBuildChanges(SvnSupport.java:135)
at jetbrains.buildServer.vcs.VcsSupportUtil.collectBuildChanges(VcsSupportUtil.java:27)
at jetbrains.buildServer.buildTriggers.vcs.svn.SvnSupport.collectBuildChanges(SvnSupport.java:18)
at jetbrains.buildServer.buildTriggers.vcs.VcsChangesLoader.collectBuildChanges(VcsChangesLoader.java:16)
at jetbrains.buildServer.buildTriggers.vcs.VcsChangesLoader.loadChanges(VcsChangesLoader.java:41)
at jetbrains.buildServer.vcs.impl.VcsManagerImpl.loadChanges(VcsManagerImpl.java:305)
at jetbrains.buildServer.vcs.impl.VcsManagerImpl.checkForModifications(VcsManagerImpl.java:400)
at jetbrains.buildServer.serverSide.impl.auth.SecuredVcsManager.checkForModifications(SecuredVcsManager.java:85)
at jetbrains.buildServer.serverSide.impl.VcsModificationChecker$1.run(VcsModificationChecker.java:5)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:417)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
at java.util.concurrent.FutureTask.run(FutureTask.java:123)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
at java.lang.Thread.run(Thread.java:595)
Caused by: org.tmatesoft.svn.core.SVNException: svn: PROPFIND request failed on '/svn/sams/Servants/trunc'
svn: No buffer space available (maximum connections reached?): JVM_Bind
at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:55)
at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:40)
at org.tmatesoft.svn.core.internal.io.dav.DAVUtil.findStartingProperties(DAVUtil.java:124)
at org.tmatesoft.svn.core.internal.io.dav.DAVUtil.getVCCPath(DAVUtil.java:148)
at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.getDatedRevision(DAVRepository.java:164)
at jetbrains.buildServer.buildTriggers.vcs.svn.SvnConnection.getRevision(SvnConnection.java:227)
... 17 more
org.tmatesoft.svn.core.SVNException: svn: PROPFIND request failed on '/svn/sams/Servants/trunc'
svn: No buffer space available (maximum connections reached?): JVM_Bind
at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:55)
at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:40)
at org.tmatesoft.svn.core.internal.io.dav.DAVUtil.findStartingProperties(DAVUtil.java:124)
at org.tmatesoft.svn.core.internal.io.dav.DAVUtil.getVCCPath(DAVUtil.java:148)
at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.getDatedRevision(DAVRepository.java:164)
at jetbrains.buildServer.buildTriggers.vcs.svn.SvnConnection.getRevision(SvnConnection.java:227)
at jetbrains.buildServer.buildTriggers.vcs.svn.SvnConnection.getEntries(SvnConnection.java:235)
at jetbrains.buildServer.buildTriggers.vcs.svn.SvnConnection.addChanges(SvnConnection.java:452)
at jetbrains.buildServer.buildTriggers.vcs.svn.SvnSupport.collectBuildChanges(SvnSupport.java:135)
at jetbrains.buildServer.vcs.VcsSupportUtil.collectBuildChanges(VcsSupportUtil.java:27)
at jetbrains.buildServer.buildTriggers.vcs.svn.SvnSupport.collectBuildChanges(SvnSupport.java:18)
at jetbrains.buildServer.buildTriggers.vcs.VcsChangesLoader.collectBuildChanges(VcsChangesLoader.java:16)
at jetbrains.buildServer.buildTriggers.vcs.VcsChangesLoader.loadChanges(VcsChangesLoader.java:41)
at jetbrains.buildServer.vcs.impl.VcsManagerImpl.loadChanges(VcsManagerImpl.java:305)
at jetbrains.buildServer.vcs.impl.VcsManagerImpl.checkForModifications(VcsManagerImpl.java:400)
at jetbrains.buildServer.serverSide.impl.auth.SecuredVcsManager.checkForModifications(SecuredVcsManager.java:85)
at jetbrains.buildServer.serverSide.impl.VcsModificationChecker$1.run(VcsModificationChecker.java:5)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:417)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
at java.util.concurrent.FutureTask.run(FutureTask.java:123)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
at java.lang.Thread.run(Thread.java:595)

15 comments

Edgar,

What is the OS?

How many agents do you have? Are they installed on separate computers?

On what computer does the error occur?

Does the error reproduce?

Did you perform any other environment changes together with TeamCity upgrade?

--
Best regards,

Yegor Yarko
Project Manager
JetBrains, Inc
http://www.jetbrains.com
"Develop with pleasure!"

0

Hi from Riga and some details:

>What is the OS?
MS Server 2003
>How many agents do you have? Are they installed on separate computers?
One agent on same computer
>On what computer does the error occur?
On the same computer with Server & Agent running
>Does the error reproduce?
It reproduced after 6-12 hours
>Did you perform any other environment changes together with TeamCity upgrade?
No

And one more - SVN server is installed on foreign computer with openSuSe 10.2.

0

Edgar,

Thank you for the more detailed information.

We are aware of the fact that such error can occur sometimes under Windows, but could not yet identify the cause and are are not sure that it is TeamCity who causes the problem.

When the error occurs again, could you please try to:
- restart TeamCity server process only. Does it help?
- restart TeamCity agent process. Does it help?

Could you please let us know what was the TeamCity version you upgraded from that did not have the problem while running it?

When the error is occurring, can you please run TCPView ( http://technet.microsoft.com/en-us/sysinternals/bb897437.aspx ) to see if there are large number of opened connections on the machine?

If order of identify the process that causes the problem, it would help to run agent on a separate computer and see if the error occur on server or agent part.


--
Best regards,

Yegor Yarko
Project Manager
JetBrains, Inc
http://www.jetbrains.com
"Develop with pleasure!"

0

Ok, but just tomorrow when it occurs again. About Version - it was previous. I don't remember bild number, but I think version was 3.0.1. About restarting agent and server - I am tested that. Agent was restarted, but server hangs and restarted only after rebooting

0

Edgar,

About restarting agent and server - I am tested that. Agent was restarted, but server hangs and restarted only after rebooting


Do you mean that agent process restart did not help and only reboot helped?
Please try restarting the server process to see if it can help?

--
Best regards,

Yegor Yarko
Project Manager
JetBrains, Inc
http://www.jetbrains.com
"Develop with pleasure!"

0

I mean, I was trying to restart TeamCity Server from system services list, but it hungs (can't stopped) and after that all attempts to connect it with browser was failed

0

Hello.
I got this listing today morning with TcpView:


:0 TCP a64:8080 ip-118-115.zb.lv:60733 TIME_WAIT
:0 TCP a64:8080 ip-118-115.zb.lv:60739 TIME_WAIT
:0 TCP a64:8080 ip-118-115.zb.lv:60741 TIME_WAIT
:0 TCP a64:8080 ip-118-115.zb.lv:60743 TIME_WAIT
:0 TCP a64:8080 ip-118-115.zb.lv:60737 TIME_WAIT
:0 TCP a64:8080 ip-118-115.zb.lv:60747 TIME_WAIT
:0 TCP a64:8080 ip-118-115.zb.lv:60749 TIME_WAIT
:0 TCP a64:8080 ip-118-115.zb.lv:60751 TIME_WAIT
:0 TCP a64:8080 ip-118-115.zb.lv:60753 TIME_WAIT
:0 TCP a64:8080 ip-118-115.zb.lv:60755 TIME_WAIT
:0 TCP a64:8080 ip-118-115.zb.lv:60757 TIME_WAIT
:0 TCP a64:8080 ip-118-115.zb.lv:60759 TIME_WAIT
:0 TCP a64:8080 ip-118-115.zb.lv:60745 TIME_WAIT
:0 TCP a64:8080 ip-118-115.zb.lv:60763 TIME_WAIT
:0 TCP a64:8080 ip-118-115.zb.lv:60765 TIME_WAIT
:0 TCP a64:8080 ip-118-115.zb.lv:60767 TIME_WAIT
:0 TCP a64:8080 ip-118-115.zb.lv:60761 TIME_WAIT
Apache.exe:1524 TCP a64:3476 a64:0 LISTENING
Apache.exe:1524 TCP a64:3476 a64:0 LISTENING
FRSServer.exe:3380 TCP a64:7885 a64:0 LISTENING
inetinfo.exe:1596 TCP a64:ftp a64:0 LISTENING
inetinfo.exe:1596 TCP a64:1037 a64:0 LISTENING
inetinfo.exe:1596 UDP a64:3456 :
inetinfo.exe:1596 UDP a64:3456 :
java.exe:1840 TCP a64:9090 a64:0 LISTENING
java.exe:416 TCP a64:31000 localhost:32000 ESTABLISHED
lsass.exe:652 TCP a64:1025 a64:0 LISTENING
lsass.exe:652 UDP a64:isakmp :
lsass.exe:652 UDP a64:ipsec-msft :
mqsvc.exe:2668 TCP a64:msmq a64:0 LISTENING
mqsvc.exe:2668 TCP a64:2107 a64:0 LISTENING
mqsvc.exe:2668 TCP a64:2105 a64:0 LISTENING
mqsvc.exe:2668 TCP a64:2103 a64:0 LISTENING
mqsvc.exe:2668 TCP a64:1042 a64:0 LISTENING
mqsvc.exe:2668 UDP a64:1040 :
mqsvc.exe:2668 UDP a64:3527 :
msdtc.exe:1320 TCP a64:1026 a64:0 LISTENING
svchost.exe:1016 UDP a64:1039 :
svchost.exe:1048 UDP a64:ntp :
svchost.exe:1048 UDP a64:ntp :
svchost.exe:1048 UDP a64:ntp :
svchost.exe:3208 TCP a64:ms-wbt-server a64:0 LISTENING
svchost.exe:956 TCP a64:epmap a64:0 LISTENING
System:4 TCP a64:netbios-ssn a64:0 LISTENING
System:4 TCP a64:microsoft-ds a64:0 LISTENING
System:4 TCP a64:9342 a64:0 LISTENING
System:4 TCP a64:http a64:0 LISTENING
System:4 TCP a64:netbios-ssn a64:0 LISTENING
System:4 UDP a64:netbios-dgm :
System:4 UDP a64:netbios-ns :
System:4 UDP a64:microsoft-ds :
System:4 UDP a64:netbios-dgm :
System:4 UDP a64:netbios-ns :
TeamCityAgentService-windows-x86-32.exe:352 TCP a64:32000 localhost:31000 ESTABLISHED
TeamCityAgentService-windows-x86-32.exe:352 TCP a64:32000 a64:0 LISTENING
tomcat6.exe:43780 TCP a64:8080 a64:0 LISTENING
tomcat6.exe:43780 TCP a64:8080 ip-118-115.zb.lv:60735 ESTABLISHED
tomcat6.exe:43780 TCP a64:8005 a64:0 LISTENING
tomcat6.exe:43780 TCP a64:8009 a64:0 LISTENING




... and when i try to stop TeamCity server & Agent with shell(cmd) i saw this messages:

C:\TeamCity\bin>runAll.bat stop
Sending agent shutdown command to: http://localhost:9090
Failed to shutdown agent gracefully: No buffer space available (maximum connections reached?): connect

... then I decide to transfer my TeamCity server on other computer with Linux ( I can't transfer build agent because there is legacy and licence limitations for my project build environment). In this point all works fine. We must wait and see wat happens later.

0

After day before TC server transfering on Lunux, I saw "..(maximum connections reached?): JVM_Bind..." error again on computer with builder agent (Windows).

jetbrains.buildServer.vcs.VcsException: java.lang.RuntimeException: Cannot request dated revision from svn: svn: PROPFIND request failed on '/svn/sams/Servants/trunc'
svn: No buffer space available (maximum connections reached?): JVM_Bind
jetbrains.buildServer.vcs.VcsException: java.lang.RuntimeException: Cannot request dated revision from svn: svn: PROPFIND request failed on '/svn/sams/Servants/trunc'
svn: No buffer space available (maximum connections reached?): JVM_Bind
at jetbrains.buildServer.vcs.svn.SvnCheckoutOnAgentSupport.updateSources(SvnCheckoutOnAgentSupport.java:79)
at jetbrains.buildServer.agent.impl.patch.GetProjectSources.checkoutSources(GetProjectSources.java:272)
at jetbrains.buildServer.agent.impl.patch.GetProjectSources.execute(GetProjectSources.java:98)
at jetbrains.buildServer.agent.impl.BuildAgentImpl$BuildRunAction.run(BuildAgentImpl.java:1033)
at java.lang.Thread.run(Thread.java:595)
Caused by: java.lang.RuntimeException: Cannot request dated revision from svn: svn: PROPFIND request failed on '/svn/sams/Servants/trunc'
svn: No buffer space available (maximum connections reached?): JVM_Bind
at jetbrains.buildServer.vcs.svn.TCSvnUpdateClient.getRevision(TCSvnUpdateClient.java:44)
at jetbrains.buildServer.vcs.svn.TCSvnUpdateClient.getExternalRevision(TCSvnUpdateClient.java:30)
at jetbrains.buildServer.vcs.svn.TCSvnUpdateClient.doCheckout(TCSvnUpdateClient.java:49)
at jetbrains.buildServer.vcs.svn.SvnCheckoutOnAgentSupport.updateSources(SvnCheckoutOnAgentSupport.java:47)
... 4 more

0

Edgar,

When the error starts to occur, it will continue to occur in different places till the resources are freed.

Please try to find the process that "holds" the resources, by restarting the process and checking if the error is still in place.

If stopping TeamCity is not possible via Services, you can stop it by killing the java.exe processes form the Windows Task Manager (actually ProcessExplorer utility from Microsoft is more convenient for that). Please note that the TeamCity server is a single java.exe process (which likely uses a more memory then other javae.exe processes) and agent has two java.exe processes.

After process killing you should be able to start it form the Services (after refreshing the view).

Thank you for TCPView dump. It does not show anything suspicious.

--
Best regards,

Yegor Yarko
Project Manager
JetBrains, Inc
http://www.jetbrains.com
"Develop with pleasure!"

0

Ygor,

TeamCity server was transfered to other computer and now it works fine as I wrote before! So error is about build agent left (who was reinstalled at whole). And more, I tried to restart builder via Services. And it did it without any messages. But unfortunately, I didn't check java processes via Process Manager, so I'll do that tomorrow, when it hangs again. One note - after agent restarting via Services, it won't connecting to TeamCity server, and do this after whole system rebooting only(agent's OS needs rebooting, not Servers OS). And one more question - Is it possible to use old builder agent from 3.0.1 package with 3.1.1 server? If it is, please, could You send me this part "agentInstaller.exe" from 3.0.1 package (as I can't find previous distributive) to test this configuration?

---
Thank You for Patience And Have a Nice Day.

0

If TeamCity was moved to another PC but problem still occurs on the agent then chances are this problem is not related to SVN. Also I do not think that previous version of agent will help, simply because we did not do any significant changes related to network communications. Is it possible that one of your builds started to produce many messages in build log? I think a great amount of messages produced at short time could cause such problems.

BTW if I understand you right restarting of the agent did not help, only whole reboot of agent PC solves the problem, in this case it looks like a bug in operating system.

--
Pavel Sher

0

>if TeamCity was moved to another PC but problem still occurs on the agent then chances are this problem is not related to SVN.
Yes. I think the same manner
>Also I do not think that previous version of agent will help
I found out 3.0.1 version and installed it. Let's see tomorrow whats happens.
>Is it possible that one of your builds started to produce many messages in build log? I think a great amount of messages produced at short time could cause such problems.
It figures right, but previous version runs without any problems. Why?
>BTW if I understand you right restarting of the agent did not help, only whole reboot of agent PC solves the problem, in this case it looks like a bug in operating system.
My opinion, this is because all new TCP sockets opening is blocked for whole system because overflow.

--
Edgar

0

I found out 3.0.1 version and installed it. Let's see tomorrow whats happens.


Did you revert your TeamCity server to version 3.0.1? Because installing of the agent from version 3.0.1 won't work.

It figures right, but previous version runs without any problems. Why?


So you had a build producing many messages and it worked fine in 3.0.1, right? Probably our agent started to produce even more messages. Could you please describe your build? How many tests do you run? What is the average test duration in the build? What test framework do you use? Are there other tasks except tests runner producing many messages?

--
Pavel Sher

0

>Did you revert your TeamCity server to version 3.0.1? Because installing of the agent from version 3.0.1 won't work.
I ask You before about compatibility between versions, but not received response. So I replaced builder agent only and it works fine with server 3.1.1 for this time.
>So you had a build producing many messages and it worked fine in 3.0.1, right?
for this time. Let's see this tomorrow.
>Could you please describe your build? How many tests do you run? What is the average test duration in the build? What test framework do you use? Are there other tasks except tests runner producing many messages?
I have no time to prepare all this things today . I'll describe it in detail tomorrow. But few words about my project: there is two solutions in (for vs2008 and the exactly the same for vs2005). Each solution has 35 projects and 7 from them is NUnit test projects. Common number of unit tests is 29 for now with average running time from 10ms to 20s. Time for building and testing for each solution is about 3m20s. And there is NSpring configs, and NHibernate mappings to MsSQL (SQLEXPRESS).
But one more important note - builder agent hangs(cause exceed sockets pool) usualy at night, when it is in idle state.

0

I ask You before about compatibility between versions, but not received response. So I replaced builder agent only and it works fine with
server 3.1.1 for this time.


Sorry, I overlooked it somehow. I just wanted to mention that to downgrade agent you should:
- copy update/buildAgent. files from previous TeamCity to update/ directory
- copy update/plugins/. files from the previous TeamCity to update/plugins/ folder

Then you should see that all agents become outdated and automatic upgrade should be done.

--
Pavel Sher

0

Please sign in to leave a comment.