Trouble with Powershell and NAnt during build

I'm trying out Teamcity as a possible replacement for our current CruiseControl installation. Everything was going great until I got the agent up and running:

My NAnt build has an EXEC task for a powershell build. Agent will hang indefinitely( > 30 minutes) on this task without moving. Same task works fine if I execute NAnt from the commandline(even from the Work directory on the Agent).

Any ideas?

0
17 comments

What version of TeamCity do you use?
What version of NAnt is used?

Could you please send me hanging part of the script to reproduce.

Could you please try to capture logs from .NET part of TeamCity. For doing it please do the following:
- select some build agent
- open /plugins/dotnet-plugin/bin - backup teamcity-log4net.xml - in teamcity-log4net-debug.xml uncomment the line containing '' - rename teamcity-log4net-debug.xml to teamcity-log4net.xml - run the build. all log files will be places to ]]>/tmp/buildTmp/TeamCity.NET folder.
Please attach all the files from that folder.

Thanks!

0
Avatar
Permanently deleted user

Eugene,

TeamCity Version - 3.1.1(build 6828)
NAnt version .85

Hangs on this target:

		
			
			
		
	
]]>


I've attached the log file that was generated by the steps you detailed.

Thanks,

-Pete



Attachment(s):
debug.zip
0

Thanks. Seems array is broken.
You may email it to eugene[dot]petrenko]at[jetbrains.com

What is done under that powershell call? Does it produces much output?

0
Avatar
Permanently deleted user

The powershell call is a DB build script. It generates about 300 lines of messages during the build process. Last week I tested without the status messages being printed by the script and it failed in the exact same fashion.

-Pete

0

I got the logs.

Could you please try taking thread dumps feature for that hanging build. What is shown for NAnt process?
Thanks!

0
Avatar
Permanently deleted user

Two processes listed in the thread dump, dumps below:

U', Microsoft.Win32.Win32Native::ReadFile]
   at System.IO.FileStream.ReadFileNative(Microsoft.Win32.SafeHandles.SafeFileHandle handle, System.Byte bytes, System.Int32 offset, System.Int32 count, System.Threading.NativeOverlapped overlapped, System.Int32 hr)
   at System.IO.FileStream.ReadCore(System.Byte buffer, System.Int32 offset, System.Int32 count)
   at System.IO.FileStream.Read(System.Byte array, System.Int32 offset, System.Int32 count)
   at System.IO.StreamReader.ReadBuffer()
   at System.IO.StreamReader.ReadLine()
   at NAnt.Core.Tasks.ExternalProgramBase.StreamReaderThread_Output()
   at System.Threading.ThreadHelper.ThreadStart_Context(System.Object state)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state)
   at System.Threading.ThreadHelper.ThreadStart()

Thread id=10652 priority=8 state=Wait reason=Executive
   at [Internal thisFrame, 'M-->U', Microsoft.Win32.Win32Native::ReadFile]
   at System.IO.FileStream.ReadFileNative(Microsoft.Win32.SafeHandles.SafeFileHandle handle, System.Byte bytes, System.Int32 offset, System.Int32 count, System.Threading.NativeOverlapped overlapped, System.Int32 hr)
   at System.IO.FileStream.ReadCore(System.Byte buffer, System.Int32 offset, System.Int32 count)
   at System.IO.FileStream.Read(System.Byte array, System.Int32 offset, System.Int32 count)
   at System.IO.StreamReader.ReadBuffer()
   at System.IO.StreamReader.ReadLine()
   at NAnt.Core.Tasks.ExternalProgramBase.StreamReaderThread_Error()
   at System.Threading.ThreadHelper.ThreadStart_Context(System.Object state)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state)
   at System.Threading.ThreadHelper.ThreadStart()


Process command line
Error -11. Timeout.

6/18/2008 8:44:25 AM

Stack trace depth is set to show all frames

Thread id=10644 priority=8 state=Wait reason=Executive
   at [Internal thisFrame, 'M-->U', Microsoft.Win32.Win32Native::ReadFile]
   at System.IO.FileStream.ReadFileNative(Microsoft.Win32.SafeHandles.SafeFileHandle handle, System.Byte bytes, System.Int32 offset, System.Int32 count, System.Threading.NativeOverlapped overlapped, System.Int32 hr)
   at System.IO.FileStream.ReadCore(System.Byte buffer, System.Int32 offset, System.Int32 count)
   at System.IO.FileStream.Read(System.Byte array, System.Int32 offset, System.Int32 count)
   at System.IO.StreamReader.ReadBuffer()
   at System.IO.StreamReader.ReadLine()
   at System.IO.TextReader+SyncTextReader.ReadLine()
   at Microsoft.PowerShell.WrappedDeserializer..ctor(Microsoft.PowerShell.SerializationDataFormat dataFormat, System.String streamName, System.IO.TextReader input)
   at Microsoft.PowerShell.Executor.ExecuteCommandAsyncHelper(System.Management.Automation.Runspaces.Pipeline tempPipeline, System.Exception exceptionThrown, Microsoft.PowerShell.ExecutorExecutionOptions options)
   at Microsoft.PowerShell.ConsoleHost.InitializeRunspace(System.String initialCommand, System.Boolean skipProfiles, System.Collections.ObjectModel.Collection initialCommandArgs)
   at Microsoft.PowerShell.ConsoleHost.DoRunspaceLoop(System.String initialCommand, System.Boolean skipProfiles, System.Collections.ObjectModel.Collection initialCommandArgs)
   at Microsoft.PowerShell.ConsoleHost.Run(System.String bannerText, System.String helpText, System.Boolean isPrestartWarned, System.String args)
   at Microsoft.PowerShell.ConsoleHost.Start(System.Management.Automation.Runspaces.RunspaceConfiguration configuration, System.String bannerText, System.String helpText, System.String preStartWarning, System.String args)
   at Microsoft.PowerShell.ConsoleShell.Start(System.Management.Automation.Runspaces.RunspaceConfiguration configuration, System.String bannerText, System.String helpText, System.String preStartWarning, System.String args)
   at Microsoft.PowerShell.UnmanagedPSEntry.Start(System.String consoleFilePath, System.String args)
   at System.RuntimeMethodHandle.InvokeMethodFast(System.Object target, System.Object arguments, System.Signature sig, System.Reflection.MethodAttributes methodAttributes, System.RuntimeTypeHandle typeOwner)
   at System.Reflection.RuntimeMethodInfo.Invoke(System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object parameters, System.Globalization.CultureInfo culture, System.Boolean skipVisibilityChecks)
   at System.Reflection.RuntimeMethodInfo.Invoke(System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object parameters, System.Globalization.CultureInfo culture)

Thread id=10664 priority=12 state=Wait reason=UserRequest

Thread id=10672 priority=9 state=Wait reason=UserRequest
   at System.Threading.WaitHandle.WaitAny(System.Threading.WaitHandle waitHandles, System.Int32 millisecondsTimeout, System.Boolean exitContext)
   at System.Management.Automation.Internal.ObjectStream.WaitRead()
   at System.Management.Automation.Internal.ObjectStream.Read(System.Int32 count)
   at System.Management.Automation.Internal.ObjectStream.Read()
   at System.Management.Automation.Internal.ObjectReader.Read()
   at System.Management.Automation.Internal.Pipe.Retrieve()
   at System.Management.Automation.CommandProcessorBase.Read()
   at System.Management.Automation.ShellFunctionCommandProcessor.ProcessRecord()
   at System.Management.Automation.CommandProcessorBase.DoExecute()
   at System.Management.Automation.Internal.PipelineProcessor.Inject(System.Object input, System.Boolean enumerate)
   at System.Management.Automation.Internal.PipelineProcessor.SynchronousExecuteEnumerate(System.Object input, System.Collections.Hashtable errorResults, System.Boolean enumerate)
   at System.Management.Automation.Internal.PipelineProcessor.Execute(System.Array input)
   at System.Management.Automation.Runspaces.LocalPipeline.InvokeHelper()
   at System.Management.Automation.Runspaces.LocalPipeline.InvokeThreadProc()
   at System.Threading.ThreadHelper.ThreadStart_Context(System.Object state)
   at System.Threading.ExecutionContext.runTryCode(System.Object userData)
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state)
   at System.Threading.ThreadHelper.ThreadStart()

]]>

0

Peter,

I had something similar happen to me in Ant when I was trying to use ]]> task without the full path to the executable for perl. I ended up having to write something like this:

...

    
        
    

]]>


I don't know if it would work in NAnt though.

Hope this helps,
Oleg.

0
Avatar
Permanently deleted user

Oleg,

Thanks for the tip, but unfortunately specifying the entire path didn't seem to have an affect.

-Pete

0

So according to logs, I see that NAnt process is waiting for powershell to finish, while powershell is trying to read some file.

What user is used for build agent service? Does that user have enough rights to work? Sometimes SYSTEM accound does not match.

Could you please try settings some working folder for the script.

Let's replace powershell script with 'hello world' like script. Does that still hang?

Thanks!

0
Avatar
Permanently deleted user

I have created the following test script, called "test.ps1":


I created a new task in my NAnt.build file:

		
			
		
	
]]>


I modified the build agent to run the new task, same behavior, hangs indefinitely.

I also checked the permissions for the service, service is set to run under SYSTEM, SYSTEM has full privileges on the BuildAgent directory and all subdirectories.

-Pete

0

Thanks!

Could you please try running it under some real user on the machine. There were several issue on SYSTEM account used for build agent.

0
Avatar
Permanently deleted user

Created a new user with administrator privileges on the machine, set build agent service to run under the new user, restarted the service.

Same behavior, hangs on the hello.world task.

0

Maybe the builder in TeamCity cannot find "..\script\test.ps1" because it starts in a different directory from the one you start in by hand?

0
Avatar
Permanently deleted user

Peter,

Did you ever find a solution to the problem of powershell hanging waiting for input? I've managed to run into the same issue, and my thread trace looks identical. PowerShell.exe seems to be waiting for input.

I've actually managed to work around the issue by writing a batch file and causing TC to execute that instead of powershell.exe directly, and in the batch file I have something like this:

@echo off
echo whatsthedeal | powershell.exe -noninteractive -noprofile -nologo -command "& 'my_script.ps1'"

But that's not ideal for obvious reasons.

0
Avatar
Permanently deleted user

Issue was never resolved. I switched to Cruise Control.

0

We created the issue to address the problem. Eugene will look into it to investigate if this powershell limitation can be workarounded.

0
Avatar
Permanently deleted user

Note than I'm executing powershell.exe directly using the command line runner, and I've also tried executing a batch file instead that launched powershell.exe instead. Neither worked. The only way I got it to work was to put in my batch file something like:

echo something_nonsensical_here | powershell.exe {command line parameters here}

... in which case it worked. I ended up working around the issue by writing a small replacement tool for powershell.exe that would execute my powershell script.

0

Please sign in to leave a comment.