Here was my problem, we currently use 1 repository for all our projects, can be large, up to a few gig of projects.
Now the way the Vault Plugin works, it will transfer the entire repository.. which ends up with errors, either running out of memory or pathtoolong
Here is how I got it working.
1. First, I would create a shared VCS root to use Vault
2. On the project, select the VCS root and make sure you select "Do NOT checkout files automatically"
3. Create a build step, make sure it is the first build step, I suggest creating a batch file since it will contain a password, and have it run the following command.
"C:\Program Files\SourceGear\Vault Client\vault.exe" GET -server serverName -user myUserName -password myPassword -repository "My Repository" $/FolderName -nonworkingfolder "%system.teamcity.build.checkoutDir%"
Way this works is.. It will detect changes from the VCS root but use the command line to pull them out of the source.
Last thing is..
4. Create a Build Trigger and add +:FolderName
Only downside is the "Changes" show for everyones checkin but only triggers on changes inside the "FolderName"