Very large Perforce clientspecs (> 3 GB)

FollowFollowed by 5 people

Hello,

  With the current checkout process, TeamCity reads data from the
Perforce server and after that sends them to build agent.

   Also, TeamCity uses other command for checkout on server than p4 sync. We use
   p4 -G print //depot/path/...@rev
   for checkout, and this command works more slow.

   But we need p4 -G print to obtain file information details (such as executable status) to send data to build agents.

   Actually, if you specify p4 mapping in TeamCity configuration (not as Perforce client name, but explicit mapping),
  TeamCity server will cache perforce sources on the TeamCity server and subsequent checkouts should be much faster.

  May be this would help in your case.

  Best regards,
  KIR

Hello,

  From the log I see that sources from server to agent were transferred during 7-8 minutes.

  Other time were spent during obtaining sources from Perforce server (3 hours). I'm not sure why does it take so long.
  You may try running p4 -G print command, which we use to get the sources, and see how long would it work.

  KIR

Could you provide the whole P4 -G command syntax please.

Would I run this on a clientspec ?

Hi,

our perforce head is ~10GB. TeamCity 5.1.3 on Windows 7 64bit can build the patch in 40min and transfer it in about 10min.
Normally the clean patch is cached, so this is the worst case (see log below). All agents are connected via Gigabit-Ethernet
and we run the server with a 64bit VM (1.6.0_18). The server has 8GB of RAM and we use 2.5GB for teamcity with
   -Xms2506m -Xmx2506m -XX:MaxPermSize=256m

Maybe this helps.

Hello,

   TeamCity runs p4 print -G for each line in your clientspec specification. Exact command format which is run on your system
   you can find in TeamCity's logs (teamcity-vcs.log) if you enable debug logging.

   The typical format is
   p4 -G print //depot/path/...@revision_num

   and this command outputs the data to output.

   Hope this helps,
   KIR

Kir,

From the tests that I've done, it appears that "p4 -G print" may run incredibly slow for binary files in p4. Try it on some .dll or .exe and see the difference from the same command on a text file (nearly instantaneous output). We've seen our TeamCity server grind to a halt if there are just 3-4 simultaneous builds that are syncing a change that includes multiple binary files. Is it not possible for TeamCity to get executable status, revision info and whatever else using "p4 -G fstat" command and then use p4 sync to get the actual files?

Thanks,
Oleg.

Hello Oleg,

  The problem here is that we run p4 print -G command once for the whole changeset (and for all files in it).
  In this case we can build the whole patch with a single call.

  What you suggest is to get list of changed files first, and than get each file individually, correct? If so, this is a kind of N+1 problem, and if it may be faster for cases of several big binary files, it will be slower for other cases. This can only be an option for a VCS root, and this VCS root should not contain many files.

  Also, didn't you consider using Perforce checkout on agent mode? In this case, simple p4 sync command will be used and there will be no overhead of p4 print -G command.

  Regards,
  KIR

Hi Kir,

Thanks for the explanation. I did not realized that the process is different between server and agent modes. We are using server mode and have not yet tried the agent mode, the reason being trying to have the server build the patch once and distribute it to the agents. It seems advantageous to do that if the patches are large (true for some of our builds).

I understand the N+1 concern, but I am not suggesting going through each file manually. In server mode, you can run "p4 fstat" on the whole changelist for each line of the client spec, followed by "p4 sync" and then a robust copy to the agent. For example, consider this p4 spec:

//depot/folder1/... //team-city-agent/depot/folder1/...
//depot/folder2/... //team-city-agent/depot/folder2/...

The pseudo-code should be like this:

// get the changelist number somehow
num = whateverWayTeamCityGetsTheChangelistNumber();
for each line in p4spec
{
    // get the depot path like //depot/folder1/...
    depotPath = extractTheFirstHalf(line);
    numberedDepotPath = depotPath+"@"+num;
    p4 fstat numberedDepotPath; // this prints all kind of info about file status, you can use -G if you need
    // do what you need to fstat info
    ...
    // sync files
    p4 sync numberedDepotPath;
    // transfer to the agent using some robust copy command that you probably have already
    ...
}

Even though you would use two commands, time spent in fstat is insignificant compared to the sync. Moreover, sync command is more natural to use to get the files from p4 onto a client computer. p4 print seems to be a workaround just to accomplish things in a single command.

I may not have a complete understand on what's under the hood of TeamCity - just trying to find ways to improve performance. If we have to, we will try agent mode, but I'm afraid of a potential redundant checkout it might bring to our installation.

Thanks,
Oleg.

Hi

We used to use Server Side checkouts for our agents using perforce, our head was about 26GB this would take 30-40Mins . However since switching to agent side checkout the only takes 16mins.

Thanks to Krill for his work on this

Pod

Hello Oleg,

  The matter is, TeamCity doesn't keep Perforce working copy on the server side - it creates the patch to the server "on the fly".
  So, we cannot use p4 sync command on the server. The reason for that is that it is possible that 2 builds are being created
  for the same repository path at the same time for different revisions, so we cannot share the working copy on the server).
  We have some caches for the built patches on the server side, but only as a baseline for incremental patches.

  Instead, you can try using checkout on agent mode - it uses a real working copy and synchronizes sources with native p4 sync command.
  This mode provides much faster checkouts, but requires installation of the perforce client on all the build agents.

   Kind regards,
   KIR

I understand now. Thanks, Kir. We'll try the agent mode and see how it performs.

Kir,
We are testing the agent checkout mode in 6.5.4. We ran against http://youtrack.jetbrains.net/issue/TW-12318, which would prevent us from switching. Bumping that to a 6.5.x release would certainly help. ;)

Oleg,

  Did you consider enabling the option "clobber writable" in the Perforce specification? Would it help?

  Regards,
  KIR

No, we didn't. :-)
We just tried it, and it seemed to work in one personal build. We'll see how it performs over time and possibly use it as a workaround. Thanks for the suggestion!