Perforce streams with checkout rules
Hello,
I'm trying to find ways to improve our Perforce integration with TeamCity. My project uses Perforce streams, so I don't have the ability to use checkout rules to filter down which parts of the workspace are relevant for each job.
This has a few downsides:
- Every build needs to sync the whole workspace tree, even if only a small part is relevant to each job. This is especially rough since I work in game development, and our depot contains lots of source art/audio which is large and not usually relevant to our builds.
- We can't re-use "suitable" snapshot dependency builds without checkout rules. For instance, we have a compile job that only cares about changes to the src folder, and an automated testing job which cares about all changes. The automated testing job will always force a rebuild of the compile job, even if no code changes have been checked in since the last build.
- It's not feasible for a job to have multiple VCS roots. The data from the stream workspace will get duplicated for every VCS permutation.
In my mind, the ideal solution would be:
- Each VCS root only gets a single workspace, regardless of differing checkout rules. This would mean we'd need to expose the client root directory via a build parameter (since it's no longer the checkout directory).
- Checkout rules are applied to the p4 sync commands instead of the workspace spec. Theoretically this would also allow us to specify individual files instead of only folders.
This would all be Perforce specific and surely would have to be opt-in to not break existing systems. But a way to use checkout rules for stream workspaces would be a massive benefit for us. Is doing something like this feasible?
Best,
Joe
Please sign in to leave a comment.
Hi Joe!
Could you expand on why you can't use checkout rules? As mentioned in the documentation, you should be able to use them with streams https://www.jetbrains.com/help/teamcity/perforce.html#Use+Perforce+Streams:
• When streams are used with the agent-side checkout mode (https://www.jetbrains.com/help/teamcity/vcs-checkout-mode.html#agent-checkout), simple checkout rules like `. => sub/directory` are supported. Exclude checkout rules, multiple include rules, or rules like `aaa=>bbb` are supported only when the "Create non-stream workspace" option is enabled (see below (https://www.jetbrains.com/help/teamcity/perforce.html#Agent+Checkout+Settings)).
Cheers,
Anatoly
Hi Anatoly! Thanks for the follow up. I should've clarified in my original post, the stream checkout rules you mentioned do work, they're just not ideal for my case. I definitely want to use stream workspaces, so enabling the "Create non-stream workspace" option won't work. The simple checkout rules also work, but they only allow you to change the root of a stream workspace within the checkout directory, not filter down which subfolders are relevant.
I understand the reason for these limitations is because of the way Perforce streams work. The checkout rules are applied to the workspace clientspec, and with streams you cannot edit the clientspec on an individual workspace, since it is authored in the stream itself.
My hope with this post is to find a way to improve this for streams, as without anyway to filter down stream workspaces, we probably won't be able to use TeamCity's VCS integration long-term. There's nothing that we can do with the existing system since it's a Perforce limitation, but potentially the method that I mentioned in the original post could work (applying checkout rules to the p4 sync command). I'd love to hear your guys' thoughts on this.
Yes, some of our builds will submit data back to Perforce. We also have multiple streams that go through our CI/CD pipeline, so using stream workspaces is extremely important since we can re-use them between streams. This helps keep sync times down and keeps us from duplicating a lot of data on disk.