Multiple build agents on one machine with projects that reference common libraries
We have several common libraries used by our projects; we have set those up to be build configurations in TC along with the projects we ship. We then set up our projects to have build triggering dependencies on the common libraries in TC. After setting up multiple build agents on the same machine I noticed that it was recommended not to use an absolute checkout path or use the clean directory before building option. I understand why it is not recommended but I do not know how to set up the common libraries/projects so they can reference correctly.
Our VCS has a structure like this:
-Common
---Library 1
---Library 2
-----Version 1
-----Version 2
-Products
---Shippable 1
-----Version 1
---Shippable 2
-----Version 1
-----Version 2
In TC we have a project for Library 2 and a build configuration for Version 2. That build configuration is attached to a VCS root that looks like $\Common\Library 2\Version 2\.
We also have a project for Shippable 1 and a build configuration for Version 1. That build configuration is attached to a VCS root that looks like $\Products\Shippable 1\Version 1\
Right now the checkout directory for S1V1 looks like D:\TeamCityCode\Products\Shippable 1\Version 1
The L2V2 checkout directory looks like D:\TeamCityCode\Common\Library 2\Version 2
Since S1V1 has a reference to L2V2 that is relative (..\..\..\Common\Library 2\Version 2), it works well (..\..\..\ = D:\TeamCityCode). But I think that if I change my checkout directory for S1V1 to be relative, it will not be able to reference L2V2 correctly.
I am wondering what is the recommended way to set up Team City for shared libraries and projects that reference them with relative paths?
Please sign in to leave a comment.
Hello Dan,
If your library build produces some artifacts (like jar files or dll files), you can pass these artifacts to shippable even if
library and main project are built on different computers.
Let's assume that your library creates
Common/Library2/Version2/library.jar
and that your S1V1 references this jar as ../../../Common/Library2/Version2/library.jar
Then, you can do the following:
- set artifact path for L2V2 to Common/Library2/Version2/library.jar
For S1V1 set VCS root to main project root and set checkout rules to
Products/Shippable1/Version1 => Products/Shippable1/Version1
Setup artifact dependency from S1V1 to L2V2 and set target path for library.jar to
Common/Library2/Version2
Thus, when S1V1 will be built, it will be checked out to
Products/Shippable1/Version1
and library will be downloaded to
Common/Library2/Version2/library.jar
So the relative path ../../../Common/Library2/Version2/library.jar
will work even when library is built on another computer.
Hope this helps,
KIR
That actually does help a lot, I didn't think of using artifact dependencies. The previous TC set up we had of the projects/build configurations were as you indicated:
"For S1V1 set VCS root to main project root and set checkout rules to Products/Shippable1/Version1 => Products/Shippable1/Version1"
This was actually a little nicer because we were able to share the VCS root between build configurations. But there was a moderatley annoying problem that I couldn't figure out how to work around. If S1V1 had a VCS root of the root node, and had build rules to only build when things are checked in the the project directory, it would build just fine but then the email notifications and build history for the project would include changes and check in comments for other projects that were not related.
Here is an example of the old emails I would get:
Build ConversionReprice.1.4.0.dev::ConversionReprice.1.4.0.dev.Configuration #1.4.0.271 successful Build results: http://devaci01:88/viewLog.html?buildId=559&buildTypeId=bt2
Changes included (7 changes)
====================================================
Change 10204 by devqa\dlash (2 files):
Servicing: updated MSI
Change 10201 by devqa\dmaxim (1 file):
LO.UI: Case BGM-5186-G0Y0 - Added SaveToDatabase event handler
Change 10205 by devqa\ehollan (2 files):
<no comment>
Change 10199 by devqa\ehollan (1 file):
CI: Add impersonation JS file to the MSI project.
Change 10203 by devqa\jfolken (1 file):
added folder for SSIS
Change 10202 by devqa\jfolken (80 files):
ExpenseReports: renamed folder to reflect the fact that I got yanked off the project before I could actually do anything with it
Change 10200 by devqa\jfolken (1 file):
IM: fixed future dated transactions display issues
What's interesting here is only the first change (10204) should have been included in the email, the other check-ins are for different projects. So I guess the question now is, if I use a common VCS root and build rules to control when things build so that the artifacts to loaded correctly, how can I control my email notifications and history so they only include changes from the project (essentially whatever build rule was triggered)?
Thanks a lot, I appreciate it!
Dan Lash
Ok I have re-configured TC in several different ways to try to get the artifact (and also the snapshot) dependencies working. It seems to be promising but I cannot get it 100% working. What I have come to realize is that S1V1 does not really depend on any particular outputs/artifacts of L1V2, it actually depends on the source files. With that in mind I figured that snapshot dependencies would be what I need because when S1V1 is built, L1V2 would be placed in the queue beforehand. But what is interesting is it doesn't seem to actually build L1V2.
When I build S1V1, it does correctly place L1V2 in the build queue, but it doesn't actually build L1V2 -- it just says time to start = delayed for a few seconds then goes away.
So I thought that maybe I need to set up each of my projects to yield their entire source tree as artifacts, and that each project that has a dependency on the sources would set up artifact dependencies that would place them in the correct relative paths. I do believe this would work and accomplish the build goals but I have to say that having every build download the dependent source files from the CI server, get the project sources from VCS, then building, then uploading the project sources as artifacts back to the CI server seems like WAY too much overhead.
I'm running out of ideas on how to get relative checkout directories, dependant sources, and build chains set up together. Do you have any suggestions?
Thanks a lot,
Dan Lash
What options are specified for the snapshot dependency: S1V1 -> L1V2? If you need to always rebuild L1V2 you should turn OFF "Do not run new build if there is a suitable one" option, otherwise TeamCity will not rebuild L1V2 if there is a build in the history created on the same sources.
--
Pavel Sher
You're right, if I turn off that option my common library's sources are downloaded and built, then the shippable project is downloaded and built. That definitely works... for one build agent. If I test that configuration with two build agents, however, it does not.
Here's how it goes:
- I have S1V1 have a snapshot dependency on L1V2.
- I have two idle build agents.
- I manually trigger a build of S1V1 on agent 2.
- The build queue correctly contains two builds; L1V2 first.
- Build agent 1 picks up the build of L1V2 and correctly downloads sources and builds.
- When it is finished agent 2 picks up the build of S1V1, dowloads it's sources, but when it executes the runner it does not have the source files for L1V2 because they were downloaded by agent 1.
So I guess the snapshot dependency doesn't take into account which build agent has the sources for the dependency. Is there any way other than yielding my entire source tree as a build artifact to ensure that agent 2 would have access to both S1V1 and L1V2 sources?
Thanks,
Dan Lash
When I previously set up my projects to share one common VCS root the problem I was experiencing was that my email notifications contained check in comments from non-related projects. How I had it set up was each build configuration had a build runner that referenced some nested solution file, and I set up build triggering rules to only build when files were checked in under that nested directory. The advantage to that set up was that each project had access to common library projects.
If I were to go back to that set up as indicated, I wonder if I could set up check out rules in addition to build rules to try to control what check comments are included in the emails. I'm not sure if that is exactly what it's meant for, but if this current set up of snapshot/artifact dependencies does not work out, it might be worth a shot.
Dan Lash
Snapshot dependency ensures that sources of two or more build configurations are synchronized, i.e. if these configurations share the same VCS roots then builds will take same sources, if VCS roots differ then sources will be taken on the same moment of time, nothing more. Snapshot dependency does not control which VCS root is used it only provides source code synchronization. In your case if S1V1 build requires sources of the L1V2 then you can add VCS root used by L1V2 to S1V1 too.
However if some component depends on a library to me it looks more natural to build this component with final artifacts of this library (not with the library source code), i.e. with jar or dll files. In your case L1V2 could produce such artifacts and then they could be used by S1V1.
--
Pavel Sher
Yes, it does make sense to use artifacts and it would probably make setting up TC a lot easier. However using common library sources is just how we organize and build our code -- that isn't going to change.
I see what you're saying about the snapshot dependency; I'll look into adding the VCS root of the common library to my project -- that sounds like it would work, but I'm a bit concerned that my notification emails would then include comments from the shared library. I'll report back on the results of that test.
Thanks,
Dan Lash
Ok, I have come up with a configuration that solves my problems. Just for a recap here is what I have:
- Shared libraries
- Shippable projects
- Multiple build agents on the same machine
- Build configuration per project version
The final configuration is set up like this:
- 1 VCS root (set to the root node of our VCS)
- Each build configuration has checkout rules
- - The rules enumerate every VCS path that is required to build
- No dependencies, artifacts, build triggers, etc
So for the example in the original post, S1V2 has these checkout rules:
+:Products/Shippable 1/Version 1
+:Common/Library 2/Version 2
With this configuration the shippable project is built when files are checked into either of those directories. My notification emails also only contain changes from those directories.
All in all I'm happy with the configuration and it seems to be working well. I really appreciate all of the help and suggestions!
Dan Lash
I have a similar situation where we have a common library and projects share that library. My VCS roots include one for the common library and one per each project. Each build configuration uses the common VCS root and one of the project VCS roots, 2 in total. I also have a separate project that only tests the library with the common VCS root. Is there any way to use the latest submission number in the union of VCS roots as the build number? If I combine the roots into a single root for each build configuration, {build.vcs.number.<VCS_root_name>}.{0} works great. If I separate the roots, I have to choose one of the roots to use within the build number, which might not reflect the latest change for the union. Here is an example:
Root1: latest change number is 123
Root2: latest change number is 125
If I use Root1, the build number will be 123.{0}, but that does not reflect change 125. From the build's point of view, it sees both changes and should use 125. I think that I'm looking for something like max({build.vcs.number.Root1}, {build.vcs.number.Root2}).{0} or {build.vcs.number.max}.{0}, whichever syntax is preferred. I see that there is a way to update the build number via service messages, but it won't work for configurations that strictly use built-in runners. Any suggestions on how to resolve this?
Thanks,
Oleg.
Hello Oleg,
We could provide a functionality like "max revision number". But there are some caveats:
- we can mix VCS roots of different types, for instance, perforce + svn.
- if both VCS roots will reference the same repository, you'll get the same revision number, for both roots, because by the moment TeamCity returns not directory revision, but repository global revision for the build. We have a request to add a per-directory revision number as well
Another question is why would you need to split repository into 2 VCS roots. IMO, using a single VCS root is more efficient.
Regards,
KIR
Hi KIR,
Thanks for a quick reply. I am, in fact, using the same repository for both roots (Perforce), so the same revision number will be fine in my case, just like you describe in point #2. That number really tells the reader that "this build includes changes up to [whatever revision] for all VCS roots in the configuration". Actually, I have a couple of configurations that use 5 roots, but the same reasoning applies there. Should I file a request in the tracker or is there one I can vote for already? By the way, my vote would be for {build.vcs.number.max} format for simplicity. "latest" or "last" might also work instead of "max".
If different type roots are mixed, perhaps the max functionality should not be available. Or you could pick the latest rev number (submission time-wise).
I don't know about the speed of having 2 VCS roots vs one, but imagine that I have 100 build configurations, all using the common root and a project root (2 roots/configuration). I then add something to the repository that I consider to be common and want to propagate to all configurations. With 2 roots, I only have to update the common root with the new directory and all configurations inherit it. With one root, it seems like I would have to update all 100 project roots. Not convenient...There is even more benefit to split the roots if the common root contains a lot of entries (let's say 20 directory mappings in perforce) and the project root has very few entries. This way, you won't be duplicating a lot of entries between the roots and reducing the possibility of error.
Thanks,
Oleg.
Hello Oleg,
If all your VCS roots reference the same repository, you can use {build.vcs.number.1} and you're done. Simply because
build.vcs.number.1 = build.vcs.number.2 (and references whole repository number).
Regarding the number of VCS roots. I suppose you can have 2: one for common library (so you can update it in one place)
and another for all your project sources + per build configuration checkout rules (so you can update other VCS settings in one place).
Does it make sense?
Regards,
KIR
KIR,
I tried using {build.vcs.number.1} for a configuration with 2 roots, and TC shows ??? instead of the latest revision number. Are you sure this is supposed to work for roots that use the same repository server, but different folders? Seems to me that TC 4.0 requires the root name as opposed to root number in this case. If I use the root name, then I'm back to my original problem. BTW, {build.vcs.number.1} works if I have a single root.
Yes, checkout rules work too. I used them previously, but it was also posing a management problem because I had to exclude folders depending on the project to reduce the checkout time, and this became difficult with each added project. I would still have to go through all configurations and modify the checkout rules to exclude folders related to other projects. Separate roots really work well for me because instead of excluding folders, I just need to add a specific root for the new configuration.
Anyway, I'd appreciate some help with the first paragraph. Am I missing something or am I seeing a bug or is this by design in TC 4.0?
Thanks,
Oleg.
Hello Oleg,
To get numeric variables you should set system property -Dteamcity.buildVcsNumberCompatibilityMode=true
I've just find out that old references were obsoleted.
But in fact, you can use symbolic names for the reference. If both VCS roots reference the same repository,
the symbolic names for revision number for any of the roots should have the same value.
So {build.vcs.number.root1} = {build.vcs.number.root2} if root1 and root2 reference the same VCS repository.
Hope this helps,
KIR
Hi KIR,
I tried what you suggested to test "{build.vcs.number.root1} = {build.vcs.number.root2}" hypothesis. I believe that I am seeing a bug. Could you please try this kind of configuration:
TeamCity 4.0 (build 8080)
Perforce root1:
//CommonDirectory/SubDirectory1/SubSubDirectory1/... //team-city-agent/CommonDirectory/SubDirectory1/SubSubDirectory1/...//ProjectsDirectory/SomeProject/... //team-city-agent/ProjectsDirectory/SomeProject/...
Perforce root2:
//CommonDirectory/SubDirectory1/someFile.txt //team-city-agent/CommonDirectory/SubDirectory1/someFile.txt
Note that root2 uses a file from a common subdirectory and root1 uses a directory from a common subdirectory.
Build Configuration Version Control Settings page has root1 listed first, root2 listed second.
General Settings has "{build.vcs.number.root1}.{0}" for build number format using literal root names like you suggested.
Step 1: Commit a change to root1 in ProjectsDirectory/SomeProject
Step 2: Commit a change to root2 (common)
After step 1, build number has the same revision from the change you just committed. That's what I want. Great, now do step 2. Build number remains the same as after step 1 whereas it was expected to have the new (latest) revision number. I believe that TC is using the latest revision of directories/files used by root1 (but this might not be the latest for the entire repository) or, more generally, whatever root is listed for build format. It seems like this functionality may be intentional to be able to reflect the state of a particular root in the build number, but it might be misleading if you want to have a single build number, which is why I suggested "max" earlier.
Update: I also tried "{build.vcs.number.root1}.{build.vcs.number.root2}.{0}" to check the revision numbers and confirmed that they are indeed different as I describe above.
Please let me know if you can reproduce this bug, and I will file it in the issue tracker.
Thanks,
Oleg.
Message was edited by: Oleg Gerovich
Hello Oleg,
I've just tried to reproduce the problem with the current TeamCity sources, but it works as expected. The only difference
of my setup from yours was that root1 and root2 had common prefix //depot. Anyway, I got the same VCS build number
for both roots and didn't manage to get different values.
I have a question - do you probably have quiet period enabled for the build configuration? How did you trigger builds in your
tests, manually or via VCS trigger? I suppose there may be situations when returned build number will not be the latest
repository build number, but for typical setup this should not be the case.
Regards,
KIR
Oops, I skipped the "//depot" part from the Perforce mapping. Yes, both root have the same depot like this:
Perforce root1:
//depot/CommonDirectory/SubDirectory1/SubSubDirectory1/... //team-city-agent/depot/CommonDirectory/SubDirectory1/SubSubDirectory1/...
//depot/ProjectsDirectory/SomeProject/... //team-city-agent/depot/ProjectsDirectory/SomeProject/...
Perforce root2:
//depot/CommonDirectory/SubDirectory1/someFile.txt //team-city-agent/depot/CommonDirectory/SubDirectory1/someFile.
Both roots have quiet period set for 60 seconds. Build is triggered by VCS trigger only. The two example steps I gave you were separated by more than 60 seconds (by hours and days, in fact). I guess I have an atypical setup. :-)
Is it possible that the Perforce command you are using to get the revision number restricted to the directories in the root? Something equivalent to "p4 changes -c client" or "p4 change [files]"? If files are specified, then the result is limited to revisions affecting those files. To get the latest number for the repository, you should be doing something equivalent to "p4 counter change" command.
Any ideas of how to work around the issue besides a single VCS root?
I noticed one other thing:
In my configuration with 2 roots, if a changelist is submitted that has changes to both roots, these changes are broken into 2 separate changes in the web UI. Both changes use the same changelist number and comment. This is confusing to the user as they only expect one change to show up to match the single submitted changelist. I'm guessing that changes are checked and reported on a per-root basis. In this case, there should be an attempt to combine the roots prior to all checks are done, especially if the root use TeamCity client mapping option for root definition (I'm talking about Perforce here, but it may apply to other VCS types).
Hello,
This is a separate issue, which is hard to fix, actually.
You're right that changes are bound to VCS roots. So if you want to see a single change for the described situation,
you should use a single VCS root. We can unite changes to one only if they relate to a single VCS root (possibly with some checkout rules).
Regards,
KIR
Hello,
In fact, this is a bug that you get different build numbers for VCS roots which reference the same VCS repository.
I'd appreciate if you describe the problem and expected behaviour in our tracker.
Thanks!
KIR
Looks like Yegor beat me to it in http://jetbrains.net/tracker/issue/TW-4527. :-)