TeamCity remote runs: not applying local changes?

Hi,

I'm able to submit remote runs from IntelliJ that involve changes that clearly break compilation, but do not fail in the remote run. I see the following in the TeamCity 4037 build log:

: Checking for changes (1s)
: Building in c:\TeamCity\buildAgent\work\Server\OpenJPA
: Getting project sources (<1s)
: Found 0 components to load on start
<Maven-related log messages from here on>

I seem to recall seeing a thread / bug about build project structure in IntelliJ not lining up properly with the TeamCity structure, and am wondering if maybe that's what's causing my problems. My project has a directory containing a top-level pom.xml and a bunch of subdirectories, each of which in turn has a pom.xml. These nested pom.xml files are referenced from the top-level one. My IntelliJ project was built by running 'mvn idea:idea' several weeks ago. The build directory in the agent is the top-level of this hierarchy, and contains all of the sub-projects.

Is there any way to debug further what might be causing my changes not to be picked up by the agent?

Thanks,

-Patrick

9 comments
Comment actions Permalink

>I seem to recall seeing a thread / bug about build project structure in
>IntelliJ not lining up properly with the TeamCity >structure, and am
>wondering if maybe that's what's causing my problems.

It is the probelm. IDEA plugin builds file relative paths according to
common parent for all source contents and agent just writes these files to
the build working directory.
--
Olesya Smirnova
JetBrains, Inc
http://www.jetbrains.com
"Develop with pleasure!"


"Patrick Linskey" <no_reply@jetbrains.com> wrote in message
news:32574656.1175827703632.JavaMail.itn@is.intellij.net...

Hi,

>

I'm able to submit remote runs from IntelliJ that involve changes that
clearly break compilation, but do not fail in the remote run. I see the
following in the TeamCity 4037 build log:

>

: Checking for changes (1s)
: Building in c:\TeamCity\buildAgent\work\Server\OpenJPA
: Getting project sources (<1s)
: Found 0 components to load on start
<Maven-related log messages from here on>

>

I seem to recall seeing a thread / bug about build project structure in
IntelliJ not lining up properly with the TeamCity structure, and am
wondering if maybe that's what's causing my problems. My project has a
directory containing a top-level pom.xml and a bunch of subdirectories,
each of which in turn has a pom.xml. These nested pom.xml files are
referenced from the top-level one. My IntelliJ project was built by
running 'mvn idea:idea' several weeks ago. The build directory in the
agent is the top-level of this hierarchy, and contains all of the
sub-projects.

>

Is there any way to debug further what might be causing my changes not to
be picked up by the agent?

>

Thanks,

>

-Patrick



0
Comment actions Permalink

Hello Olesya,

>> I seem to recall seeing a thread / bug about build project structure
>> in IntelliJ not lining up properly with the TeamCity >structure, and
>> am wondering if maybe that's what's causing my problems.
>>
OS> It is the probelm. IDEA plugin builds file relative paths according
OS> to common parent for all source contents and agent just writes these
OS> files to the build working directory.
OS>

I remember trying to play with the paths in all the ways I could think of
(that did not break other structure conventions), but was unable to make
this work.

How does it run for you at JetBrains? Do you have a recommended structure
that would make this piece of functionality work? Say one has a project with
2 modules and wants to take advantage of remote builds, how should the paths
be set up?

Thx,
Andrei


0
Comment actions Permalink

It is the probelm. IDEA plugin builds file relative paths according to
common parent for all source contents and agent just writes these
files to the build working directory.


Sure enough -- during the run with remote changes, I get an extra top-level directory in my view.

I think I figured out what's causing the problem. On disk, I have the following structure:

d:/src/openjpa/main
=> structure checked out from http://svn.apache.org/repos/asf/incubator/openjpa:
openjpa/trunk/]]>

All the modules at this location are mapped into my IntelliJ project.

In TeamCity, my configuration (called OpenJPA) has the following VCS root: http://svn.apache.org/repos/asf/incubator/openjpa/trunk

The build agent checks out to c:/TeamCity/buildAgent/work/Server/OpenJPA, and ends up with the contents of the VCS root.

When the remote changes are loaded onto the build agent, I see a new directory called 'openjpa' that corresponds to the full tree structure from http://svn.apache.org/repos/asf/incubator/openjpa.

I tried restructuring my modules so that the top-level directory was checked out directly from the location in svn, rather via the top-level openjpa dir. This was unsuccessful.

Any thoughts about how to structure things so that TeamCity doesn't try to introduce the extra two levels of hierarchy?

Also, IntelliJ should know quite a bit about the location and status of a particular file in the VCS. It seems like it should be able to send this data to TeamCity (rather than just a jar to expand), so that TeamCity could provide an error if, for example, it receives a modification to an existing file but ends up creating a new file instead of overlaying an existing one. It seems bad that I'm able to run a remote commit that successfully passes through the TeamCity run but contains compile / test errors. I think it'd be better if the TeamCity server erred on the side of caution; I'd rather have false negatives (failures that should have passed) than false positives (commits that should not have happened).

Thanks,

-Patrick

0
Comment actions Permalink

Maybe there should be the way to set up 'root directory' in plugin to
callulate relative paths according to it...

Now this root is common parent for all content roots in project.

Say we have 2 modules and they have content roots:
c:/projects/project/module1
c:/projects/project/module2

Common parent for them is c:/projects/project. So for all changed files
their relative paths will be calculated related to this direcotry and these
relative files will be applied to the working directory on agent. For
example, file
c:/projects/project/module1/src/org/test/Test.java will be copied to:

c:/Agent/work/Server/ProjectName/module1/src/org/test/Test.java (were
c:/Agent is agent installation directory and ProjectName is the name of
TeamCity project)

--
Olesya Smirnova
JetBrains, Inc
http://www.jetbrains.com
"Develop with pleasure!"


"Andrei Oprea" <andrei.oprea@rogers.com> wrote in message
news:6f7635fa2a2cf8c94642ed4612e6@news.jetbrains.com...

Hello Olesya,

>
>>> I seem to recall seeing a thread / bug about build project structure
>>> in IntelliJ not lining up properly with the TeamCity >structure, and
>>> am wondering if maybe that's what's causing my problems.
>>>

OS> It is the probelm. IDEA plugin builds file relative paths according
OS> to common parent for all source contents and agent just writes these
OS> files to the build working directory.
OS>
I remember trying to play with the paths in all the ways I could think of
(that did not break other structure conventions), but was unable to make
this work.
How does it run for you at JetBrains? Do you have a recommended structure
that would make this piece of functionality work? Say one has a project
with 2 modules and wants to take advantage of remote builds, how should
the paths be set up?

>

Thx,
Andrei

>



0
Comment actions Permalink

Yes, currrent solution is not very convenient for users, but we just wanted
to avoid vcs-part-of-team-city-plugin for each vcs.
Seems we have to try to change it somehow...

As a very light workaround we're plunning to allow you to configure 'common
root' for the project

--
Olesya Smirnova
JetBrains, Inc
http://www.jetbrains.com
"Develop with pleasure!"


"Patrick Linskey" <no_reply@jetbrains.com> wrote in message
news:18675122.1175880429458.JavaMail.itn@is.intellij.net...
>> It is the probelm. IDEA plugin builds file relative paths according to
>> common parent for all source contents and agent just writes these
>> files to the build working directory.
>

Sure enough -- during the run with remote changes, I get an extra
top-level directory in my view.

>

I think I figured out what's causing the problem. On disk, I have the
following structure:

>

d:/src/openjpa/main
=> structure checked out from
http://svn.apache.org/repos/asf/incubator/openjpa:
openjpa/trunk/<my modules and source files>

>

All the modules at this location are mapped into my IntelliJ project.

>

In TeamCity, my configuration (called OpenJPA) has the following VCS root:
http://svn.apache.org/repos/asf/incubator/openjpa/trunk

>

The build agent checks out to c:/TeamCity/buildAgent/work/Server/OpenJPA,
and ends up with the contents of the VCS root.

>

When the remote changes are loaded onto the build agent, I see a new
directory called 'openjpa' that corresponds to the full tree structure
from http://svn.apache.org/repos/asf/incubator/openjpa.

>

I tried restructuring my modules so that the top-level directory was
checked out directly from the location in svn, rather via the top-level
openjpa dir. This was unsuccessful.

>

Any thoughts about how to structure things so that TeamCity doesn't try to
introduce the extra two levels of hierarchy?

>

Also, IntelliJ should know quite a bit about the location and status of a
particular file in the VCS. It seems like it should be able to send this
data to TeamCity (rather than just a jar to expand), so that TeamCity
could provide an error if, for example, it receives a modification to an
existing file but ends up creating a new file instead of overlaying an
existing one. It seems bad that I'm able to run a remote commit that
successfully passes through the TeamCity run but contains compile / test
errors. I think it'd be better if the TeamCity server erred on the side of
caution; I'd rather have false negatives (failures that should have
passed) than false positives (commits that should not have happened).

>

Thanks,

>

-Patrick



0
Comment actions Permalink

Hello Olesya,

IDEA's pretty good at identifying the source roots (and define prefixed when
they start somewhere inside the package path). I was expecting some similar
abilities from TC, after all there are the package names that should allow
you to allign the paths correctly.

A way to set the "root directory" would work as well. And if this workaround
can be part of the Agra release, even better. The way it stands now, Agra
will ship with a broken remote run mechanism (at least from what I can tell).


You still haven't metioned how it works for you at jetbrains... Is there
a "recommended" structure, at least?

Best,
Andrei


OS> Maybe there should be the way to set up 'root directory' in plugin
OS> to callulate relative paths according to it...
OS>
OS> Now this root is common parent for all content roots in project.
OS>
OS> Say we have 2 modules and they have content roots:
OS> c:/projects/project/module1 c:/projects/project/module2
OS>
OS> Common parent for them is c:/projects/project. So for all changed
OS> files
OS> their relative paths will be calculated related to this direcotry
OS> and these
OS> relative files will be applied to the working directory on agent.
OS> For
OS> example, file
OS> c:/projects/project/module1/src/org/test/Test.java will be copied
OS> to:
OS> c:/Agent/work/Server/ProjectName/module1/src/org/test/Test.java
OS> (were c:/Agent is agent installation directory and ProjectName is
OS> the name of TeamCity project)
OS>


0
Comment actions Permalink

All our projects have the same structure for the TeamCity project as for
project in working directory (on developer's computer). The common root is
directory which on agent is working one.

This will be fixed in the minor (bug fix update) version (I cannot make such
a big change in release)


--
Olesya Smirnova
JetBrains, Inc
http://www.jetbrains.com
"Develop with pleasure!"


"Andrei Oprea" <andrei.oprea@rogers.com> wrote in message
news:a20f25435b44d8c948a29534170e@news.intellij.net...

Hello Olesya,

>

IDEA's pretty good at identifying the source roots (and define prefixed
when they start somewhere inside the package path). I was expecting some
similar abilities from TC, after all there are the package names that
should allow you to allign the paths correctly.

>

A way to set the "root directory" would work as well. And if this
workaround can be part of the Agra release, even better. The way it stands
now, Agra will ship with a broken remote run mechanism (at least from what
I can tell).

>

You still haven't metioned how it works for you at jetbrains... Is there a
"recommended" structure, at least?

>

Best,
Andrei

>
>

OS> Maybe there should be the way to set up 'root directory' in plugin
OS> to callulate relative paths according to it...
OS> OS> Now this root is common parent for all content roots in project.
OS> OS> Say we have 2 modules and they have content roots:
OS> c:/projects/project/module1 c:/projects/project/module2
OS> OS> Common parent for them is c:/projects/project. So for all changed
OS> files
OS> their relative paths will be calculated related to this direcotry
OS> and these
OS> relative files will be applied to the working directory on agent.
OS> For
OS> example, file
OS> c:/projects/project/module1/src/org/test/Test.java will be copied
OS> to:
OS> c:/Agent/work/Server/ProjectName/module1/src/org/test/Test.java
OS> (were c:/Agent is agent installation directory and ProjectName is
OS> the name of TeamCity project)
OS>



0
Comment actions Permalink

Hello Olesya,

OS> All our projects have the same structure for the TeamCity project as
OS> for project in working directory (on developer's computer). The
OS> common root is directory which on agent is working one.

I cannot change the paths in our project, but I'll try to define some symlinks
to see if I can fake some allignment in order to get this working.

OS>
OS> This will be fixed in the minor (bug fix update) version (I cannot
OS> make such a big change in release)
OS>

Too bad. You might want to document this somehow (and suggest a way to set
up the directories in order to have this running). This is a nasty limitation
for a rather important piece of functionality in TeamCity, I doubt there's
just a handfull of us with non-alligned paths (and I would have set the paths
accordingly in the beginning, had I known about such an issue).

Best,
Andrei


0
Comment actions Permalink

All our projects have the same structure for the
TeamCity project as for project in working directory (on developer's
computer). The common root is directory which on agent is working one.


So I don't really understand how my directories are unaligned. First, the modules in my project got to where they are via a simple VCS checkout (albeit one done on the command line, not via IntelliJ), and IntelliJ properly deals with other VCS operations. Second, my observations indicate that the changed files end up in a directory on the build agent corresponding to a structure one level above the VCS root.

The project that I'm seeing this behavior in is open-source; let me know if you're interested in steps to reproduce the problem.

This will be fixed in the minor (bug fix update)
version (I cannot make such a big change in release)


Do you have any rough ETA on when a patch / preview of the fix will be available?

A couple general comments:

- I would have expected your implementation to generate a diff (or a diff per VCS) on the client side, and then send the patches to the build agent for application. Based on the observed behavior (especially the lack of an error when the patch application should have failed, and the expanded output on disk), it would seem that this is not the case. You might want to consider a patch-based approach in a future release; the concepts involved in creating and applying diffs are as old as time itself, and could probably help with all sorts of unexpected scenarios.

- From a resolution standpoint, for my particular use case, something similar to the -p argument to patch (http://www.rt.com/man/patch.1.html) would be sufficient -- I think that if I could specify -p2 or maybe -p1, I'd be in good shape. This option basically tells patch to remove n directory levels from the transmitted diff data. I'm not sure if my experience is indicative of the general problem, or just a corner case, of course.

- I think that better detection of the problem would be a really good thing. As a user, I'd rather see an error saying that my project structure is unsupported than see untested and invalid changes make their way into the VCS. This is something that a diff-based approach would trivialize, but you could probably put together something without fully moving to a diff approach.

Thanks,

-Patrick

0

Please sign in to leave a comment.