Sync users and groups from Active Directory

David,

Only LDAP authentication supports user synchronization.
We would appreciate details on issues you had while trying to set-up LDAP authentication.
I agree that it's not easy to get it right, but it should be doable and provides features that can be worth it.
ANy constructive feedback on ways to improve the experience are welcome.

I'd recommend to get acquainted with instructions, then setup a test TeamCity instance, switch to LDAP auth in main-config.xml and experiment with ldap-config.properties looking into teamcity-server.log and teamcity-ldap.log at the same time.

I will keep a log of the issues that I run into trying to enabled LDAP authentication:

1. I don't know the name of my Active Directory server. I search for "find active directory server", which leads me to this ServerFault post, which has instructions on how to use nslookup to find your Active Directory server. There are a bunch of them, so I pick the first one.
   
2. I have to guess what the remaining part of java.naming.provider.url should be (remember I know very little about how LDAP works or about how Active Directory implements it). The example in the file is DC=EXAMPLE,DC=Com; I am guessing that EXAMPLE should be my domain.
3. Having set the only mandatory setting in ldap-config.properties, I try to login. It fails, with the error, "Incorrect username or password." Except of course that the username and password are correct. I check the server log, which says: "Login for user dnelson failed: Failed to login user 'dnelson' due to authentication error: javax.naming.AuthenticationException: [LDAP: error code 49 - 80090308: LdapErr: DSID-0C0903A9, comment: AcceptSecurityContext error, data 52e, v1db1 ]" Huh?? I search and find this page that basically says that "error code 49" and "data 52e" mean...incorrect username or password. Very helpful.
4. I go back to the LDAP Integration instructions and jump over to "Typical LDAP Configurations". There are three sections: "Basic LDAP Login", "Active Directory With User Details Synchronization", and "Active Directory With Group Synchronization". I feel like what I am trying to do is pretty basic (doesn't everybody?), and I don't want to do synchronization yet; I just want to do authorization for now. So I follow the instructions under "Basic LDAP Login", setting

   teamcity.auth.loginFilter

   and

   teamcity.users.login.capture

   as indicated. Except that the comments indicate that the user will have to enter the domain name along with their username, which is not what I want. So I play around with these properties for a while, trying to find a combination that will allow users to login with just their username. Long story short, nothing works.
5. I go back to the "Typical LDAP Configurations" page and take a closer look at the next two sections. I notice that neither of these even uses

   teamcity.auth.loginFilter

   or

   teamcity.users.login.capture

   ; instead, one uses

   teamcity.users.login.filter

   , and the other uses

   teamcity.auth.formatDN

   . I'm not exactly sure what the loginFilter is doing, but the formatDN is adding a domain to what the user specifies, which I think might work for me. So (having commented out teamcity.auth.loginFilter and teamcity.users.login.capture), I set formatDN to "MYDOMAIN\\$login$". Hurray! Now I am logged in. (Would the loginFilter have worked? Why are the different configurations using two different techniques? When should you use one or the other? Now that I have it working I don't really want to go back and risk breaking it again to find out.)
   
6. Now I want to enable synchronization. Looking at the typical configurations, I notice that both have set what appears to be explicit domain credentials that are used to query Active Directory; and the password is in plain text! I search and find this recent post on this forum where it is confirmed that the username and password is required and it must be in plain text. Not good. I tried leaving out the username and password, since my server is running as a domain user, but I got an error:  "Operations error (Uncategorized exception occured during LDAP processing; nested exception is javax.naming.NamingException: [LDAP: error code 1 - 00000000: LdapErr: DSID-0C090627, comment: In order to perform this operation a successful bind must be completed on the connection., data 0, vece ]')"
   
7. Having set all of the properties indicated by the typical configuration for user synchronization, I open the server log to find another authentication error. Oops - I guess that synchronization doesn't take into account the formatDN property that I already set; I will have to include the domain in the username.
8. Now I open the log and find that synchronization was apparently successful. Great! Except that it says no users were updated, and in fact that there were only 10 users in LDAP, which can't be right. Now what? Well, rooting around in the properties file, and using what little knowledge I have of LDAP and AD, I surmise that TeamCity is looking in the wrong place for the users; I think that is what "teamcity.users.base=CN=users" is doing. But I don't know how our AD is organized, so I don't know where to tell it to look. How can I find out? I search for "active directory browser", and find AD Explorer from Microsoft, which turns out to be incredibly useful (it also gives me the host name of our AD server). Looking in CN=users, I see that there are actually a few more than 10 users, but I assume that some of these are getting filtered out for reasons unknown. Regardless, this is not where the users I care about are located. In fact, they are organized by office, with each office as its own "OU". I don't know if it is possible to list multiple bases for the search, and I am not sure I would even want to, since that feels fragile. But I also don't want to leave the base blank, since I don't know how that will affect performance, and I don't want our AD admin yelling at me. So I pick the office where I am located and use that as the base. Success! 1 user updated, and when I refresh my account settings, I see that my name and email have been imported.
9. Now I want to enable automatic user creation. But I don't want to create every user in my office. I would like to restrict it to members of a specific group, but all I can find about that is an unanswered StackOverflow question. So I decide to restrict it to users whose first name is David; this will include me and the handful of other Davids in our office, and should provide a reasonable test case. I use AD Explorer to figure out what filter to use (again, VERY useful) and save the file, and check the log to confirm that I am seeing the number of LDAP users I expect; looks right. So I flip on createUsers, and check the log again, and...0 new users created. What? It has clearly applied my filter, because it's down to 4 LDAP users, which is correct; so why won't it create them? I search and can't find anything about this. On the Administration->Users and Groups page, it says "User account in the current authentication scheme are created automatically upon login." Does that mean turning on LDAP sync will not create users? But then what is the point of having the createUsers property?
10. On to group synchronization. I set teamcity.options.groups.synchronize=true and set teamcity.groups.base the same as teamcity.users.base. Server log has an error about group mapping being empty. Back to the properties, I notice a mention of ldap-mapping.xml. Find that file in the config, use AD to find the distinguishedName of the Development group, and create the mapping. I also remember the note that groups are not created automatically. So I create a Development group in TeamCity. Check the log...mandatory properties teamcity.groups.filter and teamcity.groups.property.member are not set. Back to the properties, notice that these properties that I had thought were set were commented out. Uncomment them, check the log...success? Check the server, and I have been added to the Development group; but not only that, a new user from that group (with the first name David) has also been created and added to the group. I guess createUser only applies if group synchronization is enabled? Odd.

So I think I have achieved qualified success; except that there is no way my infrastructure team is going to allow me to store AD credentials in plain text on the server, so all of that synchronization work was probably for nothing. Also, I am still restricting users to my office only; if it turns out that only users in the groups I have mapped are created, and it does not create any kind of performance problem on the server, I may be able to just remove the base and have it scan from the root.

I hope I have given you a sense of what an incredibly frustrating process this is. Undoutedly if I knew more about AD and LDAP, it would be easier; but I really don't want to know any more about AD and LDAP than I absolutely have to. That is why I loved the NTLMLoginModule: ONE property; set the default domain and you're done. I don't want to have to jump through all of these hoops. But if I am going to be forced to jump through these hoops, I need some more help than is currently provided by the TeamCity documentation. Some thoughts on what can be improved:

0. Start by understanding that most developers who are standing up a TeamCity server probably don't know much about AD and LDAP; they need to be guided through the process step by step.
1. Have an entire walkthrough specifically for AD. I have to imagine that is a large segment of the LDAP authentication population, and it would be very useful to have AD-specific instructions, rather than have to figure out which parts of the general instructions apply to me, and whether the AD-specific samples are different from the other samples for a reason, or just for the sake of being different.
2. Recommend AD Explorer at the beginning of the AD walkthrough. It's from Microsoft, it's free, and it is invaluable in this process. Reference at any point in the walkthrough where it would be helpful to someone who is not an AD expert.
3. Define every property in ldap-config.properties in the documentation, including whether or not it is required (or conditionally required), its default value if it is not required, its function, and a semantically meaningful example (e.g. the example for java.naming.provider.url should be "ldap://MyLdapHost.MyDomain.com:389/DC=MyDomain,DC=Com" instead of "ldap://example.com:389/DC=Example,DC=Com". "Example" should not be used twice to mean two different things, and it does not provide any context; don't make the user guess). Also call out anything in the example that might be implementation-specific; when dealing with LDAP, there will probably be a lot of this, but it is important information for someone who is not an LDAP expert.
4. In the typical configurations, break out each example by use case. Whether I am using Active Directory is orthogonal to whether I want to enable user or group synchonization; mixing these use cases together in the configuration samples misled me into thinking that the ones for Active Directory did not apply to me, when in fact they did (or at least one of them did).
5. Clarify exactly what createUsers and deleteUsers do. Creating users only during group synchonization is not intuitive.

Beyond documentation, I think there is more that could be done to make this process smoother:

1. By far the best thing that could be done is to allow LDAP configuration through the web UI. Ease of use is a huge selling point for TeamCity, particularly over systems which rely heavily on configuration files for builds such as CruiseControl. Extending that ease of use to LDAP configuration would go a long way toward solidifying that advantage, and reduce the frustration for those of us who have advocated for TeamCity on that basis only to find ourselves back in config file purgatory.
   
2. As much as possible, reduce the number of properties that need to be set, at least for AD. Again, the NTLMLoginModule only needs one property to achieve authentication. Based on what I have seen, I don't see why it couldn't also be that easy for synchonization. Most of the other properties already have sensible defaults, but a lot of them are commented out; and the few that don't could (for example, java.naming.provider.url could be automatically discovered, and java.naming.security.principal and java.naming.security.credentials could be left blank if automatic binding is implemented, which would be even better than not storing the password in plain text; AD explorer does both of these). If the defaults don't work, guide the user to figure out what needs to be changed.
   

Hopefully this was helpful, if you even managed to read this far. If there is any more information I can give you about my experience that would help you improve this feature, please ask.

David

I am frustrated too. I'm just trying to use LDAP but allow 1 system user that is used for automation purposes, that we know the password for, that is not in the LDAP. If there was a UI and a hidden password field then maybe the infrastructure team would allow us to add the user to the LDAP. Or if there was an option to have ldap authentication and then fallback to non-ldap if a user is not in ldap.

+1 for confused admin.