TeamCity CSRF error on login
I've moved my TeamCity server URL to a private endpoint. However, now I can no longer login through my reverse proxy (but it still works when connecting directly to TeamCity). I see the following error on the client:
Error accessing server. Check for proxy and server errors.
HTTP status: 403
Response text: 403 Forbidden: Responding with 403 status code due to failed CSRF check: authenticated POST request is made, but neither tc-csrf-token parameter nor X-TC-CSRF-Token header are provided. Consider deleting existing cookies associated with TeamCity domain and using Bearer Token authentication for you requests (such requests don't return session cookies).
I've added the debug-auth preset, and I see in the auth logs that the mentioned header is being received by TeamCity, and with the correct value:
[2024-06-20 17:59:26,830] DEBUG [no auth; http-nio-8111-exec-11] - Creating session E70A8EF58B... (created: 2024-06-20 17:59:26.830, timeout: 3600s)
[2024-06-20 17:59:26,830] DEBUG [no auth; http-nio-8111-exec-11] - Generated CSRF token 28f18984-6ea1-4b26-8f15-971c3d69286b for session E70A8EF58B2DAB5D4D5221EA79E385A8
[2024-06-20 17:59:27,171] DEBUG [5e75fc9'; Scheduled executor 3] - Expired tokens removed []
[2024-06-20 17:59:29,688] INFO [ no auth; http-nio-8111-exec-8] - Responding with 403 status code due to failed CSRF check: authenticated POST request is made, but neither tc-csrf-token parameter nor X-TC-CSRF-Token header are provided. Consider deleting existing cookies associated with TeamCity domain and using Bearer Token authentication for you requests (such requests don't return session cookies)., responding with 403 error. Request details: POST '/loginSubmit.html?username=raman&remember=true&_remember=&submitLogin=Log%20in&publicKey=x&encryptedPassword=x...', from client 10.65.0.2 (127.0.0.1:35312), client: AJAX-Prototype, no auth
[2024-06-20 17:59:29,688] DEBUG [ no auth; http-nio-8111-exec-8] -
x-forwarded-host: ci.internal.example.com
x-forwarded-for: 10.65.0.2
x-forwarded-proto: http
host: localhost:8111
content-length: 602
pragma: no-cache
cache-control: no-cache
accept: text/javascript, text/html, application/xml, text/xml, */*
x-teamcity-client: AJAX-Prototype
x-prototype-version: 1.7.3
x-requested-with: XMLHttpRequest
x-tc-csrf-token: 28f18984-6ea1-4b26-8f15-971c3d69286b
user-agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36
content-type: application/x-www-form-urlencoded; charset=UTF-8
origin: http://ci.internal.example.com
referer: http://ci.internal.example.com/login.html
accept-encoding: gzip, deflate
accept-language: en-CA,en;q=0.9,en-US;q=0.8
cookie: __test=1
tc-csrf-token parameter: null
rest.cors.origins=
[2024-06-20 17:59:29,689] INFO [ no auth; http-nio-8111-exec-8] - Failed user login attempt for POST '/loginSubmit.html?username=raman&remember=true&_remember=&submitLogin=Log%20in&publicKey=x&encryptedPassword=x...', from client 10.65.0.2 (127.0.0.1:35312), client: AJAX-Prototype, no auth: java.lang.Exception: Bad CSRF token on login page
java.lang.Exception: Bad CSRF token on login page
at jetbrains.buildServer.controllers.login.LoginSubmitController.doPost(LoginSubmitController.java:20)
at jetbrains.buildServer.controllers.BaseFormXmlController$1.handleRequest(BaseFormXmlController.java:55)
at jetbrains.buildServer.controllers.AjaxRequestProcessor.processRequest(AjaxRequestProcessor.java:48)
at jetbrains.buildServer.controllers.BaseFormXmlController.doHandle(BaseFormXmlController.java:52)
at jetbrains.buildServer.controllers.BaseController.handleRequestInternal(BaseController.java:113)
...
Can you please help me understand why this is not working?
Please sign in to leave a comment.
Hi Raman,
If you are now unable to log in through your reverse proxy, consider the following:
1. To narrow down this issue, could you please temporarily disable CSRF protection by setting the internal property
teamcity.csrf.origin.check.enabled=logOnly
to check it?2. What type of reverse proxy are you using? Ensure there are no issues with your reverse proxy configuration.
3. If you are using NGINX, you need to check the `X-Forwarded-Host` header, as it is related to the TeamCity CSRF check.
proxy_set_header X-Forwarded-Host $http_host; # Necessary for proper absolute redirects and TeamCity CSRF check
For more detailed information, please refer to Configuring Proxy Server (https://www.jetbrains.com/help/teamcity/configuring-proxy-server.html#NGINX) in the TeamCity documentation.
Thank you for your reply Tom Sun .
1. With that internal property, the error is different. I now see the following error on the frontend:
Clear the browser cookies or restart the browser to log in.
(And of course doing so makes no difference)
And in the auth log I see this:
Given this error, it seems maybe the problem is that I'm accessing the server over http and not https? Since the server is now on a private network, the nginx reverse proxy now listens on http port 80 rather than https port 443.
2. Nginx. The configuration looks good to me.
3. Yes, `X-Forwarded-Host` is set as documented. I didn't specify this because I gave the output logs from the auth log in the OP, and in those logs you can see the `X-Forwarded-Host` header is set and has the correct value.
After configuring the proxy, please make sure that you have changed the Server URL value in TeamCity Global Settings to the proxy URL.
Could you please share the NGINX config file? Files can be uploaded via https://uploads.jetbrains.com/. Please let us know the exact id after the upload.
> After configuring the proxy, please make sure that you have changed the Server URL value in TeamCity Global Settings to the proxy URL.
Tom Sun Yes, I had already done that. The server URL value matches the x-forwarded-host.
> Could you please share the NGINX config file? Files can be uploaded via https://uploads.jetbrains.com/. Please let us know the exact id after the upload.
I have uploaded the output of `nginx -T` to:
Upload id: 2024_06_21_wA8aWMmuywSCkX2sg9ubDq (file: nginx-config.txt)
The domain name was elided in the posts above, but the config you are looking for is server_name starting with “ci” listening on port 80. This server is not accessible from the Internet, you will get a 403 if trying to access it directly. Only the `/app/hooks/` endpoint is exposed to the Internet.
Thanks!
Since this is an internal server, I've tried bypassing the reverse proxy internally and logging in directly to the TeamCity port 8111. I've updated the Server URL in global settings to match.
Logging in to http://localhost:8111 works (via ssh port tunneling) but http://ci.internal.example.com:8111 fails with the same errors previously reported, including the CSRF error when CSRF is enabled again. Since no reverse proxy is involved in the connection to http://ci.internal.example.com:8111, that seems to prove the reverse proxy is not the issue here.
I see this error on the console log for the login.html page which I do not see when using localhost:8111:
It definitely has something to do with TeamCity not allowing requests from http except over localhost – when I issue a certificate for the internal server and connect to it over https through the reverse proxy, everything works fine. When connecting to it over http, TeamCity denies the request.
See the message I posted above:
I'll probably just leave https enabled, though I suspect I could “fool” TeamCity by changing my reverse proxy to set
x-forwarded-proto https
so that TeamCity thinks the proxy request was secure. But it would still be nice to understand why this is happening.Sorry for the late response. Glad to know that you have fixed the issue.
HTTPS is not mandatory. If possible, could you please try using a different port? You can use a port that is not occupied instead of port 80.
Tom Sun I figured out the problem – my TeamCity Tomcat configuration `server.xml` contained “secure=true” and “scheme=https” for the Connector. This is why switching to http was not working.