Unable to start TeamCity with systemd
Answered
Hi,
I am trying to start teamcity with systemd. When I try it by doing
/opt/jetbrains/teamcity/bin/runAll.sh start
Then there is no issue and it works fine.
[root@ip-192-168-0-10 system]# /opt/jetbrains/teamcity/bin/runAll.sh start
Spawning TeamCity restarter in separate process
TeamCity restarter running with PID 257045
Starting TeamCity build agent...
Java executable is found: '/usr/java/latest/bin/java'
Starting TeamCity Build Agent Launcher...
Agent home directory is /opt/jetbrains/teamcity/buildAgent
Agent Launcher Java runtime version is 1.8
Lock file: /opt/jetbrains/teamcity/buildAgent/logs/buildAgent.properties.lock
Using no lock
Done [257644], see log at /opt/jetbrains/teamcity/buildAgent/logs/teamcity-agent.log
[root@ip-192-168-0-10 system]#
However if I try to start it with systemd then it fails. My systemd file looks like this
[Unit]
Description=TeamCity Server
Documentation=https://unix.stackexchange.com/a/316369/5132
After=network.target
[Install]
WantedBy=multi-user.target
[Service]
Type=simple
User=teamcity
Group=teamcity
Environment=TEAMCITY_DATA_PATH=/var/jetbrains/application-data/teamcity/
Environment=TEAMCITY_DIR=/opt/jetbrains/teamcity/
Environment=TEAMCITY_SERVER_OPTS=-Djava.awt.headless=true
#Environment=CATALINA_PID=/opt/jetbrains/teamcity/bin/../logs/teamcity.pid
Environment=JAVA_HOME=/usr/java/latest
SyslogIdentifier=teamcity_server
#PIDFile=/var/run/jetbrains/teamcity.pid
ExecStart=/opt/jetbrains/teamcity/bin/runAll.sh start
ExecStop=/opt/jetbrains/teamcity/bin/runAll.sh stop
PrivateTmp=yes
RestartSec=5
Restart=on-failure
TimeoutStartSec=900
When I do the above I see nothing in the logs in <TEAMCITY DIR>/logs/. When I do journalctl -f I see in the logs:
Mar 23 08:10:51 ip-192-168-0-10 systemd[1]: Started TeamCity Server.
Mar 23 08:10:51 ip-192-168-0-10 teamcity_server[270700]: Spawning TeamCity restarter in separate process
Mar 23 08:10:51 ip-192-168-0-10 teamcity_server[270700]: TeamCity restarter running with PID 270707
Mar 23 08:10:51 ip-192-168-0-10 teamcity_server[270700]: Starting TeamCity build agent...
Mar 23 08:10:54 ip-192-168-0-10 teamcity_server[270700]: Java executable is found: '/usr/java/latest/bin/java'
Mar 23 08:10:54 ip-192-168-0-10 teamcity_server[270700]: Starting TeamCity Build Agent Launcher...
Mar 23 08:10:54 ip-192-168-0-10 teamcity_server[270700]: Agent home directory is /opt/jetbrains/teamcity/buildAgent
Mar 23 08:10:54 ip-192-168-0-10 teamcity_server[270700]: Agent Launcher Java runtime version is 1.8
Mar 23 08:10:54 ip-192-168-0-10 teamcity_server[270700]: Lock file: /opt/jetbrains/teamcity/buildAgent/logs/buildAgent.properties.lock
Mar 23 08:10:54 ip-192-168-0-10 teamcity_server[270700]: Using no lock
Mar 23 08:10:54 ip-192-168-0-10 teamcity_server[270700]: Done [271309], see log at /opt/jetbrains/teamcity/buildAgent/logs/teamcity-agent.log
Mar 23 08:10:55 ip-192-168-0-10 teamcity_server[271311]: Removing lock file so server won't automatically restart
Mar 23 08:10:56 ip-192-168-0-10 teamcity_server[271311]: Java executable is found: '/usr/java/latest/bin/java'
Mar 23 08:10:56 ip-192-168-0-10 teamcity_server[271311]: PID file found but either no matching process was found or the current user does not have permission to stop the process. Stop aborted.
Mar 23 08:10:56 ip-192-168-0-10 teamcity_server[271311]: Stopping TeamCity build agent...
Mar 23 08:11:03 ip-192-168-0-10 teamcity_server[271311]: Java executable is found: '/usr/java/latest/bin/java'
Mar 23 08:11:03 ip-192-168-0-10 teamcity_server[271311]: Starting TeamCity Build Agent Launcher...
Mar 23 08:11:03 ip-192-168-0-10 teamcity_server[271311]: Agent home directory is /opt/jetbrains/teamcity/buildAgent
Mar 23 08:11:04 ip-192-168-0-10 teamcity_server[271311]: Received stop command from console.
Mar 23 08:11:04 ip-192-168-0-10 teamcity_server[271311]: Unable to locate agent port file: /opt/jetbrains/teamcity/buildAgent/logs/buildAgent.xmlRpcPort
Mar 23 08:11:04 ip-192-168-0-10 teamcity_server[271311]: Agent is not running?
Mar 23 08:11:04 ip-192-168-0-10 teamcity_server[271311]: Sending agent shutdown command to: http://localhost:9090
Mar 23 08:11:04 ip-192-168-0-10 teamcity_server[271311]: Failed to shutdown agent gracefully: Connection refused (Connection refused)
Mar 23 08:11:04 ip-192-168-0-10 teamcity_server[271311]: Cannot stop agent gracefully, you can try to kill agent by './agent.sh stop kill' command
Mar 23 08:11:04 ip-192-168-0-10 systemd[1]: teamcity.service: Succeeded.
If I change the systemd script to be
ExecStart=/opt/jetbrains/teamcity/bin/teamcity-server.sh start
ExecStop=/opt/jetbrains/teamcity/bin/teamcity-server.sh stop
Then I see in the logs over and over
Mar 23 08:15:45 ip-192-168-0-10 systemd[1]: Started TeamCity Server.
Mar 23 08:15:45 ip-192-168-0-10 teamcity_server[272055]: Spawning TeamCity restarter in separate process
Mar 23 08:15:45 ip-192-168-0-10 teamcity_server[272055]: TeamCity restarter running with PID 272059
Mar 23 08:15:45 ip-192-168-0-10 teamcity_server[272060]: Java executable is found: '/usr/java/latest/bin/java'
Mar 23 08:15:45 ip-192-168-0-10 teamcity_server[272060]: PID file found but either no matching process was found or the current user does not have permission to stop the process. Stop aborted.
Mar 23 08:15:45 ip-192-168-0-10 systemd[1]: teamcity.service: Control process exited, code=exited status=1
Mar 23 08:15:45 ip-192-168-0-10 systemd[1]: teamcity.service: Failed with result 'exit-code'.
Mar 23 08:15:50 ip-192-168-0-10 systemd[1]: teamcity.service: Service RestartSec=5s expired, scheduling restart.
Mar 23 08:15:50 ip-192-168-0-10 systemd[1]: teamcity.service: Scheduled restart job, restart counter is at 1.
Mar 23 08:15:50 ip-192-168-0-10 systemd[1]: Stopped TeamCity Server.
Mar 23 08:15:50 ip-192-168-0-10 systemd[1]: Started TeamCity Server.
Mar 23 08:15:50 ip-192-168-0-10 teamcity_server[272373]: Spawning TeamCity restarter in separate process
Mar 23 08:15:50 ip-192-168-0-10 teamcity_server[272373]: TeamCity restarter running with PID 272377
Mar 23 08:15:50 ip-192-168-0-10 teamcity_server[272378]: Java executable is found: '/usr/java/latest/bin/java'
Mar 23 08:15:50 ip-192-168-0-10 teamcity_server[272378]: PID file found but either no matching process was found or the current user does not have permission to stop the process. Stop aborted.
Mar 23 08:15:50 ip-192-168-0-10 systemd[1]: teamcity.service: Control process exited, code=exited status=1
Mar 23 08:15:51 ip-192-168-0-10 systemd[1]: teamcity.service: Failed with result 'exit-code'.
Mar 23 08:15:56 ip-192-168-0-10 systemd[1]: teamcity.service: Service RestartSec=5s expired, scheduling restart.
Mar 23 08:15:56 ip-192-168-0-10 systemd[1]: teamcity.service: Scheduled restart job, restart counter is at 2.
Mar 23 08:15:56 ip-192-168-0-10 systemd[1]: Stopped TeamCity Server.
Mar 23 08:15:56 ip-192-168-0-10 systemd[1]: Started TeamCity Server.
Mar 23 08:15:56 ip-192-168-0-10 teamcity_server[272685]: Spawning TeamCity restarter in separate process
Mar 23 08:15:56 ip-192-168-0-10 teamcity_server[272685]: TeamCity restarter running with PID 272689
Mar 23 08:15:56 ip-192-168-0-10 teamcity_server[272690]: Java executable is found: '/usr/java/latest/bin/java'
Mar 23 08:15:56 ip-192-168-0-10 teamcity_server[272690]: PID file found but either no matching process was found or the current user does not have permission to stop the process. Stop aborted.
Mar 23 08:15:56 ip-192-168-0-10 systemd[1]: teamcity.service: Control process exited, code=exited status=1
Mar 23 08:15:56 ip-192-168-0-10 systemd[1]: teamcity.service: Failed with result 'exit-code'.
^C
I am assuming there is an issue with some sort of run time variable. Any idea what I am doing wrong?
Please sign in to leave a comment.
The runAll.sh and teamcity-server.sh scripts will spawn additional processes. Try setting the Service Type to "Forking". https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/configuring_basic_system_settings/managing-services-with-systemd_configuring-basic-system-settings#unit-file-structure_working-with-systemd-unit-files
When doing forking it hangs indefinably.
After going through this myself for teamcity agents only. I've arrived at the following working systemd service configuration:
This will effectively launch the agent and makes systemd guess the main pid.
It will also restart the agent whenever it exits, unless it is performing an auto upgrade (exit code 143).
As a side note this will make the agent restart if its child process is killed (which makes the main process exit with code 0)
Manuel Torrinha That's just for agents. I am trying to start the server as well. When I do that using forking it starts however systemd then seems to hang indefinitely.
What do you get for journalctl output? Are any logs created in <teamcity installation directory>/logs? Can you confirm what operating system are you using?
We don't have any official documentation covering this, but I am able to run it on Ubuntu 20.04 with Type=forking. I have also successfully run it in the past as forking in CentOS. Here is my Ubuntu teamcity.service for reference:
Please note, we have had some reports of issues with automatic upgrades when running as a systemd service. Refer to https://youtrack.jetbrains.com/issue/TW-58706 for more information regarding that issue.
Eric,
I am using Oracle Linux 8.3 (which is a RHEL clone). Below is what I have in my startup script
At first I thought it was not starting but it actually was. I am not sure if the reason is that it is working now is because I am only launching the server vs run-all or something else.
EDIT: I decided to dig a bit further to see what was broken and I was not able to find anything. I then went ahead and added a service for the agent so that would work as well. In doing so I changed that the pid should match the server so that the pid file is
When I did the above the systemd scripts seemed to have hung indefinitely. As it was hanging as soon as I did:
For some reason as the owner of the DIR (user teamcity) I was NOT able to copy the file. I had to do it as root. As soon as I did it showed the service as running. Switching it back to
Seems to have fixed the issue and allowed me to start the service. In the logs I see (when it works)
The same went for the agent. Setting the pid file to anything but buildAgent.pid (e.g. if I set it to /opt/jetbrains/teamcity/buildAgent/logs/buildAgent-foo.pid) would cause systemd to hang indefinitely. Is this a bug?
EDIT2: In the process of troubleshooting I set selinux to permissive thinking that was part of the issue and in the logs I saw
Apr 07 07:16:57 ip-192-168-0-10.internal setroubleshoot[130923]: SELinux is preventing systemd from unlink access on the file buildAgent.pid. For complete SELinux messages run: sealert -l 80daf6b3-bf5b-462f-9816-6a4c15e6fc8f
The above only happens when the agent is stopped so it's not the end of the world but I would like to have SELINUX running. Who creates the PID? Is it systemd or teamcity? Below is what selinux is logging:
So the server runs now? For what it's worth, I can substitute the ExecStart/Stop commands with runAll.sh and it will still run.
I'm not sure why you're trying to copy the pid files. They are used to store the pid of the running processes and should be unique between the agent and server. The pid files are automatically generated by the teamcity-server.sh script and will be removed when the service is stopped. The pid stored in the pid file is used to monitor and stop the service when the "teamcity-server.sh stop" command is issued.
For the build agent, you need the service to monitor the pid file for the build agent. See my working example below:
Hi,
I was copying the pids as it seems that systemd does not think the process is running. I was under the impression that systemd was creating the pid file. This is why systemd seemed to have hung. I looked at the documentation here https://www.freedesktop.org/software/systemd/man/systemd.service.html and it seems that systdemd may try to remove the pid. Does systemd delete it the pid or does the teamcity agent try to delete it? I am trying to determine where is the issue with selinux and why it's preventing the deletion of the pid file.
The pid file is created by the application when it is launched. Systemd will only read the PID after the start-up of the service, it doesn't create it or make any changes to it. It is important to make sure the teamcity.pid and buildAgent.pid files are not present before attempting to start the services with systemd. If they do exist, make sure the processes are not actually already running before removing them.
It is true that systemd will remove the pid after the service has stopped, but only in cases where it hasn't already been removed as some kind of fail-safe. In the case of TeamCity and build agents, the pid file should be removed by the application rather than systemd.
Last night, I set up a VM with OracleLinux 8.3 and was able to use the same .service files as my Ubuntu server. So I can confirm at least it is possible for this to work. I'm not sure what is the cause of the hanging for you. It does take a few moments for the service to finish starting up, maybe giving the appearance of being hung.
The one minute hang is due to logic in teamcity-server-restarter.sh. Note that "run" case is used in restarter script even if you pass "start" to teamcity-server.sh. Looks quite hacky, but probably for good reasons as comments suggest...