Constant: WARN - jetbrains.buildServer.SERVER - Unable to update running builds collection

Answered

Constantly getting that warning in Teamcity logs.
Any one knows where could be an issue?

Running: TeamCity Enterprise 2020.1 (build 78475)

[2020-06-17 16:50:12,908] WARN - jetbrains.buildServer.SERVER - Unable to update running builds collection:
java.lang.IllegalArgumentException: Argument for @NotNull parameter 'removeFromQueueTime' of jetbrains/buildServer/serverSide/impl/history/RunningBuildData.<init> must not be null
at jetbrains.buildServer.serverSide.impl.history.RunningBuildData.$$$reportNull$$$0(RunningBuildData.java)
at jetbrains.buildServer.serverSide.impl.history.RunningBuildData.<init>(RunningBuildData.java)
at jetbrains.buildServer.serverSide.impl.RunningBuildsCollectionImpl.loadRunningBuild(RunningBuildsCollectionImpl.java:130)
at jetbrains.buildServer.serverSide.impl.RunningBuildsCollectionImpl.lambda$null$13(RunningBuildsCollectionImpl.java:31)
at jetbrains.buildServer.serverSide.db.queries.GenericQuery.processResultSet(GenericQuery.java:380)
at jetbrains.buildServer.serverSide.db.queries.GenericQuery.doExecuteQuery(GenericQuery.java:367)
at jetbrains.buildServer.serverSide.db.queries.GenericQuery.access$500(GenericQuery.java:27)
at jetbrains.buildServer.serverSide.db.queries.GenericQuery$7.action(GenericQuery.java:339)
at jetbrains.buildServer.serverSide.db.queries.GenericQuery.underPreparedStatement(GenericQuery.java:409)
at jetbrains.buildServer.serverSide.db.queries.GenericQuery.doExecuteQuery(GenericQuery.java:336)
at jetbrains.buildServer.serverSide.db.queries.GenericQuery.execute(GenericQuery.java:97)
at jetbrains.buildServer.serverSide.impl.RunningBuildsCollectionImpl.lambda$fetchNewRunningBuilds$14(RunningBuildsCollectionImpl.java:137)
at jetbrains.buildServer.serverSide.db.DBFacade$3.run(DBFacade.java:223)
at jetbrains.buildServer.serverSide.db.DBFacade$3.run(DBFacade.java:221)
at jetbrains.buildServer.serverSide.db.DBFacade.withDB(DBFacade.java:242)
at jetbrains.buildServer.serverSide.db.DBFacade.withDB(DBFacade.java:221)
at jetbrains.buildServer.serverSide.impl.RunningBuildsCollectionImpl.fetchNewRunningBuilds(RunningBuildsCollectionImpl.java:175)
at jetbrains.buildServer.serverSide.impl.RunningBuildsCollectionImpl.lambda$updateRunningBuilds$12(RunningBuildsCollectionImpl.java:179)
at jetbrains.buildServer.serverSide.impl.RunningBuildsCollectionImpl.WRITE_LOCK(RunningBuildsCollectionImpl.java:125)
at jetbrains.buildServer.serverSide.impl.RunningBuildsCollectionImpl.updateRunningBuilds(RunningBuildsCollectionImpl.java:9)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at jetbrains.buildServer.util.executors.TeamCityScheduledThreadPoolExecutor$RunnableScheduledFutureWrapper.run(TeamCityScheduledThreadPoolExecutor.java:85)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

4 comments
Comment actions Permalink

Hi Alexey,

 

that is quite a strange error, we haven't seen it in a number of large installs including our own. We actually don't have a single reference of this issue elsewhere in our database, so I'd like to ask you to add some debug information:

-Which db type and version you are using.

-Which version you upgraded from if this started after an upgrade (otherwise, when it started)

-The result of the query select * from running

0
Comment actions Permalink

We start seeing that WARNING since a long time.
So I can not say after which actual update that happened.

I am not sure if it is causing any issues but it is very annoying and filling the logs.

Same happening on secondary node as well.

DB is: Server version: 8.0.18-9 Percona Server

I have a lot of things in running. Here is some of those:
Query result:
| build_id | agent_id | build_type_id | build_start_time_agent | build_start_time_server | is_personal | build_number | requestor | access_code | queued_ag_restr_type_id | queued_ag_restr_id | build_state_id | agent_type_id | build_counter | user_status_text | build_finish_time_server | progress_text | current_path_text | last_build_activity_time |
+----------+----------+---------------+------------------------+-------------------------+-------------+-----------------+--------------------------------------------------------------------------------------------+----------------------------------+-------------------------+--------------------+----------------+---------------+---------------+------------------------------------------+--------------------------+---------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------+
| 418634 | -1 | bt2199 | 1520255833165 | 1520255833165 | 0 | %build.counter% | ##vcsName='jetbrains.git' type='vcs' queueMergingEnabled='false' | NULL | NULL | NULL | 418634 | 0 | 72 | NULL | NULL | NULL | Collecting changes in 1 VCS root: Compute revision for 'OpenSSL_curl': Computed revision: 95a2be9c20f3ea8ecc2fc839c90c6a6f55fa011c | 1520255833165 |
| 597017 | 8 | bt2793 | 1555055436371 | 1555055439106 | 0 | 597017 | ##userId='63' type='user' | FRxFUpZ97HV1xmMFy3OAHEEw5y92uFAb | NULL | NULL | 597017 | 8 | 127 | NULL | NULL | Resolving artifact dependencies | Resolving artifact dependencies: Downloading files from <Customized Desktop Apps / SBK / Win32 MS8b / Win Apps RC (depends on MS8b win32 RC 3 last pinned), build #596579 [id 596579]> for pattern [**/*=>./QtSetup/Apps/
ALL_ASSETS.zip!**=>./QtSetup/Apps/] | 1555055453091 |
| 636819 | 16477 | bt2221 | 1559117761678 | 1559117761680 | 0 | 1974.636819 | ##vcsName='jetbrains.git' type='snapshotDependency' buildTypeId='bt2223' buildId='636831' | ZSguo6uehtJ5btewHEQoi5H2XIAVqvBS | NULL | NULL | 636819 | 438 | 1974 | NULL | NULL | compile-ios | Step 3/5: compile-ios: for: sequential: execute-ios-target: sequential: exec: ▸ Compiling et_ssms_version_remove_checksum_request.cc | 1559118446934 |
| 647873 | 16477 | bt2142 | 1560431346979 | 1560431346984 | 0 | 187.647873 | ##userId='11' type='snapshotDependency' buildTypeId='bt2144' buildId='647877' | GetYbHk9NA0hi8lIKLU8Zt5KLiELoIvE | NULL | NULL | 647873 | 438 | 187 | {build.status.text} cppcheck findings: 0 | 1560431569625 | NULL | Ant JUnit report watcher: Successfully parsed: 2 reports | 1560431569625 |
| 649131 | 16477 | bt3181 | 1560512201971 | 1560512201964 | 0 | 34-267 | ##type='snapshotDependency' userId='4' origin='rest' buildTypeId='bt3186' buildId='649138' | EkddYMM6s3WxXt9gDPTCMQukKsXUTshV | NULL | NULL | 649131 | 438 | 34 | NULL | 1560513468319 | NULL | Publishing artifacts: Artifacts path 'coverage.zip' not found | 1560513468319 |
| 649237 | 8 | bt3230 | 1560514642292 | 1560514645460 | 0 | 649237 | ##userId='64' type='user' | uiJmvnfLeOsmQ5Kcnhe3f2UtaWA1nKIr | NULL | NULL | 649237 | 8 | 232 | NULL | 1560514715178 | NULL | Step 3/3: in directory: C:\TeamCity\work\87c080b0a03309ba | 1560514715178 |

 

 

0
Comment actions Permalink

Hi Alexey,

 

may I ask you to send us the full output for the command as a file using the Submit a request button above? That will open a private ticket where you can attach it in a private form. Our suspicion is that either there is some data corruption at the database level or there is some very strange corner case we haven't found. The stacktrace indicates that there is a null parameter that should not be null at any point, and we have not seen this issue arise in many large installations using mysql.

 

It might be that Percona has some specific issue, but we would need to know the full data set to understand what's going on, since the data that fails is loaded directly from the database.

0
Comment actions Permalink

Hello Denis.
Sure

Submitted csv file with full output

Request #2719461

0

Please sign in to leave a comment.