Our TeamCity server (latest version 2022.10) is running on Windows Server with a MS SQL Server data store. At seemingly random occasions, the NuGet feeds start producing errors and are not usable. Here's what one of them looks like, from teamcity-nuget.log:
[2022-11-10 13:35:52,704] WARN - o.processor.NuGetErrorCallback - Internal Server Error: jetbrains.buildServer.serverSide.db.UnexpectedDBException: Unexpected exception: SQL error when doing 'Executing with parameters: [5, 21, System.Threading] caused SQL error: ' while performing SQL query: select e.entry_key, e.build_id, e.data_values from entry e where e.provider_id = ? and cardinality(e.data_values) <= ? and (1=1
and(cardinality(e.data_values) >= 1 and lcase(?) = lcase(e.data_values))
order by e.build_id desc, e.entry_key: java.sql.SQLException: org.hsqldb.HsqlException: java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled Java code. Caused by: org.hsqldb.HsqlException: org.hsqldb.HsqlException: java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled Java code (enable debug to see stacktrace)
The URL being retrieved looks like this (sensitive data removed):
https://[our server]/guestAuth/app/nuget/feed/[feed name]/default/v2/FindPackagesById()?id=%27[package name]%27&semVerLevel=2.0.0
This SQL query references tables that are not in the SQL Server database - combined with "HsqlException" it seems like there is still an HSQL database being used somewhere behind the scenes.
Is there a remedy to this issue? The only way we can resolve it is to restart the TeamCity service.