TeamCity NuGet feed HsqlException

Our TeamCity server (latest version 2022.10) is running on Windows Server with a MS SQL Server data store.  At seemingly random occasions, the NuGet feeds start producing errors and are not usable.  Here's what one of them looks like, from teamcity-nuget.log:

[2022-11-10 13:35:52,704]   WARN - o.processor.NuGetErrorCallback - Internal Server Error: jetbrains.buildServer.serverSide.db.UnexpectedDBException: Unexpected exception: SQL error when doing 'Executing  with parameters: [5, 21, System.Threading] caused SQL error: ' while performing SQL query: select e.entry_key, e.build_id, e.data_values from entry e where e.provider_id = ? and cardinality(e.data_values) <= ? and (1=1
and(cardinality(e.data_values) >= 1 and lcase(?) = lcase(e.data_values[1]))

)
order by e.build_id desc, e.entry_key: java.sql.SQLException: org.hsqldb.HsqlException: java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled Java code. Caused by: org.hsqldb.HsqlException: org.hsqldb.HsqlException: java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled Java code (enable debug to see stacktrace)

The URL being retrieved looks like this (sensitive data removed):

https://[our server]/guestAuth/app/nuget/feed/[feed name]/default/v2/FindPackagesById()?id=%27[package name]%27&semVerLevel=2.0.0

This SQL query references tables that are not in the SQL Server database - combined with "HsqlException" it seems like there is still an HSQL database being used somewhere behind the scenes.  

Is there a remedy to this issue?  The only way we can resolve it is to restart the TeamCity service.

 

7 comments
Comment actions Permalink

I wonder if it's related to the issue discussed at https://youtrack.jetbrains.com/issue/TW-57567

I will try the buildsMetadata cache reset next time we have the problem.

0
Comment actions Permalink

Hi! Yes, TW-57567 seems to be related. If the issue reoccurs, before you reset the buildsMetadata cache, would you be able to zip the <TeamCIty data directory>\system\caches\buildsMetadata directory, upload it via https://uploads.jetbrains.com/, and post the upload ID? (edit: it would also be useful to see the teamcity-server.log* and teamcity-nuget.log* files together with buildsMetadata)

And yes, you are correct: HSQLDB is used to store NuGet feed indices

0
Comment actions Permalink

We are still getting this error, and it seems to be happening more frequently.

I uploaded our current logs to 2022_12_19_V5SK2NqrBE33iYgmJRQcJB

It seems like there may be different kinds of errors happening, but the end result is that the NuGet feed is broken.  A note - the TeamCity server is running on Windows Server, and the data directory is on a network fileshare.

 

0
Comment actions Permalink
Thanks, Pete, we will look into it. In the meantime, please check if the client caches on the data directory mounts are disable, as recommended in the 'Disable Network Client Caches on Data Directory Mounts' section of the documentation:
https://www.jetbrains.com/help/teamcity/multinode-setup.html#Disable+Network+Client+Caches+on+Data+Directory+Mounts

Additionally, could you clarify what are the symptoms of the issue from the user's perspective? You mentioned NuGet feeds start producing errors and are not usable. Can you clarify where you see those errors, if anywhere but the server logs? And what exactly do you mean by not usable?
0
Comment actions Permalink
Additionally, it would help to get the debug log entries around the error time. To enable debug logging, go to Administration > Diagnostics > Troubleshooting and select the 'debug-nuget' logging preset. A server restart is not required when you change the preset, but you will need to wait until the errors occur again.
0
Comment actions Permalink
After looking further into the logs, it looks like either the HSQLDB with NuGet index data has been corrupted, or there was a memory issue:

```
java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled Java code. Caused by: org.hsqldb.HsqlException: org.hsqldb.HsqlException: java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled Java code (enable debug to see stacktrace)
```

The HSQLDB corruption can occur, for example, if the application is shut down ungracefully.
To address it, I'd recommend resetting the builds metadata cache following these steps:

1. Stop TeamCity;
1. Delete the `/system/caches/buildsMetadata` folder;
1. Start TeamCity.

Please note that when you do that, it may take a while to reindex the NuGet packages in the existing build artifacts and re-populate the cache. Until then, the lookup of the packages in the built-in feeds may be delayed.

As for the memory-related error above, you might need to increase the TeamCity server's JVM heap size if this error keeps appearing. Anyway, starting with resetting the cache would make the most sense for now.
0
Comment actions Permalink

Thanks for the insight.  In the meantime I have moved the cache folder to a local disk and made the registry changes requested for data folders on a network share.  I will monitor to see if the problem happens again.

0

Please sign in to leave a comment.