TeamCity "Remember me" feature and remember_me table Follow
On a large TeamCity version 2019.1 installation, we noticed that the remember_me table has about 250 million records (!), and the on disk usage, as reported by MySQL "SHOW TABLE STATUS" is about 13 GB.
When any user tries to login through the UI, with the "Remember me" checkbox selected (as it is by default) then click the Login button the page looks like it is attempting to login but then it fails after a few minutes, timing out.
If the "Remember me" options is not checked, then users are able to login instantly.
The number of users in the users table is around 4,000.
So, why there are so many entries in the remember_me table? Is the remember_me table related to the "Remember me" option on the login page? Also, if we delete all entries from this table, while the TeamCity process is still running, would have any other impact on the service other than logging out all users from the UI (which is ok for us)?
What is the purpose of the remember_me table and how it works? Thanks.
Please sign in to leave a comment.
Hi, the 'emember_me' table stores cookies related to the "Remember me" option on the login page. You can clean up the table and restart the TeamCity server, it should not have any impact other than users will have to re-login.
Anyway it's weird that the table is so big, do you have any not bundled TeamCity plugins installed on the server?
Hi Dmitry, thanks for the response. Aside from bundled plugins, we have the following external plugins installed: Static UI Extensions (0.4.5), Artifactory (2.6.0), Gerrit Integration (0.2), teamcity-build-limiter-plugin (snapshot-20180802174039), teamcity-disabler-plugin (snapshot-20180802213731), teamcity-overlay-agent (1.0.17) and Phabricator TeamCity Integration (1.0.0).
Also, about 245 million records from the table were for user ID 2 ("demo").
Deleting entries from the table while the service is running would have any side effects other than users having to re-login in the UI?
>Deleting entries from the table while the service is running would have any side effects other than users having to re-login in the UI?
Hi Dmitry, I just cleaned the remember_me table. I created a new table, same structure as remember_me, then inserted selectively some rows in it from the original table, then renamed the tables: "RENAME TABLE remember_me TO remember_me_old; RENAME TABLE remember_me_new TO remember_me". There were no error logs in the teamcity-server.log and the service works OK - I am able to login then logout without issues.
BUT ... now that the remember_me table only has a few thousand entries, the login operation when the "Remember me" checkbox is selected still takes a minute or more, while if the "Remember me" checkbox is unselected it logins almost instantly!
Do you know what else may cause this delay? I see that when the "Remember me" option is checked, the browser sets an extra cookie, but that definitely should not need more than a few milliseconds, so, is there anything else what may cause this large delay?
You need to restart TeamCity Server after deleting rows from the 'remember me' table, otherwise, it doesn't have any effect because the state of the table is read into the memory by the TeamCity Server process on start.
OK, I understand - it is caching the content of the table in memory. We will observe the behavior on the next restart of the process (we do not have one scheduled for now, but will post and update once we will restart it).
Are there any other tables whose content the TeamCity process is caching (preloading) in memory, similar to how it is doing it for the remember_me table? For example: build_data_storage, downloaded_artifacts, test_info, vcs_change, vcs_history? Thank you.
Hi, thanks for the update.
Some of the tables or parts of them can be loaded and cached in memory. It depends on the TeamCity version and can change.
We just had to restart the TeamCity service today due to an unexpected situation; after restart, with only a few thousand entries in the remember_me table, the service started much faster (in less than 2 minutes, compared to about 20 minutes before cleaning the table). And also the login operations in the UI are performed instantly, with the "Remember me" option checked.
So, keeping the number of rows in the remember_me table under control (few thousands versus 250 million) improved the overall experience of the service in a huge way. Thanks for the help.