URGENT. Newer TeamCity Data Directory, older database Format, howcan i restore the TeamCity?

Hallo ,

One particularly complex TeamCity 4.5.5  -> 5.1.4 Upgrade went wrong.

Now i have TeamCity  Data Directory v 5.1.4 and Database from 4.5.5.

There  is no 5.1.4 Database and no TeamCity Data Directory v 4.5.5. I am  basically stack with two valid data structures, one new and one old

What  can i do to either get full 5.1.4 or return to 4.5.5?
- can i  force an update to 5.1.4 data structure
- can i rollback the Data  Directory format
- alternatives?

with kind  regards // Dimitri

6 comments

Dimitri,

I'd suggest the following approach:
1. backup current data that you have so that it is not spoiled even more
2. run clean TeamCity 4.5.5 installation with new data direcotry to initialize it from scratch
3. connect 4.5.5 database to these fresh 4.5.5 data directory (probably with database.properties modificaiton)
4. upgrade this merged state to 5.1.4
5. drop the newly upgraded data directory and put the one you have with real data over it.
6. Check how the server is running.

As there might be data converters that operate both database and configuration's data stored on disk, the result can be not the same as converting both database and configuraiton on disk in a consistent state.

Step 2a can be added to put .BuildServer/config from real 5.1.2 into .BuildServer/config of the fresh data directory. This can make the conversion more consistent, but can also increase probability of failures during the conversion.

I assume you are acquainted with the structure of the data directory (incl. .BuildServer/system/version.dat file) and what is stored in the data directory and database.

And let me note that having a good backup is never an extra thing...

0

UPDATE, we have now running teamcity 5.1.4.

I see two problems in logs and one in Teamcity WEB gui

1
The TC doesn show the logs for running builds anymore :
Short log |         Full log (~0b) |

2
Log says :
[2010-09-16 14:51:39,016]  ERROR - erverSide.search.SearchService - SearchService.enqueueHistory
java.lang.IndexOutOfBoundsException: Index: 51, Size: 24
        at java.util.ArrayList.RangeCheck(ArrayList.java:547)
        at java.util.ArrayList.get(ArrayList.java:322)
        at org.apache.lucene.index.FieldInfos.fieldInfo(FieldInfos.java:288)
        at org.apache.lucene.index.FieldsReader.doc(FieldsReader.java:217)
        at org.apache.lucene.index.SegmentReader.document(SegmentReader.java:948)
        at org.apache.lucene.index.DirectoryReader.document(DirectoryReader.java:506)
        at org.apache.lucene.index.IndexReader.document(IndexReader.java:947)
        at org.apache.lucene.search.IndexSearcher.doc(IndexSearcher.java:152)
        at jetbrains.buildServer.serverSide.search.SearchService.enqueueHistory(SearchService.java:520)
        at jetbrains.buildServer.serverSide.search.BackgroundIndexer.run(BackgroundIndexer.java:32)
        at java.lang.Thread.run(Thread.java:619)

3

Log says:
[2010-09-16 14:43:07,982]   WARN - impl.MavenMetadataProviderImpl - Error generating Maven metadata for build configuration

What are those issues and how they can be fixed?

with kind regards // Dimitri Uwarov


==========================================
OLD, we managed that

Hallo Yegor,

i am now doing the step 4.

The server says:

[2010-09-16 13:07:12,559]  ERROR -   jetbrains.buildServer.SERVER - Server startup exception: jetbrains.buildServer.rootDispatcher.KnownNonRecoverableStartup
Error: Database version mismatch:
Database version mismatch: Configuration version is 280 but Database version is 379.

In DB i see:

+------------+---------------------+
| version_nr | version_time        |
+------------+---------------------+
|          0 | 2010-09-16 01:15:11 |
|        379 | 2010-09-16 01:15:11 |
+------------+---------------------+

and the .BuildServer/system/version.dat contains:

version.com.mysql.jdbc.Driver=280
version.org.hsqldb.jdbcDriver=280

What is the problem? What can i do here?

regards // D. Uwarov

0

Dmitri,

The TC doesn show the logs for running builds anymore :
Short log |         Full log (~0b) |

It's difficult to say what happened without additional information. Please provide build settings.

Log says :
[2010-09-16 14:51:39,016]  ERROR - erverSide.search.SearchService - SearchService.enqueueHistory
java.lang.IndexOutOfBoundsException: Index: 51, Size: 24
        at java.util.ArrayList.RangeCheck(ArrayList.java:547)
        at java.util.ArrayList.get(ArrayList.java:322)
        at org.apache.lucene.index.FieldInfos.fieldInfo(FieldInfos.java:288)
        at org.apache.lucene.index.FieldsReader.doc(FieldsReader.java:217)
        at org.apache.lucene.index.SegmentReader.document(SegmentReader.java:948)
        at org.apache.lucene.index.DirectoryReader.document(DirectoryReader.java:506)
        at org.apache.lucene.index.IndexReader.document(IndexReader.java:947)
        at org.apache.lucene.search.IndexSearcher.doc(IndexSearcher.java:152)
        at jetbrains.buildServer.serverSide.search.SearchService.enqueueHistory(SearchService.java:520)
        at jetbrains.buildServer.serverSide.search.BackgroundIndexer.run(BackgroundIndexer.java:32)
        at java.lang.Thread.run(Thread.java:619)


Try to delete caches folder under <TeamCity Data Directory>/system with server restart. Does it help?

Kind regards,
Marina

0

Hallo Marina,


first, i would like to thank Jetbrains Team for competent and timely answers, which helped us in rather complex TC recovery situation.


I now have solved:

0 - TC Upgrade (huge config and history) 4.5 -> DONE
I applied the solution of Yegor, and it did work

1 - The TC doesnt show the logs for running builds anymore - FIXED
2 - [2010-09-16 14:51:39,016]  ERROR - erverSide.search.SearchService - SearchService.enqueueHistory  - FIXED
3 - [2010-09-16 14:43:07,982]   WARN - impl.MavenMetadataProviderImpl - Error generating Maven metadata for build configuration

- All three fixed through removing the cache directory. After older build are through, new builds start w/o problem

I now face the following NEW problems:
======================================

1)
[2010-09-17 20:20:12,122]   WARN -   jetbrains.buildServer.SERVER - java.net.SocketTimeoutException: connect timed out

2)
[2010-09-17 19:48:03,582]   WARN - facts.ArtifactUploadController - Failed to upload build artifact due to error: java.io.IOException: Destination file [/mnt
/build_logs/teamcity/artifacts/[Projects 2000] Deal/Integration/1212333] already exists and could not be deleted

I also imagine it could be connected, because the failure to transfer may be a result of Net I/O =SocketTimeout

I already tried to restart all the agents but it doesn't seem to help. Please advice?

regards // Dimitri Uwarov

0

Dimitri,

Thank you for issues summary.

>1)
> [2010-09-17 20:20:12,122]   WARN -   jetbrains.buildServer.SERVER - java.net.SocketTimeoutException: connect timed out

It's hard to tell judjing from the single line. Some operation timed out. This can happen because of network issues or issues with responsiveness on the remote end.
Can you please attach server log around the warning? If you do not want to make it public you can send it to us to feedback email accessible from TeamCity footer or file an issue in our tracker and attach an attachment visile only for "teamcity-developers".

> 2)
> [2010-09-17 19:48:03,582]   WARN - facts.ArtifactUploadController - Failed to upload build artifact due to error: java.io.IOException: Destination file [/mnt
> /build_logs/teamcity/artifacts/[Projects 2000] Deal/Integration/1212333] already exists and could not be deleted

Does the directory exist indeed? Can it be deleted manually (please make a backup copy and restore it after the attempt)?
If it cannot be deleted, can you try to figure out why? Permission issues?

Anyway, if you can enable debug log and attach it for us to investigate that could help to figure out what is the root cause.

0

Ok, i would rather use other ways to send you the logs.. I sent you an email now, named exactlyas this topic:
"URGENT. Newer TeamCity Data Directory, older database Format, howcan i restore the TeamCity?"
Have you got the email?

I will continue to post the 'free part' of the questions regarding the upgrdade  in this thread.

0

Please sign in to leave a comment.