Long delay for initial checkout of Github repos

Hello Ben,
what version of git-plugin and what protocol do you use?

Hi - In the admin section it says: VCS support: Git 13550

We use SSH / Default Private key

Thanks

Ben

Ben,

I tried to reproduce your problem, but did not succeed.
In my test I used 1.3Gb repository from a local network and my ubuntu box.
Clone from a command line took ~ 7 minutes and clone using a TeamCity agent took ~ 10 minutes.
What OS is installed on your agents?
Did you try to use non-amazon agents? If so, was there such a problem on them?

Also check please do you have any submodules in your repository, if you do - what is your submodule checkout policy?

Ben,

Any update after Dmitry's questions?

For our reference, can you please attach the build log (initial checkout-related lines until the build script starts) so that we can analyze the times spent in each phase?

Are you aware of any public repository which has the same issue so that we can try to reproduce it?

Hi dmitry,
OS on the agent is Ubuntu 10.04LTS

I have just today tried the non-amazon agent and it still took 50Mins (which was a little odd.) The repo itself is about 500MB and the memory setting of the box is not changed from the default 384MB, but the important point to note on that is that it didn't reach OOM error (which is what i would expect if the entire repo had to be checked into memory first - thats what i've read from post else where at least) I was starting to think that possibly could it be that Swap space was being used on the box instead of memory maybe (i'm just clutching at straws on this really...)

So yes, now i've tried an Amazon instance and a local server inside our current network and on both it took 50Mins approx to download the whole GIT repsoitory.

Thanks,
Mark

PS - I work with Ben

No submodules to my knowledge, i'm not even sure of what they are to be honest but I assume as I don't know what they are related to then its un-likely we're using them? :-)

The submodules is set in the dropdown to be 'checkout'

Hi Yegor,

Thanks for your contact, I do appreciate it.

As requested, information from a build log...

Mark, today I created a big repo (> 1Gb) on github and tested
clone through ssh on amazon agent running ubuntu. In my case,
both clone from command line and clone done by TeamCity agent
took ~ 6 minutes.

From your logs I see that a working directory of an agent is set
up to /mnt/TeamCity. Did you try to do clone from command line in
this directory? Might it be the mounted disk that slows
everything down?

Also please check that when you do clone from command line you
are really connected to github.

Mark, thank you for investigation.
Could you please clarify: is TeamCity agent, which log you provided, ran on the same EC2 agent or it is your local machine?

On this occasion all logs and tests were carried out on the same instance that was running the TC agent, which is on an EC2 instance. However I have the same 50min checkout time on a server thats internally hosted on our own local network.

Thanks

Mark, check please that directory /home/ubuntu/TeamCity is not mapped onto /mnt/TeamCity. Just in case..

Hi Dmitry,
I can confirm this, one I hadn't created that directory. I'd let the TC agent create the directory I think or i did it myself, but for sure its not pointing at /mnt... and the other thing to point out is the manual git clone command pulls data out into that exact same directory, so i would have noticed the same massive slow down the /mnt directory has if they were linked in some form.

Cheers,
Mark

Mark, I couldn't reproduce it so far..
You did use /home/ubuntu/hotel-service.git for native git and /home/ubuntu/TeamCity/hotel-service for TeamCity.
For the fair test, please try make clone using TeamCity into /home/ubuntu/hotel-service.git.

Hi Dmitry,

Sorry my bad, i thought I had used the same folder, though it shouldn't have made any difference. I have deleted the hotel-service.git directory of course before doing the checkout because that would not be the test I want.. I guess the below shows the clocks are slightly out on server to agent, but I can't see that having any impact? Still we can see, the same slow checkout occurred.

Hello Mark,
I still cannot reproduce your problem.
What if we add option for you to run shallow clone on agents until we have a better solution?
It will work if you do not need entire repository history during the build.
It will save about half of the repository size, so clone should be twice as faster.
What do you think?

Hi Dmitry,

Is there really nothing that can be done. There must be a problem some where, mis-configuration or something? I don't understand why its working perfectly for you, and yet regardless if I try a Cloud machine running an TC agent vs a machine in house running a TC agent, the times are very silimarly long.

Is there no debugging that can be turned on? Extra Logging, settings to tweek?

We don't really want to go down the route of creating scripts etc, and shallow check-out although it may work and make a difference, its still possibly going to be slower than it should be. It something I will try however, if there's some guidance/notes you could provide for me to set that up, it would be great. We don't really need all the histories of the project in the repository (I don't think), but in any case I can try this for now just to see if it makes a difference.

Thanks,
Mark

Mark, when we clone on agent we use a java implementation of ssh.
Let's check if it can cause the slowdown.
Could you please do the following on the fast amazon disk:

mkdir repo;
cd repo;
git init;
git remote add origin git@github.com:<your account on github>/hotel-service.git;
date; env GIT_SSH=/path/to/git-ssh.sh git fetch origin '+refs/heads/master:refs/remotes/origin/master'; date

The script git-ssh.sh is attached. You should set variable 'agent_home_dir' to actual value of your agent home dir.

HI Dmitry,

Thanks for getting back to me, my appologis for any forcefullness on my part however i'm really trying to get everything sorted out with this.

But early indications show this is exactly the problem here!

I've not waited the time thats going to be needed to see if the checkout is going to take a long time or not, I can see its going to take ages right now...

Transfer speed when running those commands you've given me is VERY slow, compared to a normal 'git clone' without Java SSH involved which shows the tranfer speeds as much much higher (into the several MB/s) region

Currently doing this is showing me transfer speed to be at 174 KiB/s (Receiving objects:  10% (610/5561), 62.07 MiB | 174 KiB/s ), much much slower, which probably explains the reasons why its taking so long (almost certainly).

Normal git clone without Java involved running at the same time this checkout above is still running shows this speed:-
Receiving objects:  35% (1996/5561), 173.81 MiB | 11.65 MiB/s

Do you have ideas in which case on how to combat this problem?

Many Thanks and Regards,
Mark

PPS - Just Finished - Takes roughly same amount of time that we see to be a problem.
Thu Nov 25 08:16:30 UTC 2010
remote: Counting objects: 5561, done.
remote: Compressing objects: 100% (3818/3818), done.
remote: Total 5561 (delta 2421), reused 4542 (delta 1497)
Receiving objects: 100% (5561/5561), 490.25 MiB | 195 KiB/s, done.
Resolving deltas: 100% (2421/2421), done.
From github.com:holidayextras/hotel-service
* [new branch]      master     -> origin/master
Thu Nov 25 09:04:30 UTC 2010

Mark, I created an issue you can watch/vote for: http://youtrack.jetbrains.net/issue/TW-14598.
As a workaround for the problem I created build of git-plugin which uses a native ssh, it is attached to the issue.
Since plugin uses a native ssh you have to configure it properly.
In particular, you should setup ssh to automatically add new host keys to the $HOME/.ssh/known_hosts.
You can do it by putting something like this in your $HOME/.ssh/config:

Host github.com
User git
StrictHostKeyChecking no

Do not forget to save it in your amazon image.
Please let me know how it works for you.

Hi Dmitry,

Sorry for not getting back to you sooner, i've only just had the time to sort this out.

Great news though, it works perfectly and at a great speed. Now takes 1min 30secs instead of the 50mins. fantastic!!! I'm really pleased - thanks very much!

Couple of follow up quetions...
-> Why would others have not come across the same problem?
-> How comes you weren't able to reproduce? Was it that you were using http rather than SSH?
-> Why would this not be the method to use by default if Java ssh layer causes problems like this?

For anyone else - this site is good to know so you can see what happens during agents update and where to place plugins that you want to use etc.
http://confluence.jetbrains.net/display/TCD5/Plugins+Packaging

But really really fantastic, thanks again.
Mark

Hello Mark,

I'm happy to hear it works.

-> Why would others have not come across the same problem?

I think the main reason is that there are not so much people who have big repositories like yours and use git checkout on agent over ssh.

Another reason is a quality of network connection on an amazon agent.
When I tested checkout on a local agent, it was only twice slower than a native git clone, something like 100KiB/s vs 250KiB/s.
And on amazon agent clone using java SSH is still ~ 100KiB/s, but native git is 10MiB/s - so you can notice the difference.

-> How comes you weren't able to reproduce? Was it that you were using http rather than SSH?

This is still a mystery for me, but I'm sure we did use the SSH protocol.
We will investigate it futher.

-> Why would this not be the method to use by default if Java ssh layer causes problems like this?

Yes we are going to make it default method if it will not be possible to improve java implementation.

Thank you for you help!

Hi Dmitry,

Just a simple note to say thanks for answering my follow up questions. All makes sense and all is great.

Thanks again for helping to fully resolve the problems,
All the Best,
Mark