Git lfs on our agents stopped working -
We've got a 3-agent setup, and we can't get git lfs to fetch properly. The command line run of “git lfs fetch”
Step 2/9: git lfs fetch (Command Line)
Process exited with code 2 (Step: git lfs fetch (Command Line))
Step git lfs fetch (Command Line) failed
Here is what we have and tried so far:
- Can run it manually on a repo without
- Agents are mix of Win 10 and 11
- Agent TC version 129421
- Git versions up-to-date:
- git version 2.42.0.windows.2
- git-lfs/3.4.0 (GitHub; windows amd64; go 1.20.6; git d06d6e9e)
- Run as build steps:
21:10:30 Step 2/9: git setup 1 (git version) (Command Line)
21:10:30 Starting: C:\BuildAgent\temp\agentTmp\custom_script13282388280661177589.cmd
21:10:30 in directory: C:\BuildAgent\work\4ce89df19c227e8a
21:10:30 git version 2.42.0.windows.2
21:10:30 Process exited with code 0
21:10:31 Step 3/9: git setup 2 (lfs version) (Command Line)
21:10:31 Starting: C:\BuildAgent\temp\agentTmp\custom_script15185154921068000562.cmd
21:10:31 in directory: C:\BuildAgent\work\4ce89df19c227e8a
21:10:31 git-lfs/3.4.0 (GitHub; windows amd64; go 1.20.6; git d06d6e9e)
21:10:31 Process exited with code 0
- Also have TortoiseGit installed: Version 2.15.0
- Have run “git lfs install --system”
- Have run with build parameter “teamcity.git.use.native.ssh” set to “true”, no change (see this support case)
- I've tried to set extra output with
set GIT_TRACE=1 & set GIT_CURL_VERBOSE=1 & set GIT_TRACE_PERFORMANCE=1 & git lfs, but I get warnings instead of useful output:
warning: unknown trace value for 'GIT_TRACE': 1
If you want to trace into a file, then please set GIT_TRACE
to an absolute pathname (starting with /)
warning: unknown trace value for 'GIT_TRACE_PERFORMANCE': 1
If you want to trace into a file, then please set GIT_TRACE_PERFORMANCE
to an absolute pathname (starting with /)
This all seems like a mystery wrapped in a warning, wrapped in many failed test builds. Any ideas of how to fix some of this? I'll ask about the GIT_TRACE not being recognised on a git site too…
Please sign in to leave a comment.
OK, I figured out the issue with the command line environment variable setting - https://github.com/git-for-windows/git/issues/4694. I'll update with more info soon hopefully.
OK, so we're getting a credentials hang, it seems:
22:10:03 14:10:06.238834 trace git-lfs: filepathfilter: accepting "Web/projects.web/deal/actions/UploadRecord"22:10:03 14:10:06.238834 trace git-lfs: filepathfilter: accepting "git-init.bat"22:10:04 14:10:06.286521 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' 'HEAD' '--symbolic-full-name' 'HEAD'22:10:04 14:10:06.313542 trace git-lfs: tq: running as batched queue, batch size of 10022:10:04 14:10:06.313542 trace git-lfs: fetch Helpers/SurfaceTests/Data/.gitattributes [ca82a15e13b7163aef0def74f6e4a6fc5d1531268562e557084403ff032e62fc]22:10:04 14:10:06.313542 trace git-lfs: fetch Helpers/SurfaceTests/Data/Grid.serialised [47e30620bf2ab8619d0b553f2f17519e54586a0bfd4567f3ba766e9e6f9f580c]22:10:04 14:10:06.313542 trace git-lfs: fetch Helpers/SurfaceTests/Data/Fill.serialised [adb765ec6432dc106230a2b015202973747c736039a5e486dd337404c5b18678]22:10:04 14:10:06.313542 trace git-lfs: tq: sending batch of size 322:10:04 14:10:06.313542 trace git-lfs: api: batch 3 files22:10:04 14:10:06.313542 trace git-lfs: creds: git credential fill ("https", "dev.azure.com", "Org/SW/_git/SW")22:10:04 14:10:06.314578 trace git-lfs: exec: git 'credential' 'fill'I'll try setting
teamcity.git.use.native.sshto false.It still hangs at the same line of trace with teamcity.git.use.native.ssh set to false. Hmm. Do I have this parameter name wrong or something else I can tweak? Should I use 0 or 1 instead?
Setting teamcity.git.use.native.ssh to 1 seems to have fixed it! Does that match with your experience of these parameters?
Nope, spoke too soon, it still has the same issue:
trace git-lfs: creds: git credential fill ("https", "dev.azure.com", "Org/SW/_git/SW")trace git-lfs: exec: git 'credential' 'fill'And then it hangs. No good.
OK, it worked for a single build this time. It depends what folder you run
git lfs fetchfrom. It's running on my work computer, so I testedgit lfs fetchon repositories I cloned myself. But when I went to theC:\BuildAgent\workfolder to run the same command, it saidNot in a Git repository.. Now that I know how to actuallyset “GIT_TRACE=1”(🥳), I did that locally, and it says this instead (details/folders changed a bit, seems personal):...09:57:14.820421 trace git-lfs: Error running 'git rev-parse': failed to call `git rev-parse --git-dir`: exit status 128 : fatal: detected dubious ownership in repository at 'C:/BuildAgent/work/4ce89df19c227e8a''C:/BuildAgent/work/4ce89df19c227e8a' is owned by:'S-1-8-32-549'but the current user is:'S-1-8-21-3301984702-4030361170-498914958-1001'To add an exception for this directory, call:git config --global --add safe.directory C:/BuildAgent/work/4ce89df19c227e8aNot in a Git repository.Right, so for better or worse, I ran:
git config --global --add safe.directory C:/BuildAgent/workAnd then
git lfs fetchran from the folder, and the next build ran without issue. However, subsequent builds have had the same issue as previously.This seems to be relevant: https://stackoverflow.com/questions/33449967/cannot-run-git-fetch-from-team-city-command-line-build-step
Hmm, so maybe I need a credential helper. Gosh it's not obvious.
OK, so my colleage updated the credentials settings, and now we're getting a different error:
git-lfs/3.4.0 (GitHub; windows amd64; go 1.20.6; git d06d6e9e)git version 2.42.0.windows.2$ git-lfs.exe filter-processError downloading object: Helpers/CommonTests/Data/Capture76.bin (80a595e): Smudge error: Error downloading Helpers/CommonTests/Data/Capture76.bin (80a593e8f6c29c8c1ab46917d51099301010101823137730ef902ccb341184e0): batch request: unexpected end of JSON inputunexpected end of JSON inputbatch requestCan anyone help please?