Git lfs on our agents stopped working -

We've got a 3-agent setup, and we can't get git lfs to fetch properly. The command line run of “git lfs fetch”

Step 2/9: git lfs fetch (Command Line)
Process exited with code 2 (Step: git lfs fetch (Command Line))
Step git lfs fetch (Command Line) failed

Here is what we have and tried so far:

  • Can run it manually on a repo without
  • Agents are mix of Win 10 and 11
    • Agent TC version 129421
  • Git versions up-to-date:
    • git version 2.42.0.windows.2
    • git-lfs/3.4.0 (GitHub; windows amd64; go 1.20.6; git d06d6e9e)
    • Run as build steps:
      21:10:30 Step 2/9: git setup 1 (git version) (Command Line)
      21:10:30   Starting: C:\BuildAgent\temp\agentTmp\custom_script13282388280661177589.cmd
      21:10:30   in directory: C:\BuildAgent\work\4ce89df19c227e8a
      21:10:30   git version 2.42.0.windows.2
      21:10:30   Process exited with code 0
      21:10:31 Step 3/9: git setup 2 (lfs version) (Command Line)
      21:10:31   Starting: C:\BuildAgent\temp\agentTmp\custom_script15185154921068000562.cmd
      21:10:31   in directory: C:\BuildAgent\work\4ce89df19c227e8a
      21:10:31   git-lfs/3.4.0 (GitHub; windows amd64; go 1.20.6; git d06d6e9e)
      21:10:31   Process exited with code 0
  • Also have TortoiseGit installed: Version 2.15.0
  • Have run “git lfs install --system”
  • Have run with build parameter “teamcity.git.use.native.ssh” set to “true”, no change (see this support case)
  • I've tried to set extra output with set GIT_TRACE=1 & set GIT_CURL_VERBOSE=1 & set GIT_TRACE_PERFORMANCE=1 & git lfs, but I get warnings instead of useful output:
    warning: unknown trace value for 'GIT_TRACE': 1
            If you want to trace into a file, then please set GIT_TRACE
            to an absolute pathname (starting with /)
    warning: unknown trace value for 'GIT_TRACE_PERFORMANCE': 1
            If you want to trace into a file, then please set GIT_TRACE_PERFORMANCE
            to an absolute pathname (starting with /)

This all seems like a mystery wrapped in a warning, wrapped in many failed test builds. Any ideas of how to fix some of this? I'll ask about the GIT_TRACE not being recognised on a git site too…

0
8 comments

OK, I figured out the issue with the command line environment variable setting - https://github.com/git-for-windows/git/issues/4694. I'll update with more info soon hopefully.

0

OK, so we're getting a credentials hang, it seems:

22:10:03   14:10:06.238834 trace git-lfs: filepathfilter: accepting "Web/projects.web/deal/actions/UploadRecord"
22:10:03   14:10:06.238834 trace git-lfs: filepathfilter: accepting "git-init.bat"
22:10:04   14:10:06.286521 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' 'HEAD' '--symbolic-full-name' 'HEAD'
22:10:04   14:10:06.313542 trace git-lfs: tq: running as batched queue, batch size of 100
22:10:04   14:10:06.313542 trace git-lfs: fetch Helpers/SurfaceTests/Data/.gitattributes [ca82a15e13b7163aef0def74f6e4a6fc5d1531268562e557084403ff032e62fc]
22:10:04   14:10:06.313542 trace git-lfs: fetch Helpers/SurfaceTests/Data/Grid.serialised [47e30620bf2ab8619d0b553f2f17519e54586a0bfd4567f3ba766e9e6f9f580c]
22:10:04   14:10:06.313542 trace git-lfs: fetch Helpers/SurfaceTests/Data/Fill.serialised [adb765ec6432dc106230a2b015202973747c736039a5e486dd337404c5b18678]
22:10:04   14:10:06.313542 trace git-lfs: tq: sending batch of size 3
22:10:04   14:10:06.313542 trace git-lfs: api: batch 3 files
22:10:04   14:10:06.313542 trace git-lfs: creds: git credential fill ("https", "dev.azure.com", "Org/SW/_git/SW")
22:10:04   14:10:06.314578 trace git-lfs: exec: git 'credential' 'fill'

I'll try setting teamcity.git.use.native.ssh to false.

0

It still hangs at the same line of trace with teamcity.git.use.native.ssh set to false. Hmm. Do I have this parameter name wrong or something else I can tweak? Should I use 0 or 1 instead?

0

Setting teamcity.git.use.native.ssh to 1 seems to have fixed it! Does that match with your experience of these parameters?

0

Nope, spoke too soon, it still has the same issue:

trace git-lfs: creds: git credential fill ("https", "dev.azure.com", "Org/SW/_git/SW")
trace git-lfs: exec: git 'credential' 'fill'

And then it hangs. No good.

0

OK, it worked for a single build this time. It depends what folder you run git lfs fetch from. It's running on my work computer, so I tested git lfs fetch on repositories I cloned myself. But when I went to the C:\BuildAgent\work folder to run the same command, it said Not in a Git repository.. Now that I know how to actually set “GIT_TRACE=1” (🥳), I did that locally, and it says this instead (details/folders changed a bit, seems personal):

...09:57:14.820421 trace git-lfs: Error running 'git rev-parse': failed to call `git rev-parse --git-dir`: exit status 128 : fatal: detected dubious ownership in repository at 'C:/BuildAgent/work/4ce89df19c227e8a'
'C:/BuildAgent/work/4ce89df19c227e8a' is owned by:
       'S-1-8-32-549'
but the current user is:
       'S-1-8-21-3301984702-4030361170-498914958-1001'
To add an exception for this directory, call:

       git config --global --add safe.directory C:/BuildAgent/work/4ce89df19c227e8a

Not in a Git repository.

Right, so for better or worse, I ran:

git config --global --add safe.directory C:/BuildAgent/work

And then git lfs fetch ran from the folder, and the next build ran without issue. However, subsequent builds have had the same issue as previously.

0

This seems to be relevant: https://stackoverflow.com/questions/33449967/cannot-run-git-fetch-from-team-city-command-line-build-step

Hmm, so maybe I need a credential helper. Gosh it's not obvious.

0

OK, so my colleage updated the credentials settings, and now we're getting a different error:

git-lfs/3.4.0 (GitHub; windows amd64; go 1.20.6; git d06d6e9e)
git version 2.42.0.windows.2

$ git-lfs.exe filter-process
Error downloading object: Helpers/CommonTests/Data/Capture76.bin (80a595e): Smudge error: Error downloading Helpers/CommonTests/Data/Capture76.bin (80a593e8f6c29c8c1ab46917d51099301010101823137730ef902ccb341184e0): batch request: unexpected end of JSON input

unexpected end of JSON input
batch request

Can anyone help please?

0

Please sign in to leave a comment.