Agent has old files in work directory

About a month ago we upgraded to 2020.2 (build 85487). There have now been two cases where a work directory has files belonging to the previous build of a different git branch, making the build fail because of a unit test for unused source files.

• If I restart the build on the same agent, the build fails with the same issue.
• If I do a clean checkout on the same agent, the build works fine.
• In the vast majority of cases (10s of builds daily), there are no issues.
• We run Swabra at the start of each build. The .snapshot file used includes the files that should not be there.
• Before the upgrade, we never had this issue

How do I begin to troubleshoot this?