Metadata file could not be found

I have a weird situation here. I have two build configurations, one for test and one for master. With a few smaller insignificant differences, they are the same. Each configuration is building the same two solutions, when I push code to GIT. This has been working for a long time. Let's just call them Solution1 and Solution2.

In test Solution1 is built before solution2. There were one difference between the two build configurations. In master both steps for building the solutions was using .Net as runner type, but in test Solution2 was set to use one of the deprecated runner types (don't remember which one). I just upgraded it to use the same as in the master configuration

This change has for some reason caused Solution1 to fail with the following error:

CSC : error CS0006: Metadata file 'C:\Users\xxx\.nuget\packages\microsoft.codeanalysis.analyzers\2.9.4\analyzers\dotnet\cs\Microsoft.CodeAnalysis.Analyzers.dll' could not be found

I saw that the packages directory was empty, so I copied the content from the same directory from my development machine. Now I get this error instead

C:\Program Files\dotnet\sdk\3.1.120\Sdks\Microsoft.NET.Sdk\targets\Microsoft.PackageDependencyResolution.targets(234,5): error NETSDK1064: Package Microsoft.CodeAnalysis.Analyzers, version 2.9.4 was not found. It might have been deleted since NuGet r...

Keep in mind I only changed the build step for Solution2, but I'm getting the errors when building Solution1

Any idea on what could be the problem?

I have tried the following command, but that didn't make any difference:

dotnet nuget locals --clear all