When switching branches in a project, the build takes a lot of time than if we were building one branch at a time. Can this be fixed?

There is a project in which there are two branches. For the first time, assembling one of the branches takes 1 hour. Subsequent assemblies take 3-5 minutes. When switching to the second branch, the build again takes 1 hour.
Is it possible to somehow configure caching so that when switching branches, the build takes 3-5 minutes?