Pull Request build configuration still builds refs/head/master

We have a pull-request build configuration against GitHub.  It's working great - continuously builds pull requests, both when they are initially created and whenever they are updated.

However, it keeps building refs/heads/master whenever we merge against our master branch.  

My VCS Root is setup as such:

Default Branch: (empty)
Branch Specification:
    +:refs/pull/*/merge
    -:refs/heads/master

Why does it keep wanting to build against refs/heads/master ? It effectively reduces the # of available agents available to use.