Hello Sarah!
Yes you can definitely remove the "Sign in with Google" button when you are using a custom domain.
You just need to set None for SSO.
But this unfortunately does not work with the base URL domain. We have an open linear for that, which I can definitely add a +1 onto the ticket for you.
For now, you would just need to use a custom URL for the Google SSO button to be removed.