Hey @brettski!
That toggle is for the status that appears in the bottom right which shows which queries are currently running (and for how long):
At the moment those query notifications need to be disabled per query:
Though there is a feature request to disable them globally (which I see you've already discovered 
)